10. Multimodal AI

This chapter works at the model layer of the five layers. Chapters 4 to 9 each took one medium and followed it down: what the model represents, how you steer it, and where it breaks. The interesting fact about 2026 is that the media have stopped being separate. One model takes your paragraph, your photograph, and your voice memo together, and answers in whichever of those you ask for.

Until around 2023 the tools were medium-specific. A chat product did text, an image service did images, and a music system such as MusicLM did music, and moving between them meant exporting a file and starting again. A multimodal model is a single architecture that takes many media in and produces many media out.

This is one of the shorter chapters in the book, and deliberately so. Almost none of the machinery is new: it is the transformer from chapter 2 with more kinds of input attached. What is new is what you can do with it, and that is a matter of working method rather than of theory. Chapter 11 takes the next step, to systems that act rather than answer.

Many media, one model¶

A word about words before the recipe, because this book holds to two distinctions that the field is loose about. The field calls each kind of data a modality, and it calls a model that takes several of them multimodal. This book says data type instead: text, image, audio, video, 3D geometry, motion data. The reason is that psychology has long used modality for a human sense, and the two meanings are easy to confuse. Multimodal stays, since it is the field’s name for the models, but perception is a sense or a sensory modality, and people are multisensory rather than multimodal. The second distinction is that audio and video are data, the recorded, transmitted, or generated signal, while auditory and visual are perception and processing, whether in a person or in a machine. So a model takes audio rather than hears it, a film is an audio–video artefact, and watching one is an auditory–visual experience. Both distinctions come from the Sensing Sound and Music e-book and from the argument for splitting the older single word in two Jensenius, 2025.

The technical recipe is roughly three steps.

1. Encode each data type, whether text, image, or audio, into a sequence of vectors using a small encoder built for that type.
2. Pass them through a shared transformer Vaswani et al., 2017 that does not care which data type they came from.
3. Decode to whichever data type is wanted on the way out.

Figure: A multimodal model takes text, images, audio, and video in and returns text, images, audio, and actions, with a single shared representation in the middle.

Step two is the one that matters. Once a picture and a sentence have been turned into sequences of vectors, the transformer treats them the same way, because attention operates on a sequence and has no opinion about where the sequence came from. The old boundary between media was a boundary between file formats and between tools, and inside the model it simply is not there.

What makes the shared space useful is that it is trained to line up. The clearest example is the image-and-text encoder introduced in chapter 5 as the thing that steers an image model. Trained on a very large collection of pictures with their captions, it learns to place a photograph of a dog and the words a dog near each other in the same space Radford et al., 2021. In chapter 5 that alignment was a steering mechanism. Here it is the joining mechanism, and the same trick generalises to sound, to video frames, and to anything else you can find paired with language.

Two consequences are worth holding on to. The first is that alignment is learned from pairs, so a multimodal model knows about a relationship between two media roughly to the extent that humans have written that relationship down. Pictures with captions are abundant, sound with careful descriptions much less so, and movement with descriptions barely at all, which is why the media in this book get steadily harder as the chapters go on.

The second is that there is no picture in the middle of the model. There is a sequence of vectors that a picture was turned into, and vectors that a caption was turned into, and a mechanism that has learned to relate them. To say that the model sees is to describe the input port, not an experience. The critical look at the end of this chapter is about what that distinction costs you in practice.

Show rather than tell¶

The practical consequences for creative work are immediate, and each of them replaces a paragraph of description with an artefact.

• You can paste an image and ask a question about it, which is useful for design critique, architecture review, and art history.
• You can hand the model a screenshot and ask it to fix the interface.
• You can play it a thirty-second clip and ask for the genre, the tempo, and the emotional tone.
• You can talk to it like a phone call, using the voice mode that most chat products offer by 2026.
• You can give it a sketch and ask for a polished version, or a polished image and ask for a structural sketch.

Taken together these unlock a different kind of prompt: “show, don’t tell”. The most useful multimodal prompt is usually here is what I am working with, here is what I want, please help, and the first of those three is a file rather than a sentence.

A workable shape for it has four parts.

[The artefact: the image, the clip, the screenshot, the draft]
[The context: what it is for, who sees it, what stage it is at]
[The ask: the one thing you want, stated as a task rather than a topic]
[The form: how long the answer should be and what it should be a list of]

A worked example, with a photograph of a poster attached:

Here is a draft poster for a student concert, A3, printed and pinned
in a corridor where people walk past at about a metre a second.
Tell me what a passer-by can read in two seconds and what they cannot.
Answer as two lists, readable and not readable, nothing else.

Notice what the second and third lines do. Without the context the model will describe the poster, because describing is the thing it is most confident at. With the context it has a task with a criterion, and the answer becomes checkable. The single largest improvement you can make to a multimodal prompt is to say what the artefact is for, because the corridor is not in the file you uploaded.

The change-one-thing discipline from chapter 5 still applies. If you swap the image and rewrite the ask at the same time, you have learned nothing about either.

Multimodal critique¶

The most immediately useful thing a multimodal model does for your own work is critique. You upload the layout you have been staring at for three hours and ask what is wrong with it, and something comes back within seconds and at no social cost. That combination is genuinely valuable, and it is also the reason to be careful.

Sort what you get back into three piles.

Description. What is in the artefact: the elements, the colours, the text, the ordering, the composition. This is what the model is best at, and it is more useful than it sounds. A description written by something that has never met your intention shows you what the artefact actually communicates rather than what you meant it to.

Convention. Whether the artefact follows the rules of its genre: contrast for legibility, heading hierarchy, a mix that leaves room for a voice, a cut on an action. These rules are written down in many places, the model has read them, and its recall of them is decent.

Taste. Whether the piece is any good. Here the model is pattern-matching against the vocabulary that critics and caption writers have used about other people’s work, which is not the same thing as judgement. It will find your piece competent and slightly cluttered, because that is what a great deal of writing about images says. It has no stake in the outcome, no memory of what you were trying to do, and no ability to be disappointed.

The practical protocol follows from that split: ask for the description before the judgement. If the model tells you your poster has a blue background when it is green, or reports a saxophone where you played a clarinet, then everything it says afterwards is built on a misreading and you can stop reading. A wrong description is the cheapest reliable warning you will get, and it is not rare.

The second rule is to ask for the failure rather than the verdict. What is the first thing a reader will misunderstand produces something you can act on. Is this good produces a paragraph of encouragement. This is the difference that Salma and colleagues describe, and that chapter 3 took up at length, between treating a system as an executor of instructions and treating it as a participant in a process Salma et al., 2025. The participant version requires you to keep the question open long enough for an answer to be uncomfortable.

Translation between data types as a method¶

There is a second use for a multimodal model that is less obvious than critique and, for creative work, often more productive. Instead of asking it to make something, ask it to carry something across a boundary. The boundary is the one between data types, and so between media. It is not the boundary between the senses that psychology means by cross-modal, a term this section returns to at the end.

The move is simple. Take an artefact in one medium, have the model describe it in words, and then use that description as the brief for a different medium. An image becomes a paragraph becomes a sound brief. A sound becomes a paragraph becomes an image prompt. A text becomes a paragraph becomes a storyboard. You can send it back the other way afterwards, which is where it becomes interesting.

The text in the middle is a bottleneck, and the bottleneck is the point. A photograph contains far more than any paragraph about it, so the description has to commit: it has to decide that the picture is about the cold light rather than about the chair. That commitment is a design decision, and having it made explicitly, in words you can argue with, is worth more than the output at the far end.

Figure: A work crosses a boundary by being described in a paragraph, and that paragraph becomes the brief in a second medium. The paragraph is a bottleneck, and comparing the two ends is where the method does its work.

Three things make the method work in practice.

Translate your own material, not a found image. You need to be able to tell what was lost, and you can only do that for something you made.

Read the middle. If the description is bland, the far end will be bland, and the fix is to edit the paragraph rather than to regenerate the artefact. This is the same lesson as chapter 4: the intermediate text is a draft, not a result.

Compare, do not replace. Put the original and the round trip side by side. What survived the crossing is the part of the work that is legible to a stranger. What did not survive is either a failure of the artefact or the part of it that is genuinely yours, and telling those two apart is the skill this exercise trains.

There is a longer tradition behind this. Music psychology has studied cross-modal correspondences for decades: the reliable tendencies by which people match a high pitch to something small, bright, and high in space, or a rough timbre to a jagged shape. These correspondences hold across people well enough to be experimented on Spence, 2011. A multimodal model inherits a version of them at second hand, through the language people use about both media. That is why translating a sound into an image often produces something recognisable rather than arbitrary, and also why it produces the obvious answer rather than a surprising one.

Where multimodal models fail¶

The failures are systematic, they are the same across systems, and they follow from the architecture rather than from insufficient training.

Grounding. The model produces a plausible description rather than an accurate one. Asked what is in a picture, it answers with what is usually in pictures like that, so a laboratory bench acquires a microscope that is not there. The description is right about the kind of scene and wrong about the instance.

Counting. Ask how many chairs, how many people, how many bars of the graph exceed the line, and the answer is confident and frequently wrong. Attention is good at what is present and poor at how many times.

Small text. In most current systems an image is cut into patches, small square blocks of pixels, and encoded first, so detail much below the size of a patch is gone before processing begins. Fine print in a screenshot, an axis label, or the small type at the bottom of your poster may simply not be there. Upload a crop instead of asking harder.

Spatial reasoning. Left and right, in front and behind, whether two lines cross, whether one shape overlaps another. A 2024 study found the leading vision-language models of that year performing poorly on deliberately simple geometric tasks that people find trivial, and the critical look at the end of this chapter takes that study apart Rahmanzadehgervi et al., 2024.

The working rule that comes out of this is short: trust the model’s description of what kind of thing it is looking at, and verify anything that is a measurement. Counts, positions, sizes, and small print are measurements. Mood, genre, style, and convention are not, and those are the questions this chapter is really about.

This week’s lab: Explore, Reflect, Create¶

Three movements: interrogate a photograph until the model runs out of knowledge, argue with a partner about where that happened, and translate a piece of your own work across a boundary.

The project proposal is due this week. It is one to two pages plus a feasibility sketch, set in week 9, and it must name whether your project will be a performance or an installation at the gallery. It is obligatory and assessed pass or fail, so keep time clear for it before you start; the translation below is a portfolio piece and does not replace it.

Explore (about 30 min)¶

One photograph, three questions, then a redesign.

1. Take a photograph of something messy or interesting in your daily life: your desk, a drawer, a noticeboard, a chord diagram from a music book, a corner of a rehearsal room. Your own photograph, not a found image, because you need to know what is actually in it.
2. Upload it to a multimodal chat assistant and ask three increasingly specific questions about it. Start with what it is, move to how it is organised, finish with something that requires the model to read a detail or count something. Save every answer.
3. Check the third answer against the photograph. If it is wrong, say precisely how: a count, a position, a piece of small text, or an object that is not there. This is your grounding test, and a failure here is a result rather than a disappointment.
4. Pivot to a redesign. Ask the model to redesign what is in the image: a tidied desk, a different voicing of the chord, a noticeboard someone might actually read. Ask for reasons alongside the suggestions.
5. Mark each suggestion D, C, or T for description, convention, or taste, using the three piles from Multimodal critique above. Count them. The proportions are the finding.

Reflect (about 15 min)¶

Work in pairs. This is a discussion, not a writing block.

1. Show each other your redesign answers and your D, C, and T marks, and argue about the ones you disagree on. Settle on a single sentence for each partner’s transcript naming where design knowledge ended and pattern-matching began. Be specific about the sentence where it happened.
2. Compare grounding failures. Whose model got something wrong about the picture, and did anything in the redesign depend on the thing it got wrong?
3. Close the round by each stating aloud the piece of your own work in progress you are about to translate, and into which medium. Say it out loud, because saying it is what stops you picking something safe.

Create (about 45 min)¶

Translate one piece of your own work in progress into another data type.

1. Choose the material and the direction. An image described into a sound brief, a sound described into an image, or a text turned into a storyboard. It must be your own material and it must be something you are still working on.
2. Get the description. Have the model describe the artefact in words. Ask for the description before anything else, and check it against the original.
3. Edit the description into a brief. This is the real work of the block. Cut what is generic, sharpen what is specific, and make it commit to what the piece is about. Keep both versions.
4. Generate at the far end, using the medium’s own chapter: 5 for images, 6 for sound, 7 for a storyboard sequence.
5. Put the original and the translation side by side and write down what survived the crossing and what did not. Name one thing that was lost that you now want to make more explicit in the original.
6. Commit the original, the two versions of the description, the translation, and those notes to your portfolio.

At home, write this week’s entry in your practice log using the practice log template.

A critical look: does a model “see”?¶

The claim. Vision-language models see. They accept an image, describe it fluently, answer questions about it, and correct you when you misremember a detail, and the natural conclusion is that whatever is happening inside is a version of what happens when you look at something. The interfaces encourage this reading, the verb see is used without qualification in the marketing, and the outputs are good enough that the reading rarely gets tested.

The evidence. It does not survive contact with simple tasks. A 2024 study set leading vision-language models a set of visual problems that any sighted person solves immediately. The tasks were counting how many times two line plots intersect, saying whether two circles overlap, reading which letter in a word has been circled, and counting the rows of a grid. The models performed poorly across the set, and the paper’s blunt title, Vision language models are blind, reports a real finding in deliberately provocative terms Rahmanzadehgervi et al., 2024. The same systems, on the same day, will write an accurate and sensitive paragraph about the mood of a photograph. Both results are real, and it is their combination that tells you what is going on.

The method. Notice how that evidence was produced, because it is the opposite of a benchmark leaderboard. The tasks are small, synthetic, and generated in bulk with the answers known by construction. That leaves no ambiguity about what is correct, makes contamination very unlikely, and gives no room for a graceful answer to score as a right one. Standard visual benchmarks ask questions about photographs, and a model can score well on those by knowing what photographs of that kind usually contain. A synthetic test with a known answer removes exactly that escape route. When you meet a claim about what a model can perceive, ask whether the test could be passed by a system that had read a great deal about the world and looked at nothing.

The limits. The honest conclusion is not that these models are useless at images, which the rest of this chapter contradicts. It is that they are strong at semantics and weak at geometry. They are reliable about what kind of thing a picture is, what genre it belongs to, what conventions it follows, and what mood it carries. They are unreliable about counts, positions, overlaps, and anything that requires holding a spatial relation steady. That split is not a defect to be patched in the next release, or at least it is not obviously one, because it follows from encoding an image into a sequence of patches and relating it to language. For you the split is operational rather than philosophical. It tells you which half of a critique to act on and which half to check, and a designer who knows the boundary gets more out of these systems than one who believes the verb.