Multimodal models: image, audio and text in one network

In short: Multimodal models process several kinds of data together, such as text, images and audio. The trick: everything is translated into the same language of vectors that a transformer can process.

How it works

1. Image: it is cut into tiles (“patches”), each tile becomes a vector, as in Vision transformer and CLIP.
2. Audio: the signal is cut into short sections or a spectrogram and also turned into vectors, see Speech and audio.
3. Text: as usual in tokens, see Tokenisation, BPE and vocabulary.
4. All vectors end up as one common sequence in the same transformer. Self-attention can then mediate between image parts and words.

Two designs

• Attached encoders: a ready image encoder is hooked to a language model through a small adapter. Quick to train.
• Joint from the start: the model is trained from scratch on mixed data, also with images and audio as output.

Limits

Models confuse details in images, count badly and read small print unreliably. Images can also contain hidden instructions, see Prompt injection.

See also: Computer vision: tasks and methods