Vision transformer and CLIP

In short: A vision transformer (ViT) splits an image into small tiles (patches), treats them like tokens and processes them with a transformer. CLIP learns to map images and texts into the same vector space.

In more detail: This lets the same architecture process text and images. It is the basis of multimodal models that “see” and describe images.

In detail

• ViT: image 224×224 → 196 patches of 16×16 pixels → one vector each → plus position → transformer blocks.
• CLIP (OpenAI, 2021): two encoders (image and text), trained on hundreds of millions of image-text pairs so that matching pairs lie close together (contrastive learning).
• Zero-shot: an image can be assigned to classes without special training by embedding the class names as text and comparing.
• Multimodal language models: an image encoder provides image tokens that are given to the language model together with the text. That way the model can read photos, diagrams, screenshots.
• Applications: image search, captions, OCR, document understanding, accessibility (alt text).
• Limits: counting, spatial relations and fine details are often wrong.

See also: Convolutional networks (CNN) for images, Embedding (vector representation)