Skip to main content
Lab Grimoire
TW EN
Coffee
LLM Internals

Vision-language models: how AI looks at a picture and answers in words

A vision-language model (VLM) takes an image plus a question and returns text. This piece explains how it turns image patches into tokens a language model can read, what it is good for, where it misreads small print, counts, positions and specialist images, and how it differs from image generators and embedding models.

Author
CY
Published
Updated
AI from Principles to Practice · Part 6 of 70
Vision-language models: how AI looks at a picture and answers in words

An image and a question go in, a written answer comes out

A vision-language model (VLM) takes an image together with a text question and returns text.

Give one a screenshot of a bar chart and ask what the Y axis measures, and it will find the axis label and tell you in a sentence. A text-only large language model cannot do this, because the picture never reaches it. This piece zooms in on one square of a map of AI model types, the one where an image plus text goes in and text comes out. It covers how the model reads a picture, what it is useful for, where it most often gets things wrong, and how to tell it apart from its neighbours on the map.

Summary: vision-language models

How it sees: cut the picture into patches, turn them into words

Most vision-language models are two parts joined together: an image encoder and a large language model.

The image encoder first cuts the picture into a grid of small, fixed-size squares, a bit like slicing a photo into jigsaw pieces. The Vision Transformer (ViT) showed that this patch approach works for recognising images directly: each patch becomes a list of numbers, and the model works out how the patches relate to one another. LLaVA takes an existing image encoder (from CLIP) and adds one simple projection layer that reshapes each patch into the same format as the language model's word tokens. Those patch tokens are placed in front of your question. To the language model they read like a special stretch of text, and once it has read them it writes the answer word by word, as it always does. LLaVA is then trained on image question-and-answer data so that it learns to follow instructions about pictures.

Flamingo takes a different route. It inserts extra layers inside the language model so that the model can look back at the image while it writes, and it can handle several images mixed in with text.

What it is good for: a first look that reads the picture for you

The everyday uses are describing a photo, reading a screenshot of a table or chart, and pulling text out of a scanned page. Photograph a road sign or a product leaflet abroad, and it can read the words and translate them.

In research it makes a good first pass. Faced with a figure in a paper, you can ask which groups are being compared or what the horizontal axis stands for, get the layout of the figure quickly, and then decide whether to read the paper closely. With a stack of scanned forms, it can turn the fields into text for a person to check line by line.

Where it gets things wrong

It slips most often in five situations.

Tiny print and dense tables of numbers can lead it to misread a digit or pick the wrong column. Ask it how many cells or people are in a picture, and the count is often off. It also tends to miss on precise position, such as which object sits in the top-left corner. With specialist images it rarely saw in training, such as microscopy slides or radiology scans, it can produce a reading that sounds expert and has nothing behind it. And it can describe details that are not in the picture at all, in the same confident tone it uses when it is right.

So any number that will go into a report should be checked against the original image or the underlying data, and specialist images still need a trained reader.

Telling it apart from its neighbours

When choosing a tool, first check whether you have a picture, then decide what kind of output you need.

A large language model accepts only text and cannot see images; if your question does not involve a picture, it is enough. An image generation model runs the other way, taking text in and producing a picture, so reach for it when you want a new image. Among embedding models, CLIP also handles both images and text, but it places them in one shared coordinate space and uses distance to judge which caption fits which picture. That makes it good for searching images with words or grouping them, and it never writes an answer. Looking at a picture and replying in sentences is the vision-language model's job.

References

  1. https://arxiv.org/abs/2304.08485
  2. https://arxiv.org/abs/2010.11929
  3. https://arxiv.org/abs/2103.00020
  4. https://arxiv.org/abs/2204.14198

Frequently Asked Questions

Can I quote numbers that a vision-language model reads from a figure in a paper?

It is better not to quote them directly. When a vision-language model reads tiny print, dense tables or fine axis ticks, it can misread a digit or pick the wrong column, and it sounds just as sure when it is wrong. Use it to get a quick sense of what a chart compares and what its axes mean; any value that goes into a report or slide should be checked against the paper, its supplementary data, or the raw data.

Can a vision-language model interpret microscopy slides or medical images?

Only as a rough reference, not as an interpretation you can rely on. General-purpose vision-language models are trained mostly on web photos and image-text pairs, and specialist images such as microscopy slides or radiology scans are rare in that data. The model can produce a description that sounds expert and has no real basis. Specialist images still need a trained reader, and anything diagnostic belongs with a physician.

Found this useful? RSS · X · Buy me a coffee