Skip to main content
Lab Grimoire
TW EN
Coffee
LLM Internals

A map of AI model types: what goes in, what comes out, and which one to pick

AI is more than chatbots. This map sorts large language models, decision models, embedding models, rerankers, encoder classifiers, vision-language models, image generators, speech models, and biological sequence models by input and output, with what each is for and when to choose it.

Author
CY
Published
Updated
AI from Principles to Practice · Part 1 of 70
A map of AI model types: what goes in, what comes out, and which one to pick

Some AI models write essays; most return a score, a label, or a list of numbers

The most useful way to sort AI models is to ask what goes in and what comes out.

When a search engine finds the article you wanted, your phone captions a video, or your photo app finds every picture with a cat in it, how many of the models involved can hold a conversation? Very few. Some output a paragraph of text; many output only a score, a label, or a list of coordinates. This article lays out nine common model types as a map. As with any map, you begin by finding where you start and where you want to go. Here, each type gets the input, the output, what it is good for, and when to reach for something else.

Summary: A map of AI model types

Each of the nine types has its own short piece, linked under Read more at the end of its section.

Large language models: text in, text out

A large language model (LLM) reads a passage of text and continues it one token at a time, producing new text.

Its training goal is simple: given what came before, predict the next piece of a word. Most current LLMs use the Transformer architecture, whose attention mechanism decides which earlier words each position should draw on. Once a model is large enough and has read enough text, a few examples in the prompt often let it translate, summarize, rewrite, or answer questions. Reasoning models are a branch of LLMs that write out a long chain of intermediate steps before giving an answer. This makes math and multi-step planning more reliable, at the cost of more time and compute. Reach for an LLM when what you need is newly written text.

Read more: How large language models work: one token at a time, up to a whole answer

Decision models: scoring a fixed list of options

A decision model reads a situation and a predefined set of options, and returns a probability for each option.

It writes no text, so there is nothing to parse and no risk of an answer outside the list. It fits jobs such as "which study type is this abstract?" or "which department should this email go to?", where the options are fixed and you only want the most likely one. This site has a full series on the topic: start with What is a decision model, read the mechanism in How decision models work and are trained, or browse the AI from Principles to Practice series.

Embedding models: turning content into coordinates on a map

An embedding model turns a passage of text or an image into a fixed-length list of numbers, called a vector.

A vector works like a coordinate on a high-dimensional map, so sentences with similar meanings land close together. Sentence-BERT made it practical to compare sentences by measuring the distance between their vectors. CLIP places images and text on the same map, which is why a short phrase can find a matching photo. Typical uses include semantic search, clustering large document sets, spotting duplicates, and the first step of RAG (retrieval-augmented generation), where likely relevant passages are pulled from a database and handed to an LLM to answer. An embedding model does not write anything.

Read more: Embedding models: turning text into coordinates so computers can search by meaning

Rerankers: putting retrieved candidates back in order

A reranker reads a question and one candidate passage together and returns a relevance score.

An embedding model converts the question and each document into coordinates separately and compares distances. That is fast, but the two are never read side by side. A reranker is a cross-encoder: it reads the question and the candidate at the same time, so its judgment is usually more accurate, but every pair has to be computed from scratch. In practice, systems often work in two stages: an embedding model or keyword search pulls a batch of candidates, and a reranker picks the top few. A different approach lets a decision model choose material level by level; see this site's article on DecisionRAG.

Read more: Rerankers: retrieve a hundred results, then put the ones that answer the question on top

Encoder classifiers and sequence labeling: tagging text and marking names

An encoder classifier reads a whole passage and outputs one label; sequence labeling assigns a label to every word.

BERT and similar encoders are the best-known examples. They read each word together with the words before and after it, then a small classification layer is added and trained on examples you have labeled. One label per passage covers tasks like spam detection, review sentiment, and document topic. One label per word covers named entity recognition (NER), which marks names of people, drugs, or genes in text. These models are usually far smaller than generative models, so they run fast and cheaply, and they work well when the categories are fixed and labeled data exists. The limitation is that categories must be defined in advance; a category never seen in training cannot be predicted.

Read more: Encoder classifiers and sequence labeling: one label per text, or one label per word

Vision-language models: looking at an image and answering in text

A vision-language model (VLM) takes an image plus text as input and returns text.

A common design first uses an image encoder to cut the picture into small patches and convert them into a representation the model can read, then passes that to a language model. LLaVA, for example, connects an image encoder to an LLM and trains it on image question-and-answer data. You can ask a VLM to describe a photo, read a screenshot of a table, or tell you what the Y axis of a chart measures. Its output is still text; it does not draw. It can misread small print, dense numbers, or specialized images, so check important values against the original image.

Read more: Vision-language models: how AI looks at a picture and answers in words

Image generation models: drawing a picture from a description

An image generation model reads a text description and outputs an image; today the dominant approach is the diffusion model.

A diffusion model is trained to take an image with added noise and remove that noise step by step. To generate, it starts from pure noise and repeatedly removes noise in the direction the text describes, until a picture emerges. Latent diffusion runs this process in a compressed space, which cuts the compute needed considerably, and many open-source image tools build on it. These models are useful for illustrations, concept sketches, and design references. A picture that looks plausible is not necessarily accurate, though: instrument parts, anatomy, and any text in the image need human checking.

Read more: How image generation models draw: wiping a picture out of noise, and why the result is not evidence

Speech models: turning sound into text, and text into sound

Speech models mainly go in two directions: automatic speech recognition (ASR) turns audio into text, and text-to-speech (TTS) turns text into audio.

An ASR model takes an audio clip and returns a transcript. Whisper was trained on a large amount of audio paired with transcripts collected from the web. It handles many languages and accents and is often used for meeting notes and video captions. TTS goes the other way. Systems such as Tacotron 2 first predict a spectrogram of the sound from the text, then a second model turns that spectrogram into an actual waveform. Some newer products chain listening, thinking, and speaking into one flow, but these two kinds of models are usually still inside. Technical terms, heavy accents, and noisy rooms are where recognition goes wrong most often.

Read more: How speech models work: recognition turns sound into text, synthesis turns text into sound

Biological sequence and structure models: reading proteins and DNA

A biological sequence model reads a protein's amino-acid sequence or a DNA base sequence much as a language model reads text, and outputs a predicted structure or effect.

Protein language models such as ESM-2 are trained on very large numbers of known protein sequences and learn which amino-acid combinations tend to occur together. The same research team used one to predict 3D structure from a single sequence. AlphaFold is built specifically to predict how a protein folds. DNA models read genome sequence to predict how a stretch of DNA, or a variant within it, affects gene regulation; the AlphaGenome Atlas covered on this site is one example. These models produce predictions that help researchers decide which candidates to test first. They do not replace experiments.

Read more: Biological sequence and structure models: reading proteins and DNA like text

The nine types in one table

Seen side by side, the types differ mostly in what they output.

Type Input Output Typical uses
Large language model Text Text Writing, summaries, translation, Q&A
Decision model Situation plus fixed options Probability per option Routing, classification, choosing
Embedding model Text or image Vector Semantic search, clustering, RAG retrieval
Reranker Question plus candidate Relevance score Re-ordering search results
Encoder classifier Text Label Spam, sentiment, named entities
Vision-language model Image plus text Text Visual Q&A, reading charts
Image generation model Text Image Illustrations, sketches
Speech model Audio or text Text or audio Captions, read-aloud
Biological sequence model Amino-acid or DNA sequence Structure or effect prediction Structure prediction, variant ranking

How to choose: start with the input, then decide what output you need

To pick a model, first look at the form of your data, then ask whether you need new text, a choice, a score, or a vector.

It works like planning a route: confirm the starting point, then pick the destination. If your data is audio, the first step is usually speech recognition. If it is an image and you want a written answer, use a vision-language model. If it is sequence data, use the matching biological sequence model. With the input settled, look at the output. If the answer can only be one of a few fixed options, use a decision model or an encoder classifier; the results are stable and easy to count. If you need to find things in a large document collection, retrieve with an embedding model and order with a reranker. Hand the job to an LLM only when you really need a new piece of writing. Many real systems chain several types: transcribe first, then retrieve, and generate last.

Common misconceptions: all AI is a chatbot, and bigger is always better

The most common misconception is that every AI model is a chatbot.

Everyday features such as search, recommendations, captions, and spam filtering are often handled by models that never write prose. Those models output scores, labels, vectors, or transcripts. The second misconception is that bigger is always better. Large models can do more kinds of tasks, but for jobs like classifying into fixed categories, a small specialized model is often faster, cheaper, and easier to audit. The third is that one model does everything. A product may show a single chat box while several kinds of models share the work behind it. Next time you evaluate an AI tool, start by asking what goes in and what comes out.

References

  1. https://arxiv.org/abs/1706.03762
  2. https://arxiv.org/abs/2005.14165
  3. https://arxiv.org/abs/2201.11903
  4. https://arxiv.org/abs/1908.10084
  5. https://arxiv.org/abs/2103.00020
  6. https://arxiv.org/abs/2005.11401
  7. https://arxiv.org/abs/1901.04085
  8. https://arxiv.org/abs/1810.04805
  9. https://arxiv.org/abs/2304.08485
  10. https://arxiv.org/abs/2006.11239
  11. https://arxiv.org/abs/2112.10752
  12. https://arxiv.org/abs/2212.04356
  13. https://arxiv.org/abs/1712.05884
  14. Lin et al. — Science — 2023
  15. Jumper et al. — Nature — 2021

Frequently Asked Questions

How is an embedding model different from a large language model?

An embedding model turns a passage of text or an image into a fixed-length list of numbers (a vector) so you can measure how similar two pieces of content are; it never writes a sentence. A large language model continues the text before it and outputs new writing. The two often work together: the embedding model finds relevant passages in a database, and the LLM answers from those passages. That combination is called RAG (retrieval-augmented generation).

Is it simpler to use one large language model for every task?

It can be simpler to build, but it is not always the better choice. An LLM generates its answer word by word, so it is slower and costs more, and its output can drift from the format you expected. When the answer can only be one of a few fixed options, a decision model or an encoder classifier is usually faster, more consistent, and easier to score. LLMs are strongest where new text is the goal, such as summaries, translation, and rewriting.

Can predictions from biological sequence models be treated as experimental results?

No. Predicted protein structures and predicted DNA variant effects are estimates learned from training data, and their reliability varies with the protein family and with how well similar sequences are represented in that data. They are best used to set priorities, helping researchers decide which candidates to test first. Structure and function still need experimental confirmation, and the predictions are not a basis for clinical diagnosis.

Found this useful? RSS · X · Buy me a coffee