Skip to main content
Lab Grimoire
TW EN
Coffee
LLM Internals

Encoder classifiers and sequence labeling: one label per text, or one label per word

An encoder classifier reads a passage and outputs one label; sequence labeling tags every word, for example marking gene, drug and disease names in abstracts. How BERT-style encoders work, what BioBERT and PubMedBERT do, where they fall short, and when to pick one over a large language model.

Author
CY
Published
Updated
AI from Principles to Practice · Part 5 of 70
Encoder classifiers and sequence labeling: one label per text, or one label per word

Text goes in, a label comes out

An encoder classifier reads a passage of text and outputs one label. Sequence labeling gives a label to every word in the passage instead.

Say you have three thousand paper abstracts. First you want to keep only the ones relevant to your question, and then you want every gene, drug and disease name in them marked. The first job is classification: each abstract gets one label, relevant or not. Sorting reviews by sentiment or studies by design is the same kind of job. The second job is sequence labeling, and its most common form is named entity recognition (NER): the model goes word by word and decides whether each word starts a gene name, continues one, or belongs to no name at all. On a map of AI model types, this family takes text in and returns labels you defined in advance. It does not write new sentences.

Encoder classifiers and sequence labeling at a glance

How it works: read the whole sentence, then add a small classifier

These models have two parts. An encoder turns the sentence into a set of numerical representations, and a small classification head on top picks the label from them.

The classic example is BERT, published in 2018. It is pre-trained on a large amount of text by hiding some words in each sentence and asking the model to recover them from the words on both sides. Each word's representation ends up carrying information from its left and right context. For a specific task you add one output layer on top of the encoder and keep training on a set of examples that people have labeled. That step is called fine-tuning. For classification, the head looks at a representation of the whole passage and returns one label. For sequence labeling, it looks at each word's representation and returns a label per word.

Reading in both directions was already standard in neural NER before BERT; Lample and colleagues used it in 2016. Whether a word is part of a drug name depends on what comes before and after it.

Biomedical text: BioBERT and PubMedBERT

Biomedical writing is dense with technical names and abbreviations, and encoders pre-trained on biomedical literature usually handle it better than general-purpose ones.

BioBERT took BERT and continued pre-training it on PubMed abstracts and PMC full-text articles. Its authors report that it beat the original BERT on biomedical named entity recognition, relation extraction and question answering. PubMedBERT was pre-trained from scratch on PubMed text only. After comparing the two strategies, its authors concluded that in a field with this much text, training from scratch works better than continuing from a general model. In practice, a model like this can mark genes, chemicals and diseases across a large batch of abstracts so the results can be put into a searchable table.

Strengths and limits

These models are small, fast and cheap, and their output always has the same format. The trade-off is that every task needs labeled data, and the model only knows the labels it was trained on.

They are usually far smaller than large language models, so they can run on an ordinary computer and your data never has to leave it. Because the output can only be one of the predefined labels, you can count right and wrong answers directly. The cost is flexibility. Adding a new category such as "clinical trial" means labeling more examples and fine-tuning again. When the text you feed in looks very different from the training data, for example clinical notes instead of journal abstracts, accuracy can drop, so the model should be evaluated again on the new material.

When to pick it instead of asking a large language model

When the same task repeats thousands of times, the labels are fixed and you have labeled examples, a fine-tuned encoder is usually the more economical choice.

A large language model needs no extra training: describe the categories in the prompt and it will classify. That makes it a good fit for a first trial, or while the categories are still changing. It writes its answer word by word, though, so it is slower, and asking twice about the same passage can produce different wording. A decision model also picks from fixed options: it reads a situation and a list of choices and gives each choice a probability, and some decision models are themselves an encoder with a classification head. An embedding model also reads text with an encoder, but it outputs a list of numbers that stands for meaning, which you use to compare similarity rather than to assign a label.

References

  1. https://arxiv.org/abs/1810.04805
  2. https://arxiv.org/abs/1901.08746
  3. https://arxiv.org/abs/2007.15779
  4. https://arxiv.org/abs/1603.01360

Frequently Asked Questions

A large language model can classify without any training. Why fine-tune an encoder at all?

They suit different stages. While the categories are still changing, or you only want a quick trial, describing the categories in a prompt and letting a large language model classify is the easiest route. Once the categories are fixed and the same task repeats thousands of times, a fine-tuned encoder is usually more economical: it is small enough to run on an ordinary computer and fast, its output is always one of the predefined labels, and the same passage gets the same answer every time. The price is that you need a set of human-labeled examples first.

BioBERT was trained on paper abstracts. Can it find drug names in clinical notes as is?

Do not assume it can. BioBERT and PubMedBERT were pre-trained on published literature, and clinical notes read very differently, with heavy abbreviation, shorthand and typos, so performance on abstracts may not carry over. A safer approach is to label a small batch of notes as a test set, measure accuracy, and only then decide whether to use the model, fine-tuning again on clinical text if needed. Clinical notes are personal data, so the relevant privacy rules apply as well.

Found this useful? RSS · X · Buy me a coffee