Skip to main content
Lab Grimoire
TW EN
Coffee
LLM Internals

How speech models work: recognition turns sound into text, synthesis turns text into sound

Speech models run in two directions: speech recognition (ASR) turns a recording into a transcript, and text-to-speech (TTS) reads text aloud. Using Whisper and Tacotron 2, this piece explains the spectrogram step in the middle, what the models are good for, and where they fail: jargon, noise, overlapping speakers, and sentences that were never said.

Author
CY
Published
Updated
AI from Principles to Practice · Part 8 of 70
How speech models work: recognition turns sound into text, synthesis turns text into sound

Two directions: listening and speaking

Speech models come in two directions. Speech recognition (ASR) takes in audio and returns a transcript. Text-to-speech (TTS) takes in text and returns spoken audio.

On a map of AI model types, speech models sit at the border between sound and text. Dictation on your phone, automatic captions on a video, and the voice in your navigation app all rely on them. This piece walks through how each direction works, what it is good for, and where it tends to go wrong.

Summary: speech models

Speech recognition: first, turn the sound into a picture

A speech recognition model starts by converting the sound into a spectrogram: time runs along the horizontal axis, frequency up the vertical axis, and the shading shows how strong each frequency is at each moment.

The model reads that picture and writes out text, segment by segment. Whisper is a well-known example. Its authors trained it on about 680,000 hours of audio paired with transcripts collected from the web, covering many languages, so it handles a range of accents and recording conditions without being retrained for each setting. It has learned which sound patterns usually go with which words. Its output is the most likely text, not a guaranteed word-for-word match of the recording.

Text-to-speech: the same road, driven the other way

Text-to-speech runs in reverse: text becomes a spectrogram, and the spectrogram becomes a waveform you can play.

Tacotron 2 is a common reference point. It works in two stages. The first model reads the text and predicts the matching spectrogram. The second is a vocoder, which turns that spectrogram into individual audio samples. Tacotron 2 uses a modified WaveNet as its vocoder, a model that generates a sound wave one sample at a time. Voice assistants often chain three kinds of model together: speech recognition works out what you said, a large language model drafts a reply, and text-to-speech reads it back to you.

What they are good for: meeting notes, captions, reading aloud

Speech recognition is useful whenever a lot of spoken material needs to become text you can search and edit: meeting notes, interview transcripts, captions, dictated memos. Take an hour-long research meeting. Run the recording through speech recognition, hand the transcript to a large language model to pull out the main points, and a person only has to check the result. Text-to-speech covers the other direction, such as reading web pages aloud for people with low vision or producing an audio version of a long article.

Where they go wrong: names, noise, and sentences nobody said

Specialist vocabulary causes the most errors. Drug names, gene names, and personal names are rare in training data, so the model often swaps in a common word that sounds similar. Heavy accents, background noise, and people talking over each other also push the error rate up. A transcript does not say who spoke which line, either. Separating speakers is a distinct step called speaker diarization.

A less obvious risk is that speech recognition can produce fluent sentences that are not in the recording at all. In a study published in 2024, Koenecke and colleagues examined Whisper transcripts and found that roughly 1% contained whole phrases or sentences that had been invented, more often in recordings with long pauses. They tested the version available at the time, and later versions may behave differently. Before you quote anyone from a transcript, go back to the audio. On the synthesis side, these systems can imitate a specific person's voice, so get that person's consent before you synthesize speech in their voice.

One practical step helps with the vocabulary problem: if your tool accepts a prompt or a custom word list, put the names and terms you expect into it. OpenAI's speech-to-text guide recommends a prompt for exactly this, to improve recognition of names, acronyms, and technical terms.

How they divide the work with their neighbours

Once you have a transcript, the next step usually belongs to a different kind of model. For summaries and rewrites, use a large language model. To tag each passage with a fixed label such as "decision" or "action item", or to pick out every drug name, an encoder classifier or sequence tagger is faster and cheaper. Vision-language models play a similar role to speech models, bringing non-text input into text, but their way in is an image. When choosing, start with the data you have. If it is a recording, the first step is speech recognition.

References

  1. https://arxiv.org/abs/2212.04356
  2. https://arxiv.org/abs/1712.05884
  3. https://arxiv.org/abs/1609.03499
  4. https://arxiv.org/abs/2402.08021
  5. https://developers.openai.com/api/docs/guides/speech-to-text

Frequently Asked Questions

Can I quote directly from a speech recognition transcript?

Check the recording first. A speech recognition model outputs the text it judges most likely, so specialist terms can come out as similar-sounding common words, and whole sentences that were never spoken can appear. A 2024 study of Whisper transcripts found that roughly 1% contained invented phrases or sentences, more often in recordings with long pauses; it measured the version available then, and later versions may differ. A transcript is fine as a working draft for notes or captions, but when you quote what someone said, the original audio is the authority.

Why can't a transcript tell me who was speaking?

A speech recognition model converts sound into words; identifying the speaker is a separate job. Labelling which person said each line takes an additional step called speaker diarization. Some tools bundle the two steps together, but both become less reliable when people talk over each other, so important exchanges still need a human check.

Found this useful? RSS · X · Buy me a coffee