Two directions: listening and speaking
Speech models come in two directions. Speech recognition (ASR) takes in audio and returns a transcript. Text-to-speech (TTS) takes in text and returns spoken audio.
On a map of AI model types, speech models sit at the border between sound and text. Dictation on your phone, automatic captions on a video, and the voice in your navigation app all rely on them. This piece walks through how each direction works, what it is good for, and where it tends to go wrong.

Speech recognition: first, turn the sound into a picture
A speech recognition model starts by converting the sound into a spectrogram: time runs along the horizontal axis, frequency up the vertical axis, and the shading shows how strong each frequency is at each moment.
The model reads that picture and writes out text, segment by segment. Whisper is a well-known example. Its authors trained it on about 680,000 hours of audio paired with transcripts collected from the web, covering many languages, so it handles a range of accents and recording conditions without being retrained for each setting. It has learned which sound patterns usually go with which words. Its output is the most likely text, not a guaranteed word-for-word match of the recording.
Text-to-speech: the same road, driven the other way
Text-to-speech runs in reverse: text becomes a spectrogram, and the spectrogram becomes a waveform you can play.
Tacotron 2 is a common reference point. It works in two stages. The first model reads the text and predicts the matching spectrogram. The second is a vocoder, which turns that spectrogram into individual audio samples. Tacotron 2 uses a modified WaveNet as its vocoder, a model that generates a sound wave one sample at a time. Voice assistants often chain three kinds of model together: speech recognition works out what you said, a large language model drafts a reply, and text-to-speech reads it back to you.
What they are good for: meeting notes, captions, reading aloud
Speech recognition is useful whenever a lot of spoken material needs to become text you can search and edit: meeting notes, interview transcripts, captions, dictated memos. Take an hour-long research meeting. Run the recording through speech recognition, hand the transcript to a large language model to pull out the main points, and a person only has to check the result. Text-to-speech covers the other direction, such as reading web pages aloud for people with low vision or producing an audio version of a long article.
Where they go wrong: names, noise, and sentences nobody said
Specialist vocabulary causes the most errors. Drug names, gene names, and personal names are rare in training data, so the model often swaps in a common word that sounds similar. Heavy accents, background noise, and people talking over each other also push the error rate up. A transcript does not say who spoke which line, either. Separating speakers is a distinct step called speaker diarization.
A less obvious risk is that speech recognition can produce fluent sentences that are not in the recording at all. In a study published in 2024, Koenecke and colleagues examined Whisper transcripts and found that roughly 1% contained whole phrases or sentences that had been invented, more often in recordings with long pauses. They tested the version available at the time, and later versions may behave differently. Before you quote anyone from a transcript, go back to the audio. On the synthesis side, these systems can imitate a specific person's voice, so get that person's consent before you synthesize speech in their voice.
One practical step helps with the vocabulary problem: if your tool accepts a prompt or a custom word list, put the names and terms you expect into it. OpenAI's speech-to-text guide recommends a prompt for exactly this, to improve recognition of names, acronyms, and technical terms.
How they divide the work with their neighbours
Once you have a transcript, the next step usually belongs to a different kind of model. For summaries and rewrites, use a large language model. To tag each passage with a fixed label such as "decision" or "action item", or to pick out every drug name, an encoder classifier or sequence tagger is faster and cheaper. Vision-language models play a similar role to speech models, bringing non-text input into text, but their way in is an image. When choosing, start with the data you have. If it is a recording, the first step is speech recognition.