What Is Speaker Diarization? How AI Works Out Who Said What
Speaker diarization splits a recording by voice so a transcript can say who said what. How it works, why a text-only AI can't do it, where it slips, and how accurate it gets — measured.
Short answer
Speaker diarization is the step that splits a recording by voice and marks when each person speaks, so a transcript can say who said what — Speaker 1, Speaker 2. It works from the sound of each voice, not the words, which is why a text-only AI can't do it well. Measured against human annotation, a modern diarization model labelled 96–98% of meeting speech correctly.
Speaker diarization answers one question about a recording: who spoke when? It splits the audio into stretches that each belong to one voice and gives every voice a label — Speaker 1, Speaker 2 — so a transcript can show who said what. The name comes from "diary": a log of each speaker's turns.
It is a separate job from transcription. A speech-to-text model such as Whisper writes the words and their timestamps; a diarization model listens to the same audio and decides where one voice ends and the next begins. A transcript with speaker labels lays the two on top of each other by time.
How It Works
There are two broad ways to build it.
- Segment, fingerprint, cluster. The audio is cut into short pieces of speech, each piece is turned into a "voiceprint" (a list of numbers that is similar for the same voice), and the pieces are grouped so that each group is one person. It works, but the grouping needs a threshold: set it too loose and two people merge into one; too tight and one person splits into several.
- End to end. A single model is trained to output, for every moment, which of its speaker slots is talking. It needs no threshold and handles turn-taking more smoothly. NVIDIA's Nemotron-3-Diarization, the model SubGrab uses, works this way and tracks up to eight voices.
Either way, the output is a list of time spans, each with a speaker. Each line of the transcript then takes the voice heard most during it.
Why a Text-Only AI Can't Do It
Who is speaking is in the voice, not the words. A language model that only reads the transcript has to guess from phrasing — a question followed by an answer is probably two people — and on a real meeting that guess is no better than saying everyone is one speaker.
We measured exactly that on a four-person meeting from the AMI Meeting Corpus, whose speakers were marked by hand. Accuracy is the share of speech time given to the right person; labelling everything "Speaker 1" scores 48% on it.
| Approach | Listens to the audio? | Meeting accuracy |
|---|---|---|
| Text-only language models (two tested) | No | 41–47% |
| A general AI model asked to listen and label turns | Yes | 79–94% |
| Segment-and-cluster with voiceprints | Yes | 93%, but only with the threshold tuned on that same meeting |
| End-to-end diarization model (Nemotron-3-Diarization) | Yes | 94–98% |
The last row held on four more AMI meetings that nothing was tuned on: 96–98% each, and the right number of speakers every time.
How Accuracy Is Measured
Research papers report the diarization error rate (DER): missed speech, speech given to someone when nobody spoke, and speech given to the wrong person, added up as a share of all speech. pyannote.metrics documents the standard definition. For a transcript the question is simpler — is each line labelled with the right person? — so the figures on this page are the share of speech time labelled correctly. Both reward the same thing.
Where It Slips
- Crosstalk. When two people talk at once, a line can only carry one label.
- Very short turns. A one-word "yeah" from the listener is easy to fold into the main speaker's turn.
- Similar voices. Two people of similar age and accent on the same microphone are the hardest pair.
- Audio quality. A distant laptop microphone in an echoing room loses the detail that tells voices apart.
- More than eight speakers, for a model with eight slots: some people will share a number.
Getting Speaker Labels on a Transcript
In SubGrab, turn on Speaker labels before an AI transcript starts — 2 credits per started hour on top of the transcription. If only one voice is heard, those credits come back automatically. The labels appear in the transcript and in the TXT, SRT and VTT downloads. For the numbers behind them, see how accurate speaker labels are; to swap the numbers for names, see how to replace Speaker 1 and Speaker 2 with real names.
Try it on an interview — every new account gets 2 free credits.
Frequently asked
Is speaker diarization the same as speech-to-text?
No. Speech-to-text writes down the words; diarization works out who is speaking when. A transcript with speaker labels uses both: one model for the words and timestamps, another for the voices, matched up by time.
Can diarization tell me the speakers' names?
No. It tells voices apart and numbers them in the order they first speak. Matching a number to a name is a human step — see how to replace Speaker 1 and Speaker 2 with real names.
What makes diarization less accurate?
People talking over each other, very short interjections, voices that sound alike, and poor or distant microphones. Two people taking turns on clean audio is the easy case; a busy meeting is the hard one.
How many speakers can it handle?
It depends on the model. The one SubGrab uses tracks up to eight voices; on a larger panel some people will share a number.