Speaker diarization

The problem. A meeting room, a podcast, an interview: several voices on one microphone, one stream of text. You cannot tell what the client said from what the advisor said. Diarization labels each utterance and each word with a speaker number.

Where it works. Every model, on live audio.

Step by step

1. Turn it on

{ "diarization": true }

2. Start a session with it

Each program streams a 16-bit WAV file and prints what comes back. Put your API key in the GLADIA_API_KEY environment variable, then run python diarization.py meeting.wav.

# pip install gladiaio-sdk requests
import os
import sys
import threading
import wave

import requests
from gladiaio_sdk import GladiaClient, LiveV2ConnectSessionOptions

API_URL = os.environ.get("GLADIA_API_URL", "https://eu1.api.gladia.io")
API_KEY = os.environ["GLADIA_API_KEY"]

with wave.open(sys.argv[1]) as wav:  # a 16-bit PCM WAV file
    rate, channels = wav.getframerate(), wav.getnchannels()
    audio = wav.readframes(wav.getnframes())

config = {
    "encoding": "wav/pcm",
    "sample_rate": rate,
    "bit_depth": 16,
    "channels": channels,
    "model": "solaria-3",
    "language_config": {
        "languages": [
            "en"
        ]
    },
    "diarization": True,
    "diarization_config": {
        "max_speakers": 3
    }
}

client = GladiaClient(api_key=API_KEY, api_url=API_URL)
# `diarization` is a Gladia Enterprise option the Python SDK does not send yet
# (gladiaio/sdk#61): create the session over HTTP, then let the SDK stream it.
created = requests.post(f"{API_URL}/v2/live", headers={"x-gladia-key": API_KEY}, json=config, timeout=10)
created.raise_for_status()
info = created.json()
session = client.live_v2().connect_session(LiveV2ConnectSessionOptions(id=info["id"], url=info["url"]))
ended, errors = threading.Event(), []

@session.on("message")
def on_message(message):
    if message.type == "transcript" and message.data.is_final:
        utterance = message.data.utterance
        print(f"Speaker {utterance.speaker}: {utterance.text}")

session.on("error", errors.append)
session.once("ended", lambda _: ended.set())

step = rate * channels * 2 // 10  # 100 ms of audio per chunk
for start in range(0, len(audio), step):
    session.send_audio(audio[start : start + step])  # live audio: send each chunk as it is captured
session.stop_recording()

ended.wait(timeout=60)
sys.exit(1 if errors or not ended.is_set() else 0)

// npm install @gladiaio/sdk
import { readFileSync } from "node:fs";
import { GladiaClient } from "@gladiaio/sdk";

const { sampleRate, channels, pcm } = readWav(process.argv[2]); // a 16-bit PCM WAV file

const config = {
  encoding: "wav/pcm",
  sample_rate: sampleRate,
  bit_depth: 16,
  channels,
  model: "solaria-3",
  language_config: {
    languages: [
      "en"
    ]
  },
  diarization: true,
  diarization_config: {
    max_speakers: 3
  }
};

const client = new GladiaClient({
  apiKey: process.env.GLADIA_API_KEY,
  apiUrl: process.env.GLADIA_API_URL ?? "https://eu1.api.gladia.io",
});
// `diarization` is a Gladia Enterprise option not in the SDK's types yet: the SDK sends it as is.
const session = client.liveV2().startSession(config);

session.on("message", (message) => {
  if (message.type === "transcript" && message.data.is_final) {
    const { speaker, text } = message.data.utterance;
    console.log(`Speaker ${speaker}: ${text}`);
  }
});
session.on("error", (error) => {
  console.error(error);
  process.exitCode = 1;
});
session.once("ended", () => process.exit());

const step = (sampleRate * channels * 2) / 10; // 100 ms of audio per chunk
for (let start = 0; start < pcm.length; start += step) {
  session.sendAudio(pcm.subarray(start, start + step)); // live audio: send each chunk as it is captured
}
session.stopRecording();

function readWav(path) {
  const file = readFileSync(path);
  let fmt, pcm;
  for (let pos = 12; pos < file.length; ) {
    const id = file.toString("ascii", pos, pos + 4);
    const size = file.readUInt32LE(pos + 4);
    if (id === "fmt ") fmt = { channels: file.readUInt16LE(pos + 10), sampleRate: file.readUInt32LE(pos + 12) };
    if (id === "data") pcm = file.subarray(pos + 8, pos + 8 + size);
    pos += 8 + size + (size % 2);
  }
  return { ...fmt, pcm };
}

Create the session, then stream its audio to the returned url (Live WebSocket):

curl -X POST https://eu1.api.gladia.io/v2/live \
  -H "x-gladia-key: $GLADIA_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "encoding": "wav/pcm",
    "sample_rate": 16000,
    "bit_depth": 16,
    "channels": 1,
    "model": "solaria-3",
    "language_config": {
      "languages": [
        "en"
      ]
    },
    "diarization": true,
    "diarization_config": {
      "max_speakers": 3
    }
  }'

3. Read the speakers

{
  "type": "transcript",
  "data": {
    "id": "00_00000004",
    "is_final": true,
    "utterance": {
      "text": "Right. Shall we start?",
      "start": 31.2,
      "end": 32.6,
      "speaker": 1,
      "channel": 0,
      "words": [
        { "word": "Right.", "start": 31.2, "end": 31.5, "confidence": 0.97, "speaker": 0 },
        { "word": "Shall", "start": 31.7, "end": 31.9, "confidence": 0.98, "speaker": 1 },
        { "word": "we", "start": 31.9, "end": 32.0, "confidence": 0.99, "speaker": 1 },
        { "word": "start?", "start": 32.0, "end": 32.6, "confidence": 0.98, "speaker": 1 }
      ]
    }
  }
}

Read the speaker of each word, not the utterance’s. An utterance ends on a pause, not on a change of speaker, so one utterance can hold the words of two people: here speaker 0 says “Right.” and speaker 1 asks the question. The utterance’s speaker is only the one who speaks longest in it. Each word’s speaker is the label to trust.

4. Group the words by speaker

To show who said what, cut each final where its words change speaker:

def turns(utterance):
    """The utterance's words as (speaker, text) turns, in order."""
    out = []
    for word in utterance["words"]:
        if out and out[-1][0] == word["speaker"]:
            out[-1] = (word["speaker"], out[-1][1] + " " + word["word"])
        else:
            out.append((word["speaker"], word["word"]))
    return out

A label can also be wrong on one or two words (see the limits below). If short flips bother your display more than lost interjections, merge them client-side: give a run of one or two words the speaker of the runs on both sides of it when both are the same speaker. This also hides real one-word answers (“yes”, “okay”) from that speaker.

def merge_islands(words, max_words=2):
    """Relabel a run of at most max_words words framed by one same speaker on both sides."""
    words = [dict(w) for w in words]
    runs, start = [], 0  # (first, last + 1) index of each run of one speaker
    for i in range(1, len(words) + 1):
        if i == len(words) or words[i]["speaker"] != words[start]["speaker"]:
            runs.append((start, i))
            start = i
    for (a, _), (b, e), (c, _) in zip(runs, runs[1:], runs[2:]):
        if e - b <= max_words and words[a]["speaker"] == words[c]["speaker"]:
            for w in words[b:e]:
                w["speaker"] = words[a]["speaker"]
    return words

Run it on the words of consecutive finals put together, so a flip on the first word of a final is seen between the previous final and the rest of this one.

Options

option	what it does
`max_speakers`	The most speakers to tell apart, when you know it.
`number_of_speakers`	Gladia’s name for the same setting.

Limits and gotchas

Up to 4 speakers are told apart in one session. More voices than that share the 4 numbers.
Finals arrive a little later, the time the speakers are known (under a second). Partials are not labelled.
Numbers, not names. Speaker numbers are valid within the session only: the same person in two sessions may get another number. Map numbers to people in your application.
Separate channels are better. When each person has their own channel (a call recorded in stereo), send the channels instead (Audio formats): it is exact, and needs no diarization.
Overlapping speech is one utterance: two people talking at once are not split into two.
One utterance, several speakers. Utterances are not cut where the speaker changes: a quick reply right after another person spoke often lands in the same final. Read each word’s speaker (step 3).
One or two words can flip to another speaker. The speaker is chosen word by word, and a short word over another voice can take the wrong label. On Gladia’s diarization quality study (19 recordings, 1.6 h, 659 finals), the first word’s speaker differs from the utterance’s speaker in 17.3 % of the finals (114). In 55 of them the final really starts with another person’s words. In the other 59 the first words carry a wrong label, and in 46 of those the wrong run is one or two words long. Counting every place in a final, there are 0.276 runs of one or two words between two runs of one same speaker per final, real short turns included.
No smoothing by default. The inference has an optional server-side smoothing that gives a run of up to two words back to the speaker around it. It is off because it also erases real short turns. On the same study it relabelled 65 runs: 38 wrong labels fixed, 27 real interjections lost. On the meeting recordings, the share of speaker changes found fell from 67.8 % to 59.5 %. Merge short runs client-side (step 4) when you want that trade-off.
min_speakers and enhanced (Gladia’s cloud fields) are accepted and ignored.

Troubleshooting

symptom	cause	fix
everyone is speaker 0	the voices are very similar, or one microphone picks one voice much louder	use one channel per person when you can
5 people, only 4 numbers	the 4-speaker limit	split the session by channel
finals feel slower with diarization	they wait for their speakers	expected: use partials for live display
the first word or two of an utterance belong to someone else	the reply starts in the same final, or a short label flip	group by each word’s `speaker`; merge one- or two-word runs client-side if you prefer stable turns (step 4)