# Speaker diarization

**The problem.** A meeting room, a podcast, an interview: several voices on one microphone, one stream of text. You
cannot tell what the client said from what the advisor said. Diarization labels each utterance and each word with a
`speaker` number.

**Where it works.** Every model, on live audio.

> **Use the SDK.** The Gladia SDKs stream the audio, keep it until the API has transcribed it, and reconnect after a network drop
without losing a word. Most integration issues come from hand-written WebSocket clients: start from the SDK
examples below.

## Step by step

### 1. Turn it on

```json
{ "diarization": true }
```

### 2. Start a session with it

Each program streams a 16-bit WAV file and prints what comes back. Put your API key in the `GLADIA_API_KEY` environment variable, then run `python diarization.py meeting.wav`.

**Python SDK**

```python title="diarization.py"
# pip install gladiaio-sdk requests
import os
import sys
import threading
import wave

import requests
from gladiaio_sdk import GladiaClient, LiveV2ConnectSessionOptions

API_URL = os.environ.get("GLADIA_API_URL", "https://eu1.api.gladia.io")
API_KEY = os.environ["GLADIA_API_KEY"]

with wave.open(sys.argv[1]) as wav:  # a 16-bit PCM WAV file
    rate, channels = wav.getframerate(), wav.getnchannels()
    audio = wav.readframes(wav.getnframes())

config = {
    "encoding": "wav/pcm",
    "sample_rate": rate,
    "bit_depth": 16,
    "channels": channels,
    "model": "solaria-3",
    "language_config": {
        "languages": [
            "en"
        ]
    },
    "diarization": True,
    "diarization_config": {
        "max_speakers": 3
    }
}

client = GladiaClient(api_key=API_KEY, api_url=API_URL)
# `diarization` is a Gladia Enterprise option the Python SDK does not send yet
# (gladiaio/sdk#61): create the session over HTTP, then let the SDK stream it.
created = requests.post(f"{API_URL}/v2/live", headers={"x-gladia-key": API_KEY}, json=config, timeout=10)
created.raise_for_status()
info = created.json()
session = client.live_v2().connect_session(LiveV2ConnectSessionOptions(id=info["id"], url=info["url"]))
ended, errors = threading.Event(), []

@session.on("message")
def on_message(message):
    if message.type == "transcript" and message.data.is_final:
        utterance = message.data.utterance
        print(f"Speaker {utterance.speaker}: {utterance.text}")

session.on("error", errors.append)
session.once("ended", lambda _: ended.set())

step = rate * channels * 2 // 10  # 100 ms of audio per chunk
for start in range(0, len(audio), step):
    session.send_audio(audio[start : start + step])  # live audio: send each chunk as it is captured
session.stop_recording()

ended.wait(timeout=60)
sys.exit(1 if errors or not ended.is_set() else 0)
```

**JavaScript SDK**

```js title="diarization.mjs"
// npm install @gladiaio/sdk
import { readFileSync } from "node:fs";
import { GladiaClient } from "@gladiaio/sdk";

const { sampleRate, channels, pcm } = readWav(process.argv[2]); // a 16-bit PCM WAV file

const config = {
  encoding: "wav/pcm",
  sample_rate: sampleRate,
  bit_depth: 16,
  channels,
  model: "solaria-3",
  language_config: {
    languages: [
      "en"
    ]
  },
  diarization: true,
  diarization_config: {
    max_speakers: 3
  }
};

const client = new GladiaClient({
  apiKey: process.env.GLADIA_API_KEY,
  apiUrl: process.env.GLADIA_API_URL ?? "https://eu1.api.gladia.io",
});
// `diarization` is a Gladia Enterprise option not in the SDK's types yet: the SDK sends it as is.
const session = client.liveV2().startSession(config);

session.on("message", (message) => {
  if (message.type === "transcript" && message.data.is_final) {
    const { speaker, text } = message.data.utterance;
    console.log(`Speaker ${speaker}: ${text}`);
  }
});
session.on("error", (error) => {
  console.error(error);
  process.exitCode = 1;
});
session.once("ended", () => process.exit());

const step = (sampleRate * channels * 2) / 10; // 100 ms of audio per chunk
for (let start = 0; start < pcm.length; start += step) {
  session.sendAudio(pcm.subarray(start, start + step)); // live audio: send each chunk as it is captured
}
session.stopRecording();

function readWav(path) {
  const file = readFileSync(path);
  let fmt, pcm;
  for (let pos = 12; pos < file.length; ) {
    const id = file.toString("ascii", pos, pos + 4);
    const size = file.readUInt32LE(pos + 4);
    if (id === "fmt ") fmt = { channels: file.readUInt16LE(pos + 10), sampleRate: file.readUInt32LE(pos + 12) };
    if (id === "data") pcm = file.subarray(pos + 8, pos + 8 + size);
    pos += 8 + size + (size % 2);
  }
  return { ...fmt, pcm };
}
```

**HTTP**

Create the session, then stream its audio to the returned `url` ([Live WebSocket](/api-reference/v2/live/websocket/)):

```bash
curl -X POST https://eu1.api.gladia.io/v2/live \
  -H "x-gladia-key: $GLADIA_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "encoding": "wav/pcm",
    "sample_rate": 16000,
    "bit_depth": 16,
    "channels": 1,
    "model": "solaria-3",
    "language_config": {
      "languages": [
        "en"
      ]
    },
    "diarization": true,
    "diarization_config": {
      "max_speakers": 3
    }
  }'
```

### 3. Read the speakers

```json
{
  "type": "transcript",
  "data": {
    "id": "00_00000004",
    "is_final": true,
    "utterance": {
      "text": "Right. Shall we start?",
      "start": 31.2,
      "end": 32.6,
      "speaker": 1,
      "channel": 0,
      "words": [
        { "word": "Right.", "start": 31.2, "end": 31.5, "confidence": 0.97, "speaker": 0 },
        { "word": "Shall", "start": 31.7, "end": 31.9, "confidence": 0.98, "speaker": 1 },
        { "word": "we", "start": 31.9, "end": 32.0, "confidence": 0.99, "speaker": 1 },
        { "word": "start?", "start": 32.0, "end": 32.6, "confidence": 0.98, "speaker": 1 }
      ]
    }
  }
}
```

**Read the speaker of each word, not the utterance's.** An utterance ends on a pause, not on a change of speaker, so
one utterance can hold the words of two people: here speaker 0 says "Right." and speaker 1 asks the question. The
utterance's `speaker` is only the one who speaks longest in it. Each word's `speaker` is the label to trust.

### 4. Group the words by speaker

To show who said what, cut each final where its words change speaker:

```python
def turns(utterance):
    """The utterance's words as (speaker, text) turns, in order."""
    out = []
    for word in utterance["words"]:
        if out and out[-1][0] == word["speaker"]:
            out[-1] = (word["speaker"], out[-1][1] + " " + word["word"])
        else:
            out.append((word["speaker"], word["word"]))
    return out
```

A label can also be wrong on one or two words (see the limits below). If short flips bother your display more than
lost interjections, merge them client-side: give a run of one or two words the speaker of the runs on both sides of it
when both are the same speaker. This also hides real one-word answers ("yes", "okay") from that speaker.

```python
def merge_islands(words, max_words=2):
    """Relabel a run of at most max_words words framed by one same speaker on both sides."""
    words = [dict(w) for w in words]
    runs, start = [], 0  # (first, last + 1) index of each run of one speaker
    for i in range(1, len(words) + 1):
        if i == len(words) or words[i]["speaker"] != words[start]["speaker"]:
            runs.append((start, i))
            start = i
    for (a, _), (b, e), (c, _) in zip(runs, runs[1:], runs[2:]):
        if e - b <= max_words and words[a]["speaker"] == words[c]["speaker"]:
            for w in words[b:e]:
                w["speaker"] = words[a]["speaker"]
    return words
```

Run it on the words of consecutive finals put together, so a flip on the first word of a final is seen between the
previous final and the rest of this one.

## Options

| option | what it does |
|---|---|
| `max_speakers` | The most speakers to tell apart, when you know it. |
| `number_of_speakers` | Gladia's name for the same setting. |

## Limits and gotchas

- **Up to 4 speakers** are told apart in one session. More voices than that share the 4 numbers.
- **Finals arrive a little later**, the time the speakers are known (under a second). Partials are not labelled.
- **Numbers, not names.** Speaker numbers are valid within the session only: the same person in two sessions may get
  another number. Map numbers to people in your application.
- **Separate channels are better.** When each person has their own channel (a call recorded in stereo), send the
  channels instead ([Audio formats](/live/audio-formats/#several-channels)): it is exact, and needs no diarization.
- **Overlapping speech** is one utterance: two people talking at once are not split into two.
- **One utterance, several speakers.** Utterances are not cut where the speaker changes: a quick reply right after
  another person spoke often lands in the same final. Read each word's `speaker` (step 3).
- **One or two words can flip to another speaker.** The speaker is chosen word by word, and a short word over another
  voice can take the wrong label. On Gladia's diarization quality study (19 recordings, 1.6 h, 659 finals), the first
  word's speaker differs from the utterance's `speaker` in 17.3 % of the finals (114). In 55 of them the final really
  starts with another person's words. In the other 59 the first words carry a wrong label, and in 46 of those the
  wrong run is one or two words long. Counting every place in a final, there are 0.276 runs of one or two words
  between two runs of one same speaker per final, real short turns included.
- **No smoothing by default.** The inference has an optional server-side smoothing that gives a run of up to two words
  back to the speaker around it. It is off because it also erases real short turns. On the same study it relabelled 65
  runs: 38 wrong labels fixed, 27 real interjections lost. On the meeting recordings, the share of speaker changes found
  fell from 67.8 % to 59.5 %. Merge short runs client-side (step 4) when you want that trade-off.
- `min_speakers` and `enhanced` (Gladia's cloud fields) are accepted and ignored.

## Troubleshooting

| symptom | cause | fix |
|---|---|---|
| everyone is speaker 0 | the voices are very similar, or one microphone picks one voice much louder | use one channel per person when you can |
| 5 people, only 4 numbers | the 4-speaker limit | split the session by channel |
| finals feel slower with diarization | they wait for their speakers | expected: use partials for live display |
| the first word or two of an utterance belong to someone else | the reply starts in the same final, or a short label flip | group by each word's `speaker`; merge one- or two-word runs client-side if you prefer stable turns (step 4) |