End of turn

A voice agent must answer when the caller has finished, not at every pause: “my number is… zero six…” has pauses, and is not finished. End-of-turn detection listens to how the speaker talks at each pause and sends an end_of_turn message when their turn is over.

Turn it on in the init body:

{ "end_of_turn": true }

Then answer when an end_of_turn message arrives. Each program prints the caller’s sentences and each end of turn:

JavaScript SDK
Python

// npm install @gladiaio/sdk
import { readFileSync } from "node:fs";
import { GladiaClient } from "@gladiaio/sdk";

const { sampleRate, channels, pcm } = readWav(process.argv[2]); // a 16-bit PCM WAV file

const config = {
  encoding: "wav/pcm",
  sample_rate: sampleRate,
  bit_depth: 16,
  channels,
  model: "trantoria-1",
  language_config: {
    languages: [
      "en"
    ]
  },
  endpointing: 0.3,
  end_of_turn: {
    threshold: 0.8,
    stop_s: 1.2
  },
  messages_config: {
    receive_partial_transcripts: true
  }
};

const client = new GladiaClient({
  apiKey: process.env.GLADIA_API_KEY,
  apiUrl: process.env.GLADIA_API_URL ?? "https://eu1.api.gladia.io",
});
// `end_of_turn` is a Gladia Enterprise option not in the SDK's types yet: the SDK sends it as is.
const session = client.liveV2().startSession(config);

session.on("message", (message) => {
  if (message.type === "transcript" && message.data.is_final) console.log("caller:", message.data.utterance.text);
  if (message.type === "end_of_turn") console.log(`end of turn (${message.data.reason}): answer now`);
});
session.on("error", (error) => {
  console.error(error);
  process.exitCode = 1;
});
session.once("ended", () => process.exit());

const step = (sampleRate * channels * 2) / 10; // 100 ms of audio per chunk
for (let start = 0; start < pcm.length; start += step) {
  session.sendAudio(pcm.subarray(start, start + step)); // live audio: send each chunk as it is captured
}
session.stopRecording();

function readWav(path) {
  const file = readFileSync(path);
  let fmt, pcm;
  for (let pos = 12; pos < file.length; ) {
    const id = file.toString("ascii", pos, pos + 4);
    const size = file.readUInt32LE(pos + 4);
    if (id === "fmt ") fmt = { channels: file.readUInt16LE(pos + 10), sampleRate: file.readUInt32LE(pos + 12) };
    if (id === "data") pcm = file.subarray(pos + 8, pos + 8 + size);
    pos += 8 + size + (size % 2);
  }
  return { ...fmt, pcm };
}

# pip install requests websockets
# The Python SDK does not read end_of_turn messages yet (gladiaio/sdk#61): a raw WebSocket here.
import asyncio
import json
import os
import sys
import wave

import requests
import websockets

API_URL = os.environ.get("GLADIA_API_URL", "https://eu1.api.gladia.io")

async def main(path: str) -> None:
    with wave.open(path) as wav:  # a 16-bit PCM WAV file
        rate, channels = wav.getframerate(), wav.getnchannels()
        audio = wav.readframes(wav.getnframes())
    created = requests.post(
        f"{API_URL}/v2/live",
        headers={"x-gladia-key": os.environ["GLADIA_API_KEY"]},
        json={
            "encoding": "wav/pcm",
            "sample_rate": rate,
            "bit_depth": 16,
            "channels": channels,
            "model": "trantoria-1",
            "language_config": {"languages": ["en"]},
            "endpointing": 0.3,
            "end_of_turn": {"threshold": 0.8, "stop_s": 1.2},
        },
        timeout=10,
    )
    created.raise_for_status()

    async with websockets.connect(created.json()["url"]) as ws:

        async def send() -> None:
            step = rate * channels * 2 // 10  # 100 ms of audio per chunk
            for start in range(0, len(audio), step):
                await ws.send(audio[start : start + step])  # live audio: send each chunk as it is captured
            await ws.send(json.dumps({"type": "stop_recording"}))

        sender = asyncio.create_task(send())
        async for raw in ws:
            message = json.loads(raw)
            if message["type"] == "transcript" and message["data"]["is_final"]:
                print("caller:", message["data"]["utterance"]["text"])
            elif message["type"] == "end_of_turn":
                print(f"end of turn ({message['data']['reason']}): answer now")
        await sender

asyncio.run(main(sys.argv[1]))

The message:

{
  "type": "end_of_turn",
  "session_id": "0199d7a2-5c3e-7b41-9f0a-3c2d1e4f5a6b",
  "data": { "channel": 0, "probability": 0.97, "reason": "model", "end": 18.42, "speech_duration": 3.1 }
}

reason is model when the probability that the speaker finished crossed threshold, or timeout when the silence reached stop_s whatever the probability.

Options

"end_of_turn": true uses the defaults; an object turns it on and changes some of them. Omitted or false: off.

{ "end_of_turn": { "threshold": 0.8, "stop_s": 1.2 } }

option	range	what it does
`threshold`	0 to 1 (excluded)	Probability above which the turn ends. Lower: faster answers, more interruptions.
`silence_ms`	1 to 10000	Silence, in ms, before a pause is evaluated.
`stop_s`	up to 60	Silence, in seconds, that ends the turn anyway.
`emit_scores`		Also send `end_of_turn_score` for each evaluation, to calibrate `threshold`.
`window_s`	up to 8	Audio, in seconds, an evaluation reads.
`speech_dbfs`	-100 to 0	Level under which audio counts as silence.
`reeval_ms`	0 to 10000	Interval between two evaluations of the same pause.

GET /v2/live/defaults returns the defaults in force.

Building a voice agent

Use a low-latency model and profile: trantoria-1-fast or solaria-3-fast.
Turn on partial transcripts to prepare the answer while the caller speaks.
Answer on end_of_turn, with the finals received so far.
If the caller speaks again before you answer, cancel and wait for the next end_of_turn.

Limits and gotchas

Off unless you ask. A session without end_of_turn gets no end-of-turn message.
WebSocket only. end_of_turn messages are never sent to callbacks: the agent that answers must hold the WebSocket.
Per channel. Each channel has its own turns: with the caller and the agent on two channels, read channel.
threshold is a trade-off. Lower answers sooner and interrupts more; higher waits for clearer endings. Turn on emit_scores on real calls to see the probabilities, then set it.
stop_s is your ceiling. Whatever the probability, a silence that long ends the turn (reason: timeout): keep it under what your callers will tolerate.
Endpointing still applies. Finals still come on each pause: buffer them, answer on end_of_turn.
Python SDK: see above, use a raw WebSocket or the JavaScript SDK.