# Named entity recognition

**The problem.** A caller gives their name, their order number and a delivery date; your CRM needs those fields, not
a paragraph. Named entity recognition finds them in each final transcript as it arrives: people, organizations,
places, dates, amounts, emails, phone numbers and about 70 types in all.

**Where it works.** Every model.

> **Use the SDK.** The Gladia SDKs stream the audio, keep it until the API has transcribed it, and reconnect after a network drop
without losing a word. Most integration issues come from hand-written WebSocket clients: start from the SDK
examples below.

## Step by step

### 1. Turn it on

```json
{ "named_entity_recognition": true }
```

`realtime_processing.named_entity_recognition: true`, Gladia's field, does the same.

### 2. Start a session with it

Each program streams a 16-bit WAV file and prints what comes back. Put your API key in the `GLADIA_API_KEY` environment variable, then run `python entities.py meeting.wav`.

**Python SDK**

```python title="entities.py"
# pip install gladiaio-sdk requests
import os
import sys
import threading
import wave

import requests
from gladiaio_sdk import GladiaClient, LiveV2ConnectSessionOptions

API_URL = os.environ.get("GLADIA_API_URL", "https://eu1.api.gladia.io")
API_KEY = os.environ["GLADIA_API_KEY"]

with wave.open(sys.argv[1]) as wav:  # a 16-bit PCM WAV file
    rate, channels = wav.getframerate(), wav.getnchannels()
    audio = wav.readframes(wav.getnframes())

config = {
    "encoding": "wav/pcm",
    "sample_rate": rate,
    "bit_depth": 16,
    "channels": channels,
    "model": "solaria-3",
    "language_config": {
        "languages": [
            "en"
        ]
    },
    "named_entity_recognition": {
        "entities": [
            "name",
            "organization",
            "date",
            "phone_number"
        ]
    }
}

client = GladiaClient(api_key=API_KEY, api_url=API_URL)
# `named_entity_recognition` with options is a Gladia Enterprise option the Python SDK does not send yet
# (gladiaio/sdk#61): create the session over HTTP, then let the SDK stream it.
created = requests.post(f"{API_URL}/v2/live", headers={"x-gladia-key": API_KEY}, json=config, timeout=10)
created.raise_for_status()
info = created.json()
session = client.live_v2().connect_session(LiveV2ConnectSessionOptions(id=info["id"], url=info["url"]))
ended, errors = threading.Event(), []

@session.on("message")
def on_message(message):
    if message.type == "transcript" and message.data.is_final:
        print(message.data.utterance.text)
    elif message.type == "named_entity_recognition":
        for entity in message.data.results:
            print(f"  {entity.entity_type}: {entity.text}")

session.on("error", errors.append)
session.once("ended", lambda _: ended.set())

step = rate * channels * 2 // 10  # 100 ms of audio per chunk
for start in range(0, len(audio), step):
    session.send_audio(audio[start : start + step])  # live audio: send each chunk as it is captured
session.stop_recording()

ended.wait(timeout=60)
sys.exit(1 if errors or not ended.is_set() else 0)
```

**JavaScript SDK**

```js title="entities.mjs"
// npm install @gladiaio/sdk
import { readFileSync } from "node:fs";
import { GladiaClient } from "@gladiaio/sdk";

const { sampleRate, channels, pcm } = readWav(process.argv[2]); // a 16-bit PCM WAV file

const config = {
  encoding: "wav/pcm",
  sample_rate: sampleRate,
  bit_depth: 16,
  channels,
  model: "solaria-3",
  language_config: {
    languages: [
      "en"
    ]
  },
  named_entity_recognition: {
    entities: [
      "name",
      "organization",
      "date",
      "phone_number"
    ]
  }
};

const client = new GladiaClient({
  apiKey: process.env.GLADIA_API_KEY,
  apiUrl: process.env.GLADIA_API_URL ?? "https://eu1.api.gladia.io",
});
// `named_entity_recognition` with options is a Gladia Enterprise option not in the SDK's types yet: the SDK sends it as is.
const session = client.liveV2().startSession(config);

session.on("message", (message) => {
  if (message.type === "transcript" && message.data.is_final) console.log(message.data.utterance.text_raw);
  if (message.type === "named_entity_recognition") {
    for (const entity of message.data.results) console.log(`  ${entity.entity_type}: ${entity.text}`);
  }
});
session.on("error", (error) => {
  console.error(error);
  process.exitCode = 1;
});
session.once("ended", () => process.exit());

const step = (sampleRate * channels * 2) / 10; // 100 ms of audio per chunk
for (let start = 0; start < pcm.length; start += step) {
  session.sendAudio(pcm.subarray(start, start + step)); // live audio: send each chunk as it is captured
}
session.stopRecording();

function readWav(path) {
  const file = readFileSync(path);
  let fmt, pcm;
  for (let pos = 12; pos < file.length; ) {
    const id = file.toString("ascii", pos, pos + 4);
    const size = file.readUInt32LE(pos + 4);
    if (id === "fmt ") fmt = { channels: file.readUInt16LE(pos + 10), sampleRate: file.readUInt32LE(pos + 12) };
    if (id === "data") pcm = file.subarray(pos + 8, pos + 8 + size);
    pos += 8 + size + (size % 2);
  }
  return { ...fmt, pcm };
}
```

**HTTP**

Create the session, then stream its audio to the returned `url` ([Live WebSocket](/api-reference/v2/live/websocket/)):

```bash
curl -X POST https://eu1.api.gladia.io/v2/live \
  -H "x-gladia-key: $GLADIA_API_KEY" \
  -H "content-type: application/json" \
  -d '{
    "encoding": "wav/pcm",
    "sample_rate": 16000,
    "bit_depth": 16,
    "channels": 1,
    "model": "solaria-3",
    "language_config": {
      "languages": [
        "en"
      ]
    },
    "named_entity_recognition": {
      "entities": [
        "name",
        "organization",
        "date",
        "phone_number"
      ]
    }
  }'
```

### 3. Read the entities

**The final's text carries them inline**, and `text_raw` keeps the plain sentence:

```json
{
  "type": "transcript",
  "data": {
    "id": "00_00000002",
    "is_final": true,
    "utterance": {
      "text": "Hi, this is <ner:name id=\"1\">Marie Curie</ner:name> from <ner:organization id=\"1\">Gladia</ner:organization>.",
      "text_raw": "Hi, this is Marie Curie from Gladia.",
      "entities": [
        { "label": "name", "id": 1, "action": "ner", "text": "Marie Curie", "start": 12, "end": 52 },
        { "label": "organization", "id": 1, "action": "ner", "text": "Gladia", "start": 58, "end": 102 }
      ]
    }
  }
}
```

Right after it, a `named_entity_recognition` message lists them with their times in the audio:

```json
{
  "type": "named_entity_recognition",
  "error": null,
  "data": {
    "utterance_id": "00_00000002",
    "results": [
      { "entity_type": "name", "text": "Marie Curie", "start": 4.9, "end": 5.6, "id": 1, "score": 0.97, "char_start": 12, "char_end": 52 }
    ]
  }
}
```

## Options

| option | what it does |
|---|---|
| `entities` | Which types: a preset (`name`, `location`, `date`, `contact`, `financial`, `identifiers`, `pii`…), a list of types and presets, or `{"enable": [...], "disable": [...]}`. Omitted: every type. |
| `threshold` | Confidence, from 0 to 1, under which an entity is ignored. Default 0.5. |

A parent type (`name`, `location`, `date`) also selects its children (`name_given`, `name_family`,
`location_city`…). An unknown type is refused, with the list of known ones.

## Limits and gotchas

- **The text changes.** With entities on, `utterance.text` contains `<ner:…>` elements. Display `text_raw`, or render
  the elements; never show `text` as is to an end user.
- **Finals only.** Partials are not tagged.
- **`id` follows the entity through the session.** The same person keeps `id` 1, even when only their first name is
  heard again. `start`/`end` on the utterance and `char_start`/`char_end` in the message locate the element in
  `text`, not in `text_raw`.
- **The message is not stored.** The `named_entity_recognition` messages are not part of the session's result;
  the `entities` of each utterance are. Store the messages yourself (WebSocket or [callback](/live/callbacks/)) if
  you need their audio times.
- **If tagging fails on a final**, the message comes with `error` set and that final has no words. Rare; the
  session goes on.
- **Personal data stays visible.** Entities are tagged, not hidden: use [PII redaction](/features/pii-redaction/) to
  remove them. Both together: the types selected for redaction are redacted, the others tagged.

## Troubleshooting

| symptom | cause | fix |
|---|---|---|
| users see `<ner:name id="1">` in your interface | `text` displayed as is | display `text_raw`, or render the elements |
| no `named_entity_recognition` message | `messages_config.receive_realtime_processing_events: false` | turn it on |
| `rejected: … unknown entity labels` (WebSocket `1011`) | a type that does not exist | use the types and presets of the error message |
| too many false entities | low confidence spans | raise `threshold` |