For AI agents: a documentation index is available at /llms.txt. Markdown versions of all pages can be requested by appending `.md` to the URL, or by setting the `Accept` header to `text/markdown`.
Skip to main content
Speech to TextFeatures

Speaker identification

Speaker identification lets you tag speakers consistently across recordings using speaker identifiers, which are string-encoded voice representations generated from short audio samples of the target speakers.

By tagging known speakers with consistent labels, speaker identification makes transcripts more accurate, searchable, and easier to analyze over time. Providing speaker identifiers can also increase the accuracy of diarization.

Use cases​

  • Contact centers — recognize and tag individual agents and returning customers by name for personalized service, training, and compliance tracking.
  • Video conferences — automatically label participants across multiple meetings to know who said what and maintain consistent speaker records and analytics.
  • Medical consultations — identify doctors and patients across sessions for accurate records and follow-up care.
  • Media production — consistently label recurring speakers or public figures across episodes or segments, which is valuable in subtitling, media search, and archiving.

Availability​

Speaker identification is available with the Standard and Enhanced models for pre-recorded and streaming transcription, and with Linden 1 for agent STT. See Feature availability.

It requires speaker diarization to be enabled.

How it works​

Speaker identification is a two-step process.

  • Enrollment — for each speaker you want to recognize, generate identifiers from short audio clips of 5 to 30 seconds where they ideally speak alone. To improve robustness, you can enroll the same speaker with multiple clips recorded under different acoustic conditions, chosen to represent the variety and quality expected in the target audio.
  • Identification — use the enrolled identifiers in later requests to label known speakers with meaningful names such as Alice or John. The system matches voices to identifiers and tags the output with your labels.

Minimize the number of speaker identifiers for optimal accuracy. A maximum of 50 speaker identifiers across all speakers can be configured per session. Labels for identified speakers must not use reserved internal labels such as UU, S1, or S2, and must not contain leading or trailing spaces.

Enroll speakers​

Run a transcription with speaker diarization enabled on an audio sample where the speaker ideally speaks alone, then request the identifiers back.

Set the get_speakers flag in the transcription config:

{
  "type": "transcription",
  "transcription_config": {
    "model": "enhanced",
    "language": "en",
    "diarization": "speaker",
    "speaker_diarization_config": {
      "get_speakers": true
    }
  }
}

When the transcription is done, the speaker identifiers are attached to the returned transcript:

{
  "results": [
    {
      "alternatives": [
        {
          "confidence": 0.93,
          "content": "Hello",
          "language": "en",
          "speaker": "S1"
        }
      ],
      ...
    },
    {
      "alternatives": [
        {
          "confidence": 1.0,
          "content": "Hi",
          "language": "en",
          "speaker": "S2"
        }
      ],
      ...
    }],
  "speakers": [
    {
      "label": "S1",
      "speaker_identifiers": ["<id1>"]
    },
    {
      "label": "S2",
      "speaker_identifiers": ["<id2>"]
    }]
}

Identify known speakers​

Provide your stored identifiers in a later request to tag known speakers.

Note the field differs by interaction pattern: pre-recorded and streaming use speaker_diarization_config.speakers, while agent STT uses transcription_config.known_speakers. Both express the same concept.

All other speaker diarization options remain supported. The speakers_sensitivity parameter adjusts how strongly the system prefers enrolled speakers over detecting new generic ones; lower values make it more likely to match existing enrolled speakers.

{
  "type": "transcription",
  "transcription_config": {
    "model": "enhanced",
    "language": "en",
    "diarization": "speaker",
    "speaker_diarization_config": {
      "speakers": [
        {"label": "Alice", "speaker_identifiers": ["<alice_id1>", "<alice_id2>"]},
        {"label": "Bob", "speaker_identifiers": ["<bob_id1>"]}
      ]
    }
  }
}

With the config above, transcript segments are tagged with "Alice" and "Bob" whenever those speakers are detected. Any other speakers are tagged with the internal labels:

{
  "results": [
    {
      "alternatives": [
        {
          "confidence": 0.93,
          "content": "Morning",
          "language": "en",
          "speaker": "Alice"
        }
      ],
      ...
    },
    {
      "alternatives": [
        {
          "confidence": 0.93,
          "content": "Hi",
          "language": "en",
          "speaker": "S1"
        }
      ],
      ...
    },
    {
      "alternatives": [
        {
          "confidence": 1.0,
          "content": "Morning",
          "language": "en",
          "speaker": "Bob"
        }
      ],
    }]
}

Known caveats​

Speaker identifiers have the following limitations and scoping rules:

  • Model-specific — identifiers are tied to the model used to generate them. Using identifiers across different models is not supported, and any such identifiers are ignored. Whenever a model is updated, existing identifiers must be regenerated.
  • Encrypted and scoped — identifiers are securely encrypted and scoped to your account context:
    • Per customer — identifiers are unique to each customer and cannot be shared or reused across customers.
    • Per project — if you use multiple projects under the same customer, identifiers are isolated per project and cannot be used across them.

In all of these cases, including model mismatches and attempts to use identifiers across customers or projects, a warning is issued to alert you that the affected identifiers have been ignored.

Code examples​

Streaming speaker enrollment example.

import asyncio
import speechmatics
import speechmatics.models
import speechmatics.client
from speechmatics.client import (
    ServerMessageType,
    ClientMessageType
)

API_KEY = "YOUR_API_KEY"
PATH_TO_FILE = "example.wav"
LANGUAGE = "en"
CONNECTION_URL = "wss://eu.rt.speechmatics.com/v2"

async def enroll_speakers():
    handler_tasks: list[asyncio.Task] = []

    transcription_config = speechmatics.models.TranscriptionConfig(**{
            "model": "enhanced",
            "language": LANGUAGE,
            "diarization": "speaker",
        }
    )

    # Create a transcription client
    client = speechmatics.client.WebsocketClient(
        speechmatics.models.ConnectionSettings(
            url=CONNECTION_URL,
            auth_token=API_KEY,
        )
    )
    # Register the event handler for RecognitionStarted
    # to send the GetSpeakers(final=True) request
    client.add_event_handler(
        ServerMessageType.RecognitionStarted,
        lambda _: handler_tasks.append(
            asyncio.create_task(client.send_message(ClientMessageType.GetSpeakers, {"final": True}))
        ),
    )
    # Register the event handler for SpeakersResult
    # to print the speaker identifiers obtained from the server
    client.add_event_handler(
        ServerMessageType.SpeakersResult,
        lambda message: print(f"[speaker identifiers] {message['speakers']}"),
    )

    with open(PATH_TO_FILE, "rb") as fh:
        await asyncio.create_task(client.run(fh, transcription_config))

    for task in handler_tasks:
        await task

if __name__ == "__main__":
    asyncio.run(enroll_speakers())