What Breaks Speaker Identification at Scale? A Five-Model Multilingual Open-Set Benchmark

Speaker-identification benchmarks often foreground small equal-error-rate differences and a model ranking. We instead organize this study around the decision that matters in an open-set system: whether the top match is strong and unambiguous enough to accept when the probe speaker may not be enrolled.

Executive conclusion

All five models perform similarly once four seconds of speech and four enrollment examples are available. At two seconds, ReDimNet-B6 and w2v-BERT-SV form the leading group. ReDimNet has the strongest overall research case because its compact checkpoint is consistently strong across the primary held-out evaluation and the clean and large-gallery supporting checks. This is not enough evidence for a production replacement.

The most important operational result is that rejection calibration is model-specific. A universal cosine threshold compares differently scaled embedding spaces rather than only speaker-discrimination quality. Each checkpoint needs a threshold and top-one-versus-top-two margin selected on representative development identities.

The rest of the article answers four questions in order: Which models remain reliable when speech is short? Do model differences persist once four seconds are available? How should unknown speakers be rejected? And do the combined accuracy, compute, and licensing results support a production decision?

Primary held-out evaluation

Which evidence should carry the most weight? The identity-disjoint open-set comparison is primary because it separates calibration speakers from final test speakers while comparing all five models. For each language, 20 identities remain in the gallery and 10 are removed completely. Five removed identities select the similarity threshold and top-two margin; the other five test those choices unchanged. This gives a 100-speaker gallery, 25 development-unknown identities, and 25 separate test-unknown identities. Every model receives the same hashed probe cohorts.

Open-set identification contains two coupled decisions: which enrolled speaker has the highest similarity, and whether that match is confident enough to accept. We report equal error rate (EER), detection-and-identification rate (DIR)—the fraction of known probes both correctly identified and accepted—and false accept rate (FAR) on unknown speakers.

Thresholds and margins are selected only on development identities under a development-unknown FAR target of at most 1%, then applied unchanged to the test cohort. Confidence intervals use 2,000 speaker-cluster bootstrap replicates, and 50 deterministic identity splits measure sensitivity to the selected speakers. This separation makes the held-out evaluation the most credible open-set evidence in the study.

The primary split with four enrollment embeddings produced:

This evaluation establishes the strongest overall evidence. The next section separates that evidence by operating condition so the model choice is clear for short speech, four-second speech, and unknown-speaker rejection.

Results by operating condition

The primary table answers three operational questions. Each subsection begins with the stable conclusion, then shows the point estimate and uncertainty that support it.

Which models hold up with only two seconds?

ReDimNet and w2v-BERT form the leading group when only two seconds are available. On the primary split, ReDimNet has the highest DIR. Paired speaker-bootstrap comparisons against TitaNet estimate an 11.0-point DIR gain for ReDimNet (95% CI 7.4–15.2), an 8.4-point gain for w2v-BERT (4.6–12.4), and a 5.8-point gain for WeSpeaker (2.2–9.6). All three remain significant after Holm correction; ECAPA’s one-point gain is not significant.

The stability analysis qualifies that point estimate rather than contradicting it. Across 50 deterministic splits, median DIR is 92.8% for w2v-BERT and 92.0% for ReDimNet, with substantially overlapping 5th-to-95th percentile ranges. The evidence therefore supports a leading short-speech pair, not a strict one-through-five ranking.

Does model choice still matter at four seconds?

Not meaningfully in the current evidence. At four seconds, none of the candidate-versus-TitaNet operational differences remains significant after correction. The intervals overlap and all five models are close to saturation. TitaNet’s 98.8% observed DIR therefore does not establish superiority, just as ReDimNet’s 0% observed FAR does not establish zero unknown-speaker risk. With 250 test-unknown probes, zero accepts still has a 1.46% upper 95% exact binomial bound. Once four seconds are available, uncertainty and operating risk matter more than the model ranking.

How should unknown speakers be rejected?

Use model-specific calibration rather than a universal cosine threshold. The selected similarity threshold controls absolute match strength, while the top-one/top-two margin rejects ambiguous matches. Both must be chosen without using the identities reserved for final testing. The held-out results show that this rule transfers to unseen unknown speakers, but the small test cohort leaves material uncertainty.

These are the primary held-out conclusions. Before applying them more broadly, the next section asks whether they remain consistent under a simpler clean benchmark and a much larger gallery.

Supporting checks

Do the primary findings depend on one protocol? Two supporting checks answer different parts of that question. The clean benchmark tests baseline model strength and duration sensitivity across all five models; the 900-speaker diagnostic tests gallery growth for TitaNet and ReDimNet only. Neither replaces the held-out evaluation.

Clean multilingual benchmark

Are all five models fundamentally strong on clean speech? Yes. The clean benchmark uses 30 known and 10 unknown identities per language. Its primary condition is a balanced 150-speaker gallery with four-second probes and four enrollment embeddings. This is a supporting model-strength check, not the primary unknown-speaker or meeting-readiness result.

At four seconds, no model has an EER above 0.74%, and every model exceeds 99.5% closed-set top-1 accuracy. ReDimNet records the lowest EER, but the full spread is only 0.33 percentage points.

The 39.6-point spread in unknown acceptance at a raw 0.5 threshold illustrates why one threshold cannot be transferred across models. Duration and enrollment have more practical effect than the small four-second EER gaps: additional enrollment examples generally help, two-second probes reveal clearer differences, and the benchmark begins to saturate by four seconds.

The direction is consistent with published results but is not a reproduction. The TitaNet paper reports 0.68% EER on cleaned VoxCeleb1 verification trials, and the ReDimNet paper reports 0.40% for ReDimNet-B6 SF2-LM on VoxCeleb1-O Cleaned. Those trials differ from our multilingual identification gallery in audio, duration, identities, score population, and task.

The clean check establishes baseline strength but not resilience to gallery growth. The next diagnostic asks that narrower question for the two models it covers.

900-speaker gallery diagnostic

Does ReDimNet remain competitive as the gallery expands? Directionally, yes. This supporting diagnostic expands the same clean source corpus to 900 enrolled speakers and 100 withheld identities, but covers only TitaNet and ReDimNet. At four seconds with four enrollment embeddings, ReDimNet and TitaNet have nearly identical EER (0.82% and 0.84%) and closed-set top-1 accuracy (98.92% and 98.81%). ReDimNet retains the more favorable score distribution under this diagnostic.

This result is directional, not primary evidence: the same unknown cohort was used for FAR calibration and evaluation. It suggests that ReDimNet remains competitive as the gallery grows, but it is not a held-out result or a deployment guarantee.

Together, the supporting checks strengthen the consistency case without changing the primary conclusion. Consistent accuracy still does not establish deployability, so the next section considers compute, licensing, and the measurements that remain missing.

Deployment evaluation

Do these findings support a production choice? Not yet. Accuracy is only one part of deployment selection. The current run recorded embedding real-time factor (RTF) on an NVIDIA GeForce RTX 5090. Multiplying RTF by the four-second input length gives the approximate embedding time below; it does not include model loading, audio transfer, gallery scoring, networking, concurrency, or application logic.

Peak VRAM, checkpoint size, cold-start time, CPU production performance, sustained throughput, and hardware sizing were not measured in this publication run. A deployment decision therefore needs a separate benchmark on the intended runtime and concurrency profile; the table does not establish a deployment winner.

Those missing measurements are not side notes; they define the boundary of the evidence. The next section makes that boundary explicit.

Limitations

What does this study not establish? It does not establish robustness outside clean, mostly single-session audio. Although enrollment and probe windows do not overlap in time, 4,398 of 4,469 identities—98.4%—have only one source recording. The results may therefore retain microphone, channel, room, background, and session characteristics and do not establish cross-session or real-meeting robustness.

The held-out unknown-speaker test contains only 25 identities and 250 probes per condition. The study does not cover overlapping speech, diarization errors, noisy rooms, production concurrency, or end-to-end latency. Its conclusions support model and calibration research, not an immediate production replacement.

These limits constrain how the results should be used. The appendix documents how each result was produced and why the three evaluation protocols carry different evidentiary weight.

Methodology appendix

How were the results produced and made comparable? The appendix records checkpoint provenance, deterministic sampling, enrollment construction, calibration search, uncertainty estimation, and the role of each protocol.

Models and checkpoint provenance

All five public checkpoint families use the same cosine-scoring harness:

The evaluated TitaNet is the stock public checkpoint; none of its results should be attributed to a separately fine-tuned TitaNet checkpoint.

Data, sampling, and enrollment

The evaluation manifest contains 5,089 recordings from 4,469 labeled identities: English 1,665/1,489 recordings/identities, Japanese 1,704/1,476, Korean 438/438, Vietnamese 425/209, and Chinese 857/857. These curated single-speaker talks and long-form interview or channel recordings include TED and language-specific collections. They are evaluation data, not descriptions of the models’ training corpora. An overlap audit found no public evaluation recordings in the documented pretraining corpora of the evaluated checkpoints.

For each duration, the harness deterministically samples fixed 2-, 4-, or 8-second windows. A known identity contributes four enrollment and ten probe windows; an unknown identity contributes ten probe windows. Time bins do not overlap. Audio is read as float32, mixed to mono, and resampled to 16 kHz when necessary. The harness adds no VAD, denoising, loudness normalization, augmentation, or diarization.

Each crop embedding is L2-normalized. The first one, two, or four enrollment vectors are averaged and the mean is normalized again to create a centroid. Each normalized probe is scored against every centroid by cosine similarity. Stable hashes and seed 1337 give every model the same identities and crop plan.

Rule selection and uncertainty

Similarity-threshold candidates are empirical quantiles of development-unknown top scores, using up to 101 evenly spaced levels plus one value above the maximum. Margin candidates are zero plus up to 51 empirical quantile levels of development-known and development-unknown top-one-minus-top-two margins.

The harness keeps rules with development-unknown FAR at or below 1% and selects the highest development DIR. Ties prefer lower FAR, then a lower similarity threshold, then a lower margin. The rule is applied unchanged to identity-disjoint test speakers. Uncertainty uses 2,000 speaker-cluster bootstrap replicates, and 50 deterministic identity splits test result stability.

Evidence history

The clean 150-speaker benchmark establishes baseline model strength and duration/enrollment sensitivity. The 900-speaker diagnostic tests gallery growth for TitaNet and ReDimNet but reuses its unknown cohort for calibration and evaluation, so its operating figures are directional. The identity-disjoint five-model evaluation separates development and test unknown speakers and is therefore the primary evidence reported above. These protocols answer different questions and must not be substituted for one another.