In this article
VoicePing Diarization v0.1 is our first public diarization model release: a speaker segmentation model for multilingual meetings, designed to identify who spoke when before downstream speaker identity matching.
This article introduces the model through a July 2026 benchmark on a 42-file multilingual suite. NeMo, pyannoteAI precision-2, AssemblyAI, and Deepgram are included as evaluation context so readers can understand where VoicePing Diarization v0.1 sits against familiar open and commercial speaker-labeling options.
One caveat matters: VoicePing Diarization v0.1 is the public model identity for the selected benchmark row, not a live production registry check. Production model selection is controlled by runtime configuration, and production also includes downstream speaker identity matching outside this diarization-only benchmark.
Evaluation setup
The benchmark contains 42 files and about 10.5 hours of audio: five monolingual sets in English, Japanese, Korean, Vietnamese, and Mandarin, plus two code-switched multilingual files. Scenarios range from 30 seconds to 1 hour, with 2-9 speakers and 0-30% overlapping speech.
The files are synthetic conversations stitched from real single-speaker recordings. That gives exact reference labels and repeatable scoring, but it is cleaner than many real meetings. Results should be treated as a controlled benchmark, not as a replacement for out-of-domain meeting evaluation.
The evaluation set is intentionally narrower than the raw internal export. NeMo is included as the main local open baseline, using the NeMo Neural MSDD result from the rerun artifacts. pyannoteAI precision-2 is included as the clearest dedicated commercial diarization service. Deepgram and AssemblyAI are included because buyers often compare them during speaker-attributed transcript evaluations, but masked cells are not used for direct headline ranking.
Methodology details: DER was scored with pyannote.metrics-style diarization error rate, a fair collar, overlap scored, and corpus time-weighted aggregation. The July 2026 research evaluation export used the VoicePing Diarization v0.1 benchmark row from the voiceping-inc/titanet Hugging Face snapshot (titanet_finetuned.nemo), the NeMo Neural MSDD baseline, pyannoteAI precision-2, and speaker-label outputs from AssemblyAI and Deepgram.
Overall results
| System | Files | DER | Role | RTF |
|---|---|---|---|---|
| VoicePing Diarization v0.1 | 42 | 4.01% | VoicePing diarization model | 0.024 |
| NeMo | 42 | 6.64% | NeMo Neural MSDD baseline | 0.020 |
| pyannoteAI precision-2 | 42 | 8.55% | Dedicated commercial diarization API | 0.028 |
| AssemblyAI | 42 | xx | Speaker-attributed transcript output | 0.049 |
| Deepgram | 41 | xx | Speaker-attributed transcript output | 0.006 |
The clean headline is simple: VoicePing Diarization v0.1 leads the controlled benchmark, followed by NeMo, then pyannoteAI precision-2. The dedicated diarization API remains close enough to be the serious external comparison. AssemblyAI and Deepgram are included as buying-path context, while the core accuracy comparison remains VoicePing Diarization v0.1, NeMo, and pyannoteAI precision-2.
The component view includes all five public rows. VoicePing Diarization v0.1, NeMo, and pyannoteAI precision-2 show their headline DER labels next to miss, false alarm, and speaker-confusion segments. AssemblyAI and Deepgram are included as speaker-attributed transcript references with proportional component segments.
Systems compared
These five entries follow the benchmark table order. DER measures speaker-timeline errors; lower is better. RTF is processing time divided by audio duration. API and local timings use different serving paths and are not hardware-normalized.
1. VoicePing Diarization v0.1
42 files · DER 4.01% · RTF 0.024. VoicePing had the lowest reported overall diarization error rate in the controlled suite. DER combines missed speech, speech detected where none exists, and speech assigned to the wrong speaker. It is not the percentage of people correctly named, and the 4.01% score does not measure transcript accuracy.
The language rows ranged from 3.50% on the five-language mixed bucket to 4.37% on Mandarin. The scenario analysis adds a different view: errors rose in the high-overlap, many-speaker and hour-long conditions. Those scenario figures describe their stated segments and should not be treated as another full-corpus average. Together, the results support further evaluation of the selected pipeline while showing why an aggregate score alone is insufficient for meeting design.
The public model name identifies this research row; it does not verify the currently active production model. Downstream matching of anonymous labels to workspace identities was excluded. The near-continuous synthetic speech also provides limited evidence about false alarms in quiet rooms or background noise. A deployment evaluation should include those conditions and natural overlapping conversations before interpreting the result as reliable named-speaker behavior.
2. NeMo Neural MSDD
42 files · DER 6.64% · RTF 0.020. The evaluated NeMo Neural MSDD pipeline is the local open baseline. NVIDIA’s documentation explains the multiscale approach: longer speech windows help characterize a voice, while shorter windows locate rapid turn changes. The decoder combines evidence at different scales.
Its overall DER was 2.63 percentage points above VoicePing’s in this corpus, while its observed RTF was slightly lower. Language behavior was uneven: the published English and Mandarin rows were 4.50% and 5.10%, compared with 10.86% for Korean. These differences identify conditions worth investigating; the small constructed corpus does not establish a general language-support ranking or the cause of each error.
For teams maintaining a local pipeline, NeMo provides a useful comparison point for segmentation and embedding choices. Keep the exact voice-activity, clustering and decoder settings with the result, since those stages affect the final timeline. This row does not represent every NeMo diarizer, including newer end-to-end alternatives. Nor does an RTF below one establish live turn-update latency, concurrent capacity or the hardware cost of a production service.
3. pyannoteAI precision-2
42 files · DER 8.55% · RTF 0.028. pyannoteAI precision-2 is the dedicated commercial diarization API evaluated here. It returns speaker turns on the audio timeline, making its complete DER directly interpretable alongside the two local pipelines within this corpus.
The overall ordering hides useful variation. Its English DER was 4.40%, slightly below NeMo’s 4.50% point estimate, while the five-language mixed file reached 16.54%. The four-language mixed file was much lower at 4.62%. Two different mixed files therefore do not establish a simple relationship between language count and accuracy. Audio, turns and speaker conditions need to be examined together, and the article provides no uncertainty test establishing superiority from the small English difference.
API RTF includes the observed serving path, so it is not a controlled GPU-speed comparison. This is also a historical precision-2 result, not a measurement of the newer precision-3 listed in current documentation. pyannoteAI offers voiceprint identification separately; that capability was not evaluated in this diarization-only run. A buyer comparing named-speaker workflows should test that additional stage instead of assuming these anonymous-label scores cover it.
4. AssemblyAI
42 files · Overall DER masked (xx) · RTF 0.049. This row evaluates speaker labels attached to transcription output. AssemblyAI’s documentation
describes utterances with text, timestamps and speaker labels, a convenient output when the application primarily needs to display who said each sentence.
The public export reports English DER of 21.73% and Mandarin DER of 25.78%, but masks the overall result and the remaining language cells. Those two values cannot reconstruct a reliable full-suite score. xx means no score is published here; it is neither zero nor evidence that a language is unsupported. A speaker timeline reconstructed from recognized words also inherits gaps in the transcript, which complicates comparison with a standalone full-audio segmenter.
For a transcript-centered product, evaluate word accuracy and speaker attribution together, then inspect short interjections and overlapping turns. AssemblyAI exposes expected-speaker-count controls; its documentation cautions that an incorrect count can merge people or split one person into multiple labels. The present export does not establish gains from changing those settings. Preserve the request configuration in any rerun, and do not treat this incomplete row as a complete assessment of the current service.
5. Deepgram
41 files · Overall DER masked (xx) · RTF 0.006. Deepgram’s diarization output
attaches speaker labels to recognized words. That can simplify building a speaker-attributed transcript, but the evaluated word-derived timeline and the dedicated diarization outputs need their differences kept visible.
The published language cells were 8.23% DER for English, 28.76% for Japanese and 11.67% for Mandarin. Other cells and the overall result are masked. One fewer file was included than in the other rows, so even a complete mean would need its coverage reconciled before a paired comparison. The low RTF is an observed processing result; it cannot compensate for missing accuracy evidence or prove the fastest usable meeting workflow.
Current documentation distinguishes batch and streaming diarizer versions and describes response metadata identifying the diarizer that ran. A new evaluation should save that version alongside the transcription model, language and audio coverage. These July results must not silently become a claim about whatever the current default selects. For short responses and code-switching, inspect whether words were omitted before attributing every missing speaker segment to the diarizer itself; this article does not separate those error sources quantitatively.
Results by language
| Language | VoicePing Diarization v0.1 | NeMo | pyannoteAI precision-2 | AssemblyAI | Deepgram |
|---|---|---|---|---|---|
| English | 3.54% | 4.50% | 4.40% | 21.73% | 8.23% |
| Japanese | 3.79% | 7.30% | 10.87% | xx | 28.76% |
| Korean | 4.08% | 10.86% | 11.21% | xx | xx |
| Vietnamese | 4.12% | 5.56% | 7.95% | xx | xx |
| Mandarin | 4.37% | 5.10% | 7.98% | 25.78% | 11.67% |
| Mixed, 5 languages | 3.50% | 7.53% | 16.54% | xx | xx |
| Mixed, 4 languages | 4.14% | 4.59% | 4.62% | xx | xx |
Language-wise, VoicePing Diarization v0.1 is the strongest row across all seven language buckets in this public table. NeMo remains a useful open baseline; this row uses NeMo Neural MSDD. pyannoteAI precision-2 is consistently viable across languages, but trails VoicePing Diarization v0.1 in every bucket here.
The incomplete API rows cannot establish whether either service is an adequate replacement for a dedicated diarizer in a particular workflow. That requires comparable audio coverage, published scores and a defined output contract: a speaker-attributed transcript or a full speech timeline.
Scenario and short-audio takeaways
The scenario view should be read segment by segment, not as a single suite-level average. VoicePing Diarization v0.1 is strongest on the 5-minute base bucket at 3.92% DER, stays in the mid-single digits on 2-minute audio at 6.04%, 5-minute no-overlap audio at 5.93%, and 30-minute 7-9 speaker audio at 5.57%, then rises on harder meeting buckets: 8.23% for 5 minutes with 30% overlap, 8.69% for 5 minutes with 7-9 speakers, and 8.28% for 60-minute audio.
For product planning, that means we should separate three questions. First, which diarization pipeline is strongest for full meetings? Second, which embedding model is robust when each speaker has little speech? Third, how does the full production pipeline behave after MSDD refinement and speaker identity matching? This post answers only the first part at the segmentation layer.
Speed remains favorable for local systems. The PC-54 full-suite export reports VoicePing Diarization v0.1 at 0.024 RTF, close to NeMo at 0.020 RTF and pyannoteAI precision-2 at 0.028 RTF. The API timings include provider behavior and should be read as operational context, not a hardware-normalized benchmark.
Transcript APIs: useful, but different
Deepgram and AssemblyAI attach speaker labels to transcription output. That is useful when the user needs a speaker-attributed transcript, but it is not the same thing as diarizing the full audio timeline. If speech is not transcribed or the transcript is unstable in a language, the speaker timeline inherits that limitation.
The public results do not isolate the internal acoustic and transcription contributions of either provider. Both remain in the comparison because they are common options for speaker-attributed transcripts; their masked cells prevent a complete accuracy ranking.
What this means for VoicePing
This article presents the selected VoicePing benchmark result as VoicePing Diarization v0.1. That keeps the story focused on the customer-facing model identity instead of internal experiment names. VoicePing Diarization v0.1 leads the benchmark, and the serious comparison to watch is pyannoteAI precision-2.
That does not reduce the importance of continued internal diagnosis. Production diarization is only one stage: transcript alignment, speaker segmentation, and then speaker identity matching against known workspace voices. That final stage aims to associate anonymous labels with a named colleague across meetings. This benchmark did not test such identity matching. Some providers offer it separately, including pyannoteAI voiceprints , so its absence from this evaluation is not evidence that competing services lack the feature. The next public follow-up should evaluate the full production pipeline, not only this isolated segmentation row.
This is also why the article avoids publishing unstable short-clip anecdotes as ratios. Short files can expose real failure modes, but they need a separate acceptance test with enough examples to avoid overfitting the story to one scenario. The production question is not only whether the segmenter wins a controlled corpus; it is whether users consistently see the right speaker names, stable turns, and useful transcripts across live meetings, uploaded calls, and short voice snippets.
Limitations and conclusion
The main limitation is the evaluation set. Synthetic conversations give exact references, but they are cleaner than real meetings and come from a domain related to our training data. The timeline is also close to wall-to-wall speech, so false-alarm behavior is not stressed enough. A real out-of-domain multi-speaker set, including harder overlapping-speech recordings, is still required before making final production claims.
Within those limits, the main conclusion is: VoicePing Diarization v0.1 leads the 42-file benchmark, pyannoteAI precision-2 is the dedicated commercial API row to watch, and the transcript-oriented products should remain qualitative references where masked cells appear. The next work should focus on production model identity, short-audio behavior, and an end-to-end production pipeline benchmark.
