Introducing VoicePing Diarization v0.1, a multilingual speaker diarization model evaluated on a 42-file benchmark with NeMo, pyannoteAI precision-2, AssemblyAI, and Deepgram.
VoicePing Diarization v0.1 is our first public diarization model release: a speaker segmentation model for multilingual meetings, designed to identify who spoke when before downstream speaker identity matching.
This article introduces the model through a July 2026 benchmark on a 42-file multilingual suite. NeMo, pyannoteAI precision-2, AssemblyAI, and Deepgram are included as evaluation context so readers can understand where VoicePing Diarization v0.1 sits against familiar open and commercial speaker-labeling options.
One caveat matters: VoicePing Diarization v0.1 is the public model identity for the selected benchmark row, not a live production registry check. Production model selection is controlled by runtime configuration, and production also includes downstream speaker identity matching outside this diarization-only benchmark.
Evaluation setup
The benchmark contains 42 files and about 10.5 hours of audio: five monolingual sets in English, Japanese, Korean, Vietnamese, and Mandarin, plus two code-switched multilingual files. Scenarios range from 30 seconds to 1 hour, with 2-9 speakers and 0-30% overlapping speech.
The files are synthetic conversations stitched from real single-speaker recordings. That gives exact reference labels and repeatable scoring, but it is cleaner than many real meetings. Results should be treated as a controlled benchmark, not as a replacement for out-of-domain meeting evaluation.
The evaluation set is intentionally narrower than the raw internal export. NeMo is included as the main local open baseline, using the NeMo Neural MSDD result from the rerun artifacts. pyannoteAI precision-2 is included as the clearest dedicated commercial diarization service. Deepgram and AssemblyAI are included because buyers often compare them during speaker-attributed transcript evaluations, but masked cells are not used for direct headline ranking.
Methodology details: DER was scored with pyannote.metrics-style diarization error rate, a fair collar, overlap scored, and corpus time-weighted aggregation. The July 2026 research evaluation export used the VoicePing Diarization v0.1 benchmark row from the voiceping-inc/titanet Hugging Face snapshot (titanet_finetuned.nemo), the NeMo Neural MSDD baseline, pyannoteAI precision-2, and speaker-label outputs from AssemblyAI and Deepgram.
Overall results
| System | Files | DER | Role | RTF |
|---|---|---|---|---|
| VoicePing Diarization v0.1 | 42 | 4.01% | VoicePing diarization model | 0.024 |
| NeMo | 42 | 6.64% | NeMo Neural MSDD baseline | 0.020 |
| pyannoteAI precision-2 | 42 | 8.55% | Dedicated commercial diarization API | 0.028 |
| AssemblyAI | 42 | xx | Speaker-attributed transcript output | 0.049 |
| Deepgram | 41 | xx | Speaker-attributed transcript output | 0.006 |
The clean headline is simple: VoicePing Diarization v0.1 leads the controlled benchmark, followed by NeMo, then pyannoteAI precision-2. The dedicated diarization API remains close enough to be the serious external comparison. AssemblyAI and Deepgram are included as buying-path context, while the core accuracy comparison remains VoicePing Diarization v0.1, NeMo, and pyannoteAI precision-2.
The component view includes all five public rows. VoicePing Diarization v0.1, NeMo, and pyannoteAI precision-2 show their headline DER labels next to miss, false alarm, and speaker-confusion segments. AssemblyAI and Deepgram are included as speaker-attributed transcript references with proportional component segments.
Results by language
| Language | VoicePing Diarization v0.1 | NeMo | pyannoteAI precision-2 | AssemblyAI | Deepgram |
|---|---|---|---|---|---|
| English | 3.54% | 4.50% | 4.40% | 21.73% | 8.23% |
| Japanese | 3.79% | 7.30% | 10.87% | xx | 28.76% |
| Korean | 4.08% | 10.86% | 11.21% | xx | xx |
| Vietnamese | 4.12% | 5.56% | 7.95% | xx | xx |
| Mandarin | 4.37% | 5.10% | 7.98% | 25.78% | 11.67% |
| Mixed, 5 languages | 3.50% | 7.53% | 16.54% | xx | xx |
| Mixed, 4 languages | 4.14% | 4.59% | 4.62% | xx | xx |
Language-wise, VoicePing Diarization v0.1 is the strongest row across all seven language buckets in this public table. NeMo remains a useful open baseline; this row uses NeMo Neural MSDD. pyannoteAI precision-2 is consistently viable across languages, but trails VoicePing Diarization v0.1 in every bucket here.
The API rows tell a different story. Deepgram and AssemblyAI are useful references for speaker-attributed transcripts, but the public table shows why they should not be presented as direct diarization replacements for multilingual speaker timelines.
Scenario and short-audio takeaways
The scenario view should be read segment by segment, not as a single suite-level average. VoicePing Diarization v0.1 is strongest on the 5-minute base bucket at 3.92% DER, stays in the mid-single digits on 2-minute audio at 6.04%, 5-minute no-overlap audio at 5.93%, and 30-minute 7-9 speaker audio at 5.57%, then rises on harder meeting buckets: 8.23% for 5 minutes with 30% overlap, 8.69% for 5 minutes with 7-9 speakers, and 8.28% for 60-minute audio.
For product planning, that means we should separate three questions. First, which diarization pipeline is strongest for full meetings? Second, which embedding model is robust when each speaker has little speech? Third, how does the full production pipeline behave after MSDD refinement and speaker identity matching? This post answers only the first part at the segmentation layer.
Speed remains favorable for local systems. The PC-54 full-suite export reports VoicePing Diarization v0.1 at 0.024 RTF, close to NeMo at 0.020 RTF and pyannoteAI precision-2 at 0.028 RTF. The API timings include provider behavior and should be read as operational context, not a hardware-normalized benchmark.
Transcript APIs: useful, but different
Deepgram and AssemblyAI attach speaker labels to transcription output. That is useful when the user needs a speaker-attributed transcript, but it is not the same thing as diarizing the full audio timeline. If speech is not transcribed or the transcript is unstable in a language, the speaker timeline inherits that limitation.
AssemblyAI behaves more acoustically than Deepgram, but some rows are still masked in the public table. In this article, both providers stay in the methodology, overall view, and language views because they are common buying-path references, not because they are the strongest diarization competitors.
What this means for VoicePing
This article presents the selected VoicePing benchmark result as VoicePing Diarization v0.1. That keeps the story focused on the customer-facing model identity instead of internal experiment names. VoicePing Diarization v0.1 leads the benchmark, and the serious comparison to watch is pyannoteAI precision-2.
That does not reduce the importance of continued internal diagnosis. Production diarization is only one stage: transcript alignment, speaker segmentation, and then speaker identity matching against known workspace voices. That final stage turns anonymous labels into the same named colleague across meetings, which the benchmarked APIs do not provide. The next public follow-up should evaluate the full production pipeline, not only this isolated segmentation row.
This is also why the article avoids publishing unstable short-clip anecdotes as ratios. Short files can expose real failure modes, but they need a separate acceptance test with enough examples to avoid overfitting the story to one scenario. The production question is not only whether the segmenter wins a controlled corpus; it is whether users consistently see the right speaker names, stable turns, and useful transcripts across live meetings, uploaded calls, and short voice snippets.
Limitations and conclusion
The main limitation is the evaluation set. Synthetic conversations give exact references, but they are cleaner than real meetings and come from a domain related to our training data. The timeline is also close to wall-to-wall speech, so false-alarm behavior is not stressed enough. A real out-of-domain multi-speaker set, including harder overlapping-speech recordings, is still required before making final production claims.
Within those limits, the main conclusion is: VoicePing Diarization v0.1 leads the 42-file benchmark, pyannoteAI precision-2 is the dedicated commercial API row to watch, and the transcript-oriented products should remain qualitative references where masked cells appear. The next work should focus on production model identity, short-audio behavior, and an end-to-end production pipeline benchmark.
