Introducing VoicePing MT v0.1 in a 100-row English-to-Japanese benchmark with GPT-5.5 quality judging, observed latency, and comparisons against DeepL, Sakana Translate, GPT-5 mini, Google Translate, Qwen, Azure, and Llama.
VoicePing translation is built for a simple but demanding job: make English conversations usable in Japanese, with wording people can trust, share, and act on. This benchmark evaluates that practical task: translating English into Japanese while preserving meaning and naturalness.
Benchmark Overview
| Model | Overall | Fluency | Naturalness | Accuracy | Completeness |
|---|---|---|---|---|---|
| DeepL | 89.4 | 91.1 | 89.0 | 88.7 | 89.7 |
| Sakana Translate | 88.0 | 86.7 | 83.2 | 88.5 | 90.4 |
| VoicePing MT v0.1 | 87.2 | 90.7 | 87.5 | 86.0 | 86.8 |
| GPT-5 mini | 87.0 | 88.4 | 85.5 | 86.9 | 87.1 |
| Google Translate | 86.7 | 87.1 | 83.0 | 86.4 | 88.6 |
| Qwen3.6-27B-FP8 dequant | 86.3 | 88.0 | 84.9 | 86.4 | 86.2 |
| Azure Translate | 79.2 | 78.5 | 73.3 | 79.1 | 82.6 |
| Llama 3.1 8B | 72.2 | 71.7 | 65.4 | 72.8 | 75.1 |
Evaluation Setup
We evaluated 100 English source rows from an internal dataset. There is no trusted Japanese gold reference for this set, so the benchmark uses a one-model-at-a-time GPT-5.5 judge. The judge reviewed the English source and one Japanese translation for each row.
The compared systems are DeepL, Sakana Translate, VoicePing MT v0.1, GPT-5 mini, Google Translate, Qwen3.6-27B-FP8, Azure Translate, and Llama 3.1 8B.
Each row is scored from 0 to 100 for fluency, naturalness, accuracy, and completeness. The headline overall score is computed as accuracy x 0.40 + completeness x 0.30 + fluency x 0.15 + naturalness x 0.15, so semantic fidelity and completeness matter more than style.
Latency is reported separately from quality. It is observed per-row generation latency from each candidate CSV, not a normalized production serving benchmark. API providers, local model hardware, batching, and network paths differ, so latency should be read as operational context rather than a quality score.
Quality Results
DeepL has the highest overall score at 89.4. Sakana Translate follows at 88.0, and VoicePing MT v0.1 stays close at 87.2, ahead of GPT-5 mini, Google Translate, Qwen, Azure Translate, and Llama 3.1 8B.
Fluency
DeepL leads fluency at 91.1, with VoicePing MT v0.1 close behind at 90.7, a 0.4-point gap.
Naturalness
DeepL also leads naturalness at 89.0. VoicePing MT v0.1 ranks next at 87.5, ahead of GPT-5 mini, Qwen, Google Translate, Sakana Translate, Azure Translate, and Llama 3.1 8B.
Accuracy
DeepL leads accuracy with 88.7, with Sakana Translate very close at 88.5. GPT-5 mini, Google Translate, Qwen, and VoicePing MT v0.1 form a tight group in the mid-86 range, while Azure Translate and Llama 3.1 8B trail behind.
Completeness
Sakana Translate leads completeness at 90.4. DeepL and Google Translate are next, while GPT-5 mini, VoicePing MT v0.1, and Qwen remain close together in the 86-87 range.
Observed Latency
Azure Translate and Google Translate are the fastest systems by observed median latency in this run. Latency is reported separately from quality because APIs, local models, hardware, and network paths are not normalized here.
| Model | Run type | Median latency | Mean latency | P95 latency |
|---|---|---|---|---|
| Azure Translate | API | 0.18s | 0.18s | 0.27s |
| Google Translate | API | 0.38s | 0.38s | 0.50s |
| DeepL | API | 1.18s | 1.21s | 1.37s |
| Sakana Translate | Hosted service | 1.92s | 2.07s | 3.37s |
| GPT-5 mini | API | 2.20s | 2.26s | 3.03s |
| VoicePing MT v0.1 | Local model | 2.70s | 2.73s | 3.80s |
| Llama 3.1 8B | Local model | 3.14s | 3.12s | 4.35s |
| Qwen3.6-27B-FP8 dequant | Local model | 3.96s | 4.19s | 6.09s |
What We Learned
- DeepL leads the quality ranking with 89.4 overall and the strongest fluency and naturalness scores.
- Sakana Translate follows at 88.0 overall and has the strongest completeness score in this run.
- VoicePing MT v0.1 remains close at 87.2 and is especially competitive on fluency and naturalness.
- GPT-5 mini, Google Translate, and Qwen3.6-27B-FP8 dequant form a tight middle cluster around 86-87 overall.
- Azure and Google are the fastest systems in this observed run, while local models are slower under this setup.
Conclusion
VoicePing MT v0.1 is already in the leading group for English-to-Japanese translation quality. Its strongest results are in fluency and naturalness, where it comes very close to DeepL and produces Japanese that reads smoothly rather than mechanically.
DeepL still leads overall, and Sakana Translate is especially strong on completeness. But VoicePing MT v0.1 shows that our own translation model can compete in the quality range that matters for real product use: preserving the English meaning, keeping the output complete, and producing Japanese that people can read and use with confidence.
This benchmark gives VoicePing a clear baseline for the next stage of model development. The priority is to keep improving accuracy and completeness while preserving the natural Japanese style that already makes MT v0.1 competitive.
