GPT-5.5 evaluation ke saath 100-row English-to-Japanese benchmark: VoicePing MT v0.1 ko DeepL, Sakana Translate, GPT-5 mini, Google Translate, Qwen, Azure aur Llama se quality aur dekhi gayi latency par compare kiya gaya.
Introducing VoicePing Translation Model MT V0.1
VoicePing ka translation model ek seedhe lekin mushkil kaam ke liye banaya gaya hai: English content ko aisi Japanese mein badalna jis par bharosa kiya ja sake, jise share kiya ja sake, aur jise seedhe use kiya ja sake. Yeh benchmark isi practical use case ko evaluate karta hai: English text ko Japanese mein translate karna, meaning ko bachate hue aur output ko natural banate hue.
Benchmark ka saaransh
| Model | Kul score | Pravah (Fluency) | Swabhavikta (Naturalness) | Sateekta (Accuracy) | Poornata (Completeness) |
|---|---|---|---|---|---|
| DeepL | 89.4 | 91.1 | 89.0 | 88.7 | 89.7 |
| Sakana Translate | 88.0 | 86.7 | 83.2 | 88.5 | 90.4 |
| VoicePing MT v0.1 | 87.2 | 90.7 | 87.5 | 86.0 | 86.8 |
| GPT-5 mini | 87.0 | 88.4 | 85.5 | 86.9 | 87.1 |
| Google Translate | 86.7 | 87.1 | 83.0 | 86.4 | 88.6 |
| Qwen3.6-27B-FP8 dequant | 86.3 | 88.0 | 84.9 | 86.4 | 86.2 |
| Azure Translate | 79.2 | 78.5 | 73.3 | 79.1 | 82.6 |
| Llama 3.1 8B | 72.2 | 71.7 | 65.4 | 72.8 | 75.1 |
Evaluation setup
Humne internal dataset se 100 English source rows evaluate ki. Is dataset ke liye reliable Japanese reference translation nahi tha, isliye GPT-5.5 ko single-model judge ke roop mein use kiya gaya: har evaluation mein sirf English source aur uski Japanese translation thi.
Comparison mein DeepL, Sakana Translate, VoicePing MT v0.1, GPT-5 mini, Google Translate, Qwen3.6-27B-FP8, Azure Translate aur Llama 3.1 8B shamil the.
Har row ko sateekta (Accuracy), poornata (Completeness), pravah (Fluency), aur swabhavikta (Naturalness) par 0 se 100 tak score diya gaya. Kul score sateekta (Accuracy) x 0.40 + poornata (Completeness) x 0.30 + pravah (Fluency) x 0.15 + swabhavikta (Naturalness) x 0.15 se nikala gaya, isliye meaning fidelity aur information completeness ko style se zyada weight diya gaya.
Latency ko quality score se alag dikhaya gaya hai. Yeh har candidate CSV ke latency_ms se liya gaya per-row observed time hai, normalized production benchmark nahi. API providers, local hardware aur network paths alag-alag hain.
Quality results
DeepL 89.4 kul score ke saath sabse upar hai. Sakana Translate 88.0 ke saath second hai, aur VoicePing MT v0.1 87.2 ke saath top group ke kaafi paas hai, GPT-5 mini, Google Translate, Qwen, Azure Translate aur Llama 3.1 8B se aage.
Pravah (Fluency)
Pravah (Fluency) mein DeepL 91.1 ke saath lead karta hai. VoicePing MT v0.1 90.7 ke saath bahut kareeb hai; difference sirf 0.4 point hai.
Swabhavikta (Naturalness)
Swabhavikta (Naturalness) mein bhi DeepL 89.0 ke saath lead karta hai. VoicePing MT v0.1 87.5 ke saath second hai, GPT-5 mini, Qwen, Google Translate, Sakana Translate, Azure Translate aur Llama 3.1 8B se upar.
Sateekta (Accuracy)
Sateekta (Accuracy) mein DeepL 88.7 ke saath lead karta hai, aur Sakana Translate 88.5 ke saath bahut kareeb hai. GPT-5 mini, Google Translate, Qwen aur VoicePing MT v0.1 lagbhag 86 points ke aas-paas ek tight group banate hain.
Poornata (Completeness)
Poornata (Completeness) mein Sakana Translate 90.4 ke saath lead karta hai. DeepL aur Google Translate uske baad aate hain; GPT-5 mini, VoicePing MT v0.1 aur Qwen 86 se 87 points ke beech kaafi kareeb hain.
Dekhi gayi latency
Is run mein median latency ke hisaab se Azure Translate aur Google Translate sabse tez rahe. Latency ko quality se alag padhna chahiye, kyunki API, local model, hardware aur network paths normalized nahi hain.
| Model | Run type | Median latency | Mean latency | P95 latency |
|---|---|---|---|---|
| Azure Translate | API | 0.18s | 0.18s | 0.27s |
| Google Translate | API | 0.38s | 0.38s | 0.50s |
| DeepL | API | 1.18s | 1.21s | 1.37s |
| Sakana Translate | Hosted service | 1.92s | 2.07s | 3.37s |
| GPT-5 mini | API | 2.20s | 2.26s | 3.03s |
| VoicePing MT v0.1 | Local model | 2.70s | 2.73s | 3.80s |
| Llama 3.1 8B | Local model | 3.14s | 3.12s | 4.35s |
| Qwen3.6-27B-FP8 dequant | Local model | 3.96s | 4.19s | 6.09s |
Mukhya baatein
- DeepL 89.4 kul score ke saath sabse aage hai aur pravah (Fluency) aur swabhavikta (Naturalness) mein sabse high score leta hai.
- Sakana Translate 88.0 ke saath second hai aur is run mein poornata (Completeness) mein sabse upar hai.
- VoicePing MT v0.1 87.2 ke saath top group ke paas hai aur khas taur par pravah (Fluency) aur swabhavikta (Naturalness) mein strong hai.
- GPT-5 mini, Google Translate aur Qwen3.6-27B-FP8 dequant 86 se 87 ke aas-paas ek close middle group banate hain.
- Is setup mein Azure Translate aur Google Translate sabse tez hain; local models is run mein slower rahe.
Nishkarsh
VoicePing MT v0.1 English-to-Japanese translation quality mein ab leading group mein aa chuka hai. Iske strongest results pravah (Fluency) aur swabhavikta (Naturalness) mein hain, jahan yeh DeepL ke bahut kareeb hai aur aisa Japanese output deta hai jo smooth padhta hai, mechanical nahi lagta.
DeepL abhi bhi overall lead karta hai, aur Sakana Translate poornata (Completeness) mein khas taur par strong hai. Lekin VoicePing MT v0.1 dikhata hai ki VoicePing ka apna translation model real product use ke liye zaroori quality range mein compete kar sakta hai: English meaning ko preserve karna, output ko complete rakhna, aur aisa Japanese dena jise log confidence ke saath padh aur use kar saken.
Yeh benchmark VoicePing ko model development ke next stage ke liye clear baseline deta hai. Priority hai sateekta (Accuracy) aur poornata (Completeness) ko aur improve karna, saath hi natural Japanese style ko preserve rakhna jo MT v0.1 ko already competitive banata hai.
