8 English-to-Japanese Translation Models Compared: VoicePing MT v0.1

VoicePing translation is built for a simple but demanding job: make English conversations usable in Japanese, with wording people can trust, share, and act on. This benchmark evaluates that practical task: translating English into Japanese while preserving meaning and naturalness.

Benchmark Overview

Translation systems compared

Numbering is an article index, not a score ranking. DeepL has the highest measured overall score. All scores below come from the same 100 English source rows and automated GPT-5.5 judge; there is no trusted human Japanese reference. Timing includes different API and local serving paths.

1. VoicePing MT v0.1

Overall quality: 87.2/100 · Observed median latency: 2.70 seconds · Local model.

VoicePing’s fluency (90.7) and naturalness (87.5) were its strongest component scores. Accuracy was 86.0 and completeness 86.8, below DeepL on both dimensions. Because those two semantic dimensions account for 70% of the overall weighting, matching DeepL’s fluency and naturalness scores alone would not close the overall gap. This profile comes first because the article introduces our model, not because it won the comparison.

The local run had a 3.80-second P95 (95th-percentile) latency, compared with its 2.70-second median. Neither figure includes evidence about a full live translation experience with speech recognition and speech output. For the next model iteration, inspect whether subjects, negation, amounts and qualifying clauses survive translation before polishing style. Those are evaluation priorities, not newly observed error counts. The results support a development baseline; the single automated judge and missing human reference do not establish publication-ready Japanese or a statistically proven advantage over the nearby middle group.

2. DeepL

Overall quality: 89.4/100 · Observed median latency: 1.18 seconds · API.

DeepL’s translation API had the highest overall, fluency (91.1), naturalness (89.0) and accuracy (88.7) point estimates here. Completeness was 89.7, slightly below Sakana’s 90.4. That makes it the strongest measured all-round baseline in this dataset, while showing that even the overall leader did not lead every dimension. The article does not report uncertainty estimates for those differences.

DeepL’s observed P95 was 1.37 seconds, below the local models’ values in this run. Current API documentation exposes context, glossary and model-selection options; the public benchmark does not specify enough request detail to attribute this result to a particular combination. Do not assume the score applies equally to every DeepL mode or that an untested glossary would improve it. A useful follow-up would hold the source rows and settings fixed, then separately assess terminology consistency and human editing effort.

3. Sakana Translate

Overall quality: 88.0/100 · Observed median latency: 1.92 seconds · Hosted service.

Sakana Translate is a hosted translation experience built on Sakana’s Namazu model family. Its release describes translation, proofreading and follow-up questions for Japanese, English and Chinese. The benchmark row represents the evaluated translation outputs; it does not score the proofreading or question-answering workflows, and it should not be read as evidence of a particular API deployment.

Sakana led completeness at 90.4 and had an accuracy score of 88.5, close to DeepL’s 88.7 point estimate. Its naturalness score was lower at 83.2. This combination suggests a useful follow-up question: does the output preserve details while requiring more Japanese editing? That is an interpretation of the aggregate scores, not a measured editing-time result. Observed P95 latency was 3.37 seconds. Evaluate both the first translation and any subsequent revision as separate steps rather than assuming the hosted interactive features were included in this timing.

4. GPT-5 mini

Overall quality: 87.0/100 · Observed median latency: 2.20 seconds · API.

GPT-5 mini is a general-purpose reasoning model used here through an API. Its accuracy (86.9) and completeness (87.1) were close, while naturalness was 85.5. The 87.0 overall score lies in the narrow middle group. A few tenths of a point in this one-judge evaluation do not establish a stable ordering among GPT-5 mini, VoicePing, Google and Qwen.

A prompted model gives the application control over instructions, but that also makes the exact prompt and generation settings part of the tested system. The article does not publish enough detail to reproduce or optimize those choices, so it cannot show that prompting changes would improve the score. Its observed P95 was 3.03 seconds. For a translation-only integration, test that the response preserves the full source and contains only the requested translation, and score any format failures explicitly; this publication does not report that failure rate.

5. Google Translate

Overall quality: 86.7/100 · Observed median latency: 0.38 seconds · API.

The Google API row combined a middle-group quality score with short observed latency: its P95 was 0.50 seconds. Completeness was 88.6, whereas naturalness was 83.0. That difference matters for workflow design: retaining the source information and producing polished Japanese are separate goals, even when the overall score is similar to a competing model.

Google Cloud Translation offers multiple APIs and model options. The public result labels the service without identifying a complete endpoint/model configuration, so it should not be attributed to the consumer Google Translate app or every current Cloud translation option. For high-volume text processing, the recorded latency makes this row worth further evaluation, but this run does not measure sustained throughput or total cost. Review terminology and register with native readers before deciding whether the lower naturalness point estimate translates into additional editing work.

6. Qwen3.6-27B-FP8 dequant

Overall quality: 86.3/100 · Observed median latency: 3.96 seconds · Local model.

The Qwen3.6-27B-FP8 model card describes FP8 weights; the article’s row specifically records a dequantized local configuration. Keep that distinction attached to the result. It scored 88.0 for fluency, 84.9 for naturalness, 86.4 for accuracy and 86.2 for completeness, placing it near other middle-group systems on this automated evaluation.

The local run had the longest observed median and P95 latencies, with P95 reaching 6.09 seconds. That is a result for this serving path, not evidence that every Qwen deployment is slower than every API. Hardware, batching, precision and generation settings were not normalized, and dequantization alone does not explain the timing. For teams evaluating local operation, record those settings and measure memory and concurrent load separately. The current table supports a quality-and-latency baseline; it does not establish a capacity requirement or an optimal quantization choice.

7. Azure Translate

Overall quality: 79.2/100 · Observed median latency: 0.18 seconds · API.

Azure had the shortest observed median and P95 latencies in the table, at 0.18 and 0.27 seconds. Its component scores were also below the leading group: fluency 78.5, naturalness 73.3, accuracy 79.1 and completeness 82.6. Speed and quality therefore need separate acceptance criteria; a fast response is not automatically useful if the intended Japanese requires substantial correction.

Microsoft Translator includes text translation and customization options. This generic benchmark label does not establish that a domain-customized system, every current model option or document translation was evaluated. The public record also does not identify individual failure examples, so the lower scores should not be turned into a claim that a specific grammar or terminology error dominated. Test representative domain sentences with native reviewers, and measure any customization benefit in a new controlled comparison rather than assigning it to this row.

8. Llama 3.1 8B

Overall quality: 72.2/100 · Observed median latency: 3.14 seconds · Local model.

The Llama 3.1 8B Instruct model card lists eight supported languages, and Japanese is not among them. That is important context for an English-to-Japanese evaluation. It does not mean the model is incapable of generating Japanese, but its inclusion here should be understood as an exploratory local baseline rather than validation of an officially supported Japanese workflow.

It had the lowest overall point estimate in this set, with naturalness at 65.4, accuracy at 72.8 and completeness at 75.1. The run was not the fastest local option either: median latency was 3.14 seconds and P95 4.35 seconds. Smaller parameter count alone therefore did not establish the best quality/speed choice. These observations do not cover Japanese-adapted derivatives or later Llama models. Any adaptation would need its own evaluation; do not assume a different prompt, fine-tune or serving runtime inherits this score or necessarily resolves the gap.

Evaluation Setup

We evaluated 100 English source rows from an internal dataset. There is no trusted Japanese gold reference for this set, so the benchmark uses a one-model-at-a-time GPT-5.5 judge. The judge reviewed the English source and one Japanese translation for each row.

The compared systems are VoicePing MT v0.1, DeepL, Sakana Translate, GPT-5 mini, Google Translate, Qwen3.6-27B-FP8, Azure Translate, and Llama 3.1 8B.

Each row is scored from 0 to 100 for fluency, naturalness, accuracy, and completeness. The headline overall score is computed as accuracy x 0.40 + completeness x 0.30 + fluency x 0.15 + naturalness x 0.15, so semantic fidelity and completeness matter more than style.

Latency is reported separately from quality. It is observed per-row generation latency from each candidate CSV, not a normalized production serving benchmark. API providers, local model hardware, batching, and network paths differ, so latency should be read as operational context rather than a quality score.

Quality Results

DeepL has the highest overall score at 89.4. Sakana Translate follows at 88.0, and VoicePing MT v0.1 stays close at 87.2, ahead of GPT-5 mini, Google Translate, Qwen, Azure Translate, and Llama 3.1 8B.

Fluency

DeepL leads fluency at 91.1, with VoicePing MT v0.1 close behind at 90.7, a 0.4-point gap.

Fluency concerns whether the Japanese is grammatically readable. A sentence can read smoothly while changing a name, quantity or condition, so this dimension should not substitute for checking meaning. The 0.4-point difference is a judge-score observation; no human preference test or significance result is reported.

Naturalness

DeepL also leads naturalness at 89.0. VoicePing MT v0.1 ranks next at 87.5, ahead of GPT-5 mini, Qwen, Google Translate, Sakana Translate, Azure Translate, and Llama 3.1 8B.

Naturalness asks whether the wording feels appropriate in Japanese, beyond being grammatical. Register, phrasing and the intended audience matter. The automated score can identify candidates for closer review, but a native reviewer still needs to assess whether the tone fits a customer email, a meeting caption or an internal note.

Accuracy

DeepL leads accuracy with 88.7, with Sakana Translate very close at 88.5. GPT-5 mini, Google Translate, Qwen, and VoicePing MT v0.1 form a tight group in the mid-86 range, while Azure Translate and Llama 3.1 8B trail behind.

Accuracy receives 40% of the overall weight, the largest contribution. In a practical review, compare who did what, negation, numbers and conditions against the English source. These are useful review categories, not published error counts. An 88.7 judge score also does not mean that exactly 88.7% of sentences were correct.

Completeness

Sakana Translate leads completeness at 90.4. DeepL and Google Translate are next, while GPT-5 mini, VoicePing MT v0.1, and Qwen remain close together in the 86-87 range.

Completeness measures whether the source information is retained. A concise translation can be fluent while omitting a qualification or a second action, which is why this category contributes 30% of the overall score. The aggregate does not reveal which details were missing or whether errors clustered in particular domains; sentence-level review is needed for that diagnosis.

Observed Latency

Azure Translate and Google Translate are the fastest systems by observed median latency in this run. Latency is reported separately from quality because APIs, local models, hardware, and network paths are not normalized here.

What We Learned

• DeepL leads the quality ranking with 89.4 overall and the strongest fluency and naturalness scores.
• Sakana Translate follows at 88.0 overall and has the strongest completeness score in this run.
• VoicePing MT v0.1 remains close at 87.2 and is especially competitive on fluency and naturalness.
• GPT-5 mini, Google Translate, and Qwen3.6-27B-FP8 dequant form a tight middle cluster around 86-87 overall.
• Azure and Google are the fastest systems in this observed run, while local models are slower under this setup.

Conclusion

DeepL recorded the highest overall score on these 100 rows, and Sakana Translate led completeness. VoicePing MT v0.1 was close on fluency and naturalness, while its accuracy and completeness scores leave clear room for improvement.

These are useful model-development signals, not a human certification of usable Japanese. The evaluation uses one automated judge and provides no trusted Japanese reference or uncertainty analysis. A product decision should combine this baseline with native review of representative text and measurements from the intended serving setup.

This benchmark gives VoicePing a clear baseline for the next stage of model development. The priority is to keep improving accuracy and completeness while preserving the natural Japanese style that already makes MT v0.1 competitive.

References

• DeepL API
• Sakana Translate release
• Google Cloud Translation
• Azure AI Translator
• OpenAI API
• Qwen3.6-27B-FP8
• Meta Llama 3.1 8B Instruct

Scope and reproducibility

VoicePing publishes this evaluation of its own model. The results describe the dataset, model labels and scoring method reported above; they are not an independent certification or a guarantee for every language, recording or product workflow.

The article does not provide a complete public rerun package with all source data, exact service versions, requests, scoring code and environment details. Where those details are absent, do not infer them or treat a small score difference as statistically established. To evaluate your use case, test representative permitted samples, record the model versions and settings, and review errors as well as aggregate scores.