Multilingual OCR Benchmark: Hunyuan and Qwen Lead Accuracy, Paddle Leads Throughput

Models and references:

• PaddleOCR PP-OCRv5 — 3.1M–5.3M active parameters, depending on the selected language recognizer; mobile detector-recognizer OCR returning polygons and confidence.
• EasyOCR — 24.6M–74.6M active parameters, depending on the language reader; CRAFT detection plus PyTorch recognition returning regions and confidence.
• docTR — 26.4M parameters in the evaluated pretrained predictor; two-stage PyTorch document OCR returning structured regions and confidence.
• GLM-OCR — 1.33B parameters; vision-language OCR evaluated with Transformers generation for plain-text and Markdown extraction.
• Granite Docling 258M — 257.5M parameters; compact Transformers document conversion producing DocTags with layout locations.
• HunyuanOCR — 1.12B parameters; dedicated OCR vision-language generation covering text spotting, parsing, and semantic document tasks.
• Qwen3-VL-8B-Instruct — 8.77B parameters; general vision-language generation used for multilingual OCR and Markdown extraction.
• Surya OCR 2 — 686.2M parameters; layout-aware Transformers generation producing text or HTML with region geometry.

Counts describe the evaluated active checkpoint weights, not download size. Classic OCR totals combine the detector with the language-specific recognizer loaded for that page; generative-model totals use the published checkpoint tensors.

Abstract

We evaluated eight open OCR systems on the same 900-page suite: five languages, six image types, and 30 pages in every language-by-type cell. Every system ran on one NVIDIA RTX PRO 6000 Blackwell Max-Q GPU under a strict no-CPU-fallback policy.

There is no universal winner. HunyuanOCR recorded the lowest observed mean character error rate (CER) at 0.1637, with QwenOCR effectively tied at 0.1651. PaddleOCR delivered the strongest low-latency grounded profile: 0.145-second warm inference p50, 268 measured wall pages per minute, and polygons plus confidence.

The leading accuracy gaps are not statistically decisive in this sample. Deployment cost and output contract are clearer differentiators: Hunyuan reached 96,358 MiB peak VRAM, Qwen reached 21,010 MiB, while Paddle stayed below 4 GiB and returned usable geometry. We therefore report accuracy, latency, resources, truncation, and output capabilities separately rather than manufacturing one composite winner.

Evaluation Methodology

The benchmark uses a balanced page grid and keeps text accuracy separate from operational measurements:

Page Set

The source mix contains 548 controlled-rendered pages, 50 controlled-degraded pages, 195 MDPBench-origin public-real pages, and 107 licensed web-real Wikimedia Commons pages. All 900 pages contribute operational metrics. Text accuracy uses only pages with independent reference text; evaluated-model output was never reused as ground truth.

Inference Prompt Set

Detector-recognizer systems do not accept natural-language instructions, so PaddleOCR, EasyOCR, and docTR received the image plus the configured language reader. Generative systems received the same task objective—transcribe all visible text in reading order without summarizing—but used the output format native to each model.

Execution Controls

• One locked uv environment and pinned model revision per system.
• Inference used each model’s native CUDA path: PaddleOCR.predict() through PaddlePaddle GPU; EasyOCR.Reader.readtext() and docTR’s ocr_predictor() through PyTorch; and autoregressive generate() through Hugging Face Transformers for GLM-OCR, Granite Docling, HunyuanOCR, QwenOCR, and Surya OCR 2.
• The pinned Python ML stack used PaddleOCR 3.6.0 with PaddlePaddle GPU 3.3.1; EasyOCR 1.7.2; python-doctr 1.0.1; Surya OCR 0.21.1; PyTorch 2.11.0 and torchvision 0.26.0; Transformers 5.10.2–5.12.1 (plus the pinned Hunyuan-compatible revision); Accelerate 1.13.0–1.14.0; and Pillow 10.4.0–12.2.0.
• One persistent process per run; model load was recorded once and excluded from warm inference percentiles.
• Sequential single-page inference with no batching or concurrency.
• CUDA was required and automatic CPU fallback was forbidden.
• Generative decoding was deterministic and capped at 1,024 new tokens; reaching the cap without an end-of-sequence token counted as truncation.
• GPU and process resources were sampled around every page.

Metrics

• Completion and truncation: successful pages, failures, timeouts, and outputs that reached the generation cap. ↑ completion ↓ failures / truncations
• CER: character edit distance divided by reference length after Unicode NFKC and language-specific normalization. ↓ lower is better
• NormED and text similarity: edit distance divided by the longer of prediction and reference, plus its 1 − NormED similarity form. ↓ NormED ↑ similarity
• Latency: cold load, warm inference p50/p95, and end-to-end p50/p95. ↓ lower is better
• Throughput: successful wall pages per minute, including measured per-page orchestration overhead. ↑ higher is better
• Resources: peak VRAM and process-tree RAM, plus mean/peak GPU utilization and board power. ↓ peak memory
• Energy: active GPU-board Wh per page above the pre-process idle sample. ↓ lower is better
• Output contract: plain text, Markdown/HTML/DocTags, boxes, confidence, overlays, and emitted reading order.
• Official structure metrics: MDPBench text edit, formula CDM, table TEDS, and reading-order edit on the 195-page source-faithful subset. ↓ text / order edit ↑ CDM / TEDS

Results

The results move from the overall accuracy-speed trade-off to language, image type, domain shift, truncation, official structure scores, and GPU cost. We keep these dimensions separate because a low CER does not imply low latency, grounded output, or reliable document structure.

Overall accuracy, completion, and speed

The first view places mean per-page CER against warm inference p50; the most favorable region is the lower left. The table then adds completion, normalized edit distance, and generation-cap counts that the two-axis chart cannot show.

CER is mean per-page character edit distance divided by reference length. It can exceed 1.0 on a page when insertions outnumber reference characters. Normalized edit distance (NormED) divides by the longer of prediction and reference and remains bounded between 0 and 1.

docTR demonstrates why speed cannot be read alone: its default recognizer was extremely fast but used a Latin-oriented vocabulary that could not represent most CJK or Devanagari text.

Accuracy by language

This split checks whether the overall ranking survives a change in language and script. Each language aggregates 180 pages across all six image types.

Hunyuan recorded the lowest observed CER for English (0.1457), Korean (0.0889), and Hindi (0.2264); Qwen led Japanese (0.1425), and GLM led Chinese (0.1264). These are directional leaders: every paired top-versus-runner-up 95% bootstrap interval crossed zero.

Configuration matters as much as model size. Paddle and EasyOCR changed language readers, while the evaluated docTR checkpoint was Latin-oriented. Hunyuan reports training across more than 130 languages; Qwen3-VL reports broad multilingual document pretraining. GLM’s published multilingual evaluation does not list Hindi, and it capped 136 of 180 Hindi pages in this run.

Accuracy by image type

This view holds the language mix constant and groups the pages by document form. Each category contains 150 pages, making it possible to separate clean text recognition from layout, handwriting, screenshot, and degradation behavior.

Paddle led simple print (0.1650). Hunyuan led complex layouts (0.2716), tables/forms (0.1650), and controlled handwriting (0.0095). Qwen led screenshots/UI (0.0423) and camera/degraded pages (0.3014). Hunyuan and Qwen were nearly tied on complex and degraded pages, and all six top-versus-runner-up bootstrap intervals crossed zero.

The architectural split is useful: conventional detector-recognizer OCR excels on clean localized text, while page-level VLM context helps with layout and serialization. The handwriting result remains controlled-heavy and should not be generalized to natural handwriting from multiple writers.

Controlled-to-real domain shift

To expose benchmark optimism, we compare controlled-rendered pages with public-real MDPBench-origin pages. The groups are source-stratified rather than matched page pairs, so the chart measures a deployment-relevant domain gap rather than a causal degradation estimate.

Every system was substantially worse on the 195 public-real MDPBench-origin pages. Hunyuan moved from 0.057 CER to 0.477, Qwen from 0.051 to 0.523, GLM from 0.141 to 0.519, and Paddle from 0.152 to 0.587. This is source-stratified rather than a matched-pair experiment, but it clearly shows that rendered pages overstate deployed accuracy.

Token-limit effect

For generative OCR, a response that reaches the 1,024-token ceiling may be a partial transcription rather than a completed page. The chart separates completed and capped outputs so that this decoding limit is visible instead of being hidden inside aggregate CER.

Generative systems used deterministic decoding. Capped pages had approximately 0.40 higher CER than completed pages across every model. Surya reached the limit on 507 pages and Granite on 468; both require an official higher-cap rerun before their rankings are stable. Qwen capped 222 pages, Hunyuan 184, and GLM 263.

An actual page and inference result

The example below is MDPBench-origin page ocr900v2_en_tables_forms_10. Click either image to inspect it at full resolution.

Source page

PaddleOCR geometry overlay

The red regions come directly from PaddleOCR’s inference JSON; they were not recreated manually.

Paddle detected useful geometry but flattened the table into a reading order that lost cell relationships. The generative models reconstructed it more faithfully but did not emit confidence or grounded regions. Plain CER and document structure must remain separate metrics.

Cross-check with the official MDPBench evaluator

We passed the 195-page MDPBench-origin subset through the pinned official evaluator. This is a subset result, not a leaderboard submission. Direction arrows in the table headings identify how each official metric should be read.

Hunyuan led text edit distance, table TEDS, and reading order on this subset; GLM led formula CDM; Qwen stayed close on text and tables.

Latency and throughput

This visual answers the operational speed question independently of OCR accuracy. Inference percentiles exclude one-time model load, while wall throughput includes measured per-page end-to-end overhead in the sequential run. Filled dots mark p50, outlined dots mark p95, and every value remains printed beside its mark.

Memory and GPU-board energy

Peak VRAM determines whether a model fits the target GPU; active energy shows the measured GPU-board work per page during inference. These values describe this single-page, unbatched configuration and should not be treated as datacenter power estimates.

Energy is trapezoid-integrated sampled GPU-board power during each inference window. It excludes cold load, CPU and host power, cooling, and batching effects; it is not a wall-socket cost estimate.

Output contracts

Accuracy metrics score the transcription, but production systems also need to know what can be rendered or traced back to the page. This capability matrix distinguishes plain or structured text from geometry and confidence outputs in the evaluated model path.

A text-only RAG ingestion system and a click-to-highlight document viewer have different winners even if their CER is identical. Geometry and confidence are first-class deployment requirements, not secondary presentation features.

Deployment interpretation

• Fast multilingual text with geometry and confidence: PaddleOCR is the strongest evaluated baseline. It combines 0.145-second p50, 268 wall pages per minute, modest VRAM, and 0.2509 CER.
• Lowest observed plain-text error: HunyuanOCR and QwenOCR form the leading tier. Hunyuan has the lower mean CER; Qwen has the lower NormED and stronger observed screenshot/degraded results.
• Smaller generative OCR: GLM-OCR offers 0.2299 CER at 6.4 GiB peak VRAM and the best observed Chinese result, with no boxes in the tested base-model path.
• Structured HTML OCR: SuryaOCR cannot be judged from headline CER until its generation budget matches the expected output length.
• Maximum speed: docTR becomes compelling only with a recognizer configured for the required scripts. The default checkpoint in this run is not a five-language solution.

A practical production architecture may use a cascade: grounded PaddleOCR first, then a page-level VLM when confidence, language, layout, or table signals indicate that line OCR is insufficient.

Limitations

1. The suite remains 66.4% controlled content; every system was worse on public-real pages.
2. Pages without independent reference text contribute runtime data but not text-accuracy metrics.
3. Thirty pages per language/type cell expose large effects but cannot prove small winner gaps.
4. The 1,024-token ceiling disproportionately penalizes Surya and Granite.
5. CER does not fully score structure or geometry, and the runtime profile covers sequential single-page inference on one GPU only.

Next research direction

Conclusion

The benchmark separates three deployment choices that a single OCR score would hide.

PaddleOCR is the strongest low-latency grounded baseline. HunyuanOCR and QwenOCR provide the lowest observed text error when page-level reasoning matters, at much higher compute and memory cost. GLM-OCR is a smaller generative alternative with particularly strong Chinese performance. Surya, Granite, and docTR need corrected generation, vocabulary, or pipeline configurations before their current rows should be read as final capability rankings.

The central result is not that one system wins every page. Language, document structure, output contract, and GPU budget change the correct model choice—and controlled OCR accuracy is not a substitute for public-real evaluation.

Evaluation reference

• MDPBench dataset