How Mandarin TTS Models Read Numbers: A 600-Sample Voice-Cloning Benchmark

Abstract

Numeric speech failures are semantic, not cosmetic. A TTS system can sound perfectly fluent while dropping zeros from an identifier, reading 9*14 as ninety-four, or stripping the exponent from an area measurement. Mandarin exposes this blind spot because the correct reading of 001000000066, 9*14, or 12cm² depends on the role the notation plays in the sentence—not merely on the symbols themselves.

We tested four voice-cloning model families—Fish Audio S2.1, CosyVoice3-RL, IndexTTS2, and Qwen3-TTS Base—using two Mandarin reference voices from our internal dataset. Each model-and-voice condition received the same 75 business-context sentences, for 600 generated audio samples in total.

On our automatic numeric exactness measure, Fish Audio S2.1 ranked first at 79.3% (119/150), ahead of CosyVoice3-RL at 74.7% (112/150). The headline hides sharply different behavior: CosyVoice3-RL reached 90.0% on identifiers, while Qwen3-TTS Base scored 0% on identifiers but led formulas and scientific units at 86.7% each.

This is an automatic screening benchmark, not a human listening test. Appendix B provides a separately generated listening set; those clips are not benchmark samples.

Why Numeric TTS Is a Semantic Problem

Reading a number correctly is not the same as pronouncing its digits clearly. A TTS model first has to infer the number’s role:

A fluent sentence can therefore contain a serious numeric error. For IDs, money, measurements, and formulas, this can be more damaging than a small pronunciation defect.

Benchmark at a Glance

Experiment Design

1. Build Five Balanced Prompt Groups

We used 75 Mandarin sentences per model-and-voice condition: 15 prompts in each of five numeric dimensions.

1. Decimals tested internal zeros, repeated digits, and trailing fractional zeros.
2. Identifiers used synthetic, non-attributable long digit strings with leading and repeated zeros that must not be collapsed into cardinal numbers.
3. Currencies covered USD, CNY, JPY, CAD, AUD, SGD, and HKD, including ambiguous $ and ¥ symbols.
4. Formulas covered multiplication and division, including equivalent readings such as 二分之三 and 三除以二.
5. Scientific and engineering units covered cm, µm, and nm.

Every carrier sentence was individually authored as a plausible business, engineering, logistics, or operations message rather than produced by swapping targets into one template.

2. Hold the Voice References Constant

All four model families cloned the same two Mandarin reference voices from our internal dataset.

Scores combine both references, producing 150 evaluated samples per model family.

3. Send the Display Sentence Exactly as Written

The benchmark deliberately did not expand digits before synthesis.

For example:

1

维修平台确认，本次设备校验码为001000000066。

We did not replace the number with the expected reading:

1

维修平台确认，本次设备校验码为零零一零零零零零零零六六。

This isolates the model’s text-normalization behavior. In production, pre-expanding critical numbers is often the safer design; here it would have removed the question we wanted to measure.

4. Generate and Normalize the Audio

The open-weight models ran from pinned checkpoints with deterministic seeds where supported. Every Fish Audio S2.1 request used normalize=true. The service ran in normal latency mode, so its scores describe Fish Audio S2.1 with the provider normalizer enabled.

Provider outputs were normalized to mono, 24 kHz, 16-bit PCM WAV, then verified for channel count, sample rate, sample width, and decodability.

All 600 generated samples passed this format check.

5. Transcribe and Score Numeric Meaning

Each generated file was transcribed with the same pinned Qwen/Qwen3-ASR-1.7B revision. We converted the ASR transcript and target into category-aware canonical values, then required an exact match. A missing zero, changed operator, conflicting currency, or lost exponent therefore failed even when the surrounding sentence was fluent. The category-specific rules are documented in Appendix A .

Results

The results aggregate the two internal reference voices. Currency outputs were rescored from the stored validator transcripts using the ambiguity-aware policy in Appendix A. Each combined score is based on 150 samples; each category score is based on 30.

Fish Audio S2.1 led overall at 79.3%, but no model won every category. The more useful result is how differently the models failed depending on whether the same digits represented an identifier, a decimal, a currency amount, a formula, or a unit.

Identifiers: The Largest Model Gap

Identifiers produced the widest score gap. The main failure was not pronunciation quality but sequence integrity: models grouped digits, omitted repeated zeros, or inserted an extra digit.

Decimals: Broadly Reliable

Decimals were the most reliable category. Their familiar spoken structure made them easier than identifiers, which require strict preservation of every position.

Currency: Separate the Amount from the Symbol

We judged the spoken amount and currency meaning separately. A bare $ does not uniquely specify a regional currency, while an explicit CAD or 加拿大元 requires a Canadian-dollar reading. In the scored samples, Fish Audio S2.1 read an explicit HK$100 as US dollars, while most CosyVoice3-RL failures changed the amount itself.

Formulas: Operators Are Easily Lost

A correct-sounding number could conceal a semantic failure: when the multiplication operator disappeared, 9*14 became 94.

Scientific and Engineering Units: Exponents Must Survive

The base value and unit could remain intelligible even when the squared exponent disappeared, changing an area measurement into a different meaning.

Model Takeaways

Reproducibility Snapshot

Limitations

• The judge is an ASR model, not a native-listener panel. ASR errors can be mistaken for TTS errors.
• The two internal reference voices do not represent all Mandarin speakers, accents, pitches, or recording conditions.
• The benchmark tests raw display notation. A production text-normalization layer can improve results substantially.
• Percentages come from 30 samples per model family and dimension after aggregating two voices; small differences should not be over-interpreted.
• Provider services and model implementations can change after the pinned evaluation date.

Conclusion

There was no universal winner for Mandarin numeric TTS.

Fish Audio S2.1 was the most balanced model family and the combined leader at 79.3%. CosyVoice3-RL was strongest for identifiers. IndexTTS2 was perfect on decimals but weaker on long digit sequences. Qwen3-TTS Base was strongest across the three symbol-heavy categories while failing every identifier item.

The practical lesson is broader than the ranking: numeric meaning is a separate reliability dimension. If a spoken number can change a payment, measurement, route, identity, or access decision, normalize it explicitly and verify it after synthesis.

Appendix A: Category-Specific Normalization Rules

Appendix B: Public Audio Examples

Why include listening examples? Aggregate scores show how often a model produced the intended reading, but not what its errors sound like. These examples make the main failure modes concrete: a missing or grouped digit, a changed decimal structure, an omitted operator, or a dropped exponent. Use them only as listening examples: this separate 20-clip demonstration set was not part of the 600-sample benchmark and did not contribute to the reported percentages.

Synthetic Reference Used for the Demos

All four systems used the same AI-generated Qwen3-TTS Serena clip as their voice reference. Listen to it first to establish the common voice that the demonstrations below attempt to reproduce.

Reference text: 今天我们将讨论项目进度、客户反馈与后续安排。请各位确认会议内容，并在结束后整理需要继续跟进的事项，感谢大家的配合。

Identifier Example

Input: 维修平台确认，本次设备校验码为001000000066。

Target: 零零一零零零零零零零六六

Decimal Example

Input: 值班工程师在电话会上报告：“主回路当前读数是3168.846。”

Target: 三千一百六十八点八四六

Currency Example

Input: 设计负责人提醒大家，加拿大外包插画的尾款还有$640未付。

Accepted meaning: the amount 640 plus a dollar-denominated reading

Formula Example

Input: 生成标签底稿时，把画布网格参数填写为9*14。

Target: 九乘以十四

Scientific and Engineering Unit Example

Input: 封装测试要求导热贴的接触面积达到12cm^2。

Target: 十二平方厘米