4 Mandarin TTS Models Compared: A 600-Sample Number-Reading Benchmark

Abstract

Numeric speech failures are semantic, not cosmetic. A TTS system can sound perfectly fluent while dropping zeros from an identifier, reading 9*14 as ninety-four, or stripping the exponent from an area measurement. Mandarin exposes this blind spot because the correct reading of 001000000066, 9*14, or 12cm² depends on the role the notation plays in the sentence—not merely on the symbols themselves.

We tested four voice-cloning model families—Fish Audio S2.1, CosyVoice3-RL, IndexTTS2, and Qwen3-TTS Base—using two Mandarin reference voices from our internal dataset. Each model-and-voice condition received the same 75 business-context sentences, for 600 generated audio samples in total.

On our automatic numeric exactness measure, Fish Audio S2.1 ranked first at 79.3% (119/150), ahead of CosyVoice3-RL at 74.7% (112/150). The headline hides sharply different behavior: CosyVoice3-RL reached 90.0% on identifiers, while Qwen3-TTS Base scored 0% on identifiers but led formulas and scientific units at 86.7% each.

This is an automatic screening benchmark, not a human listening test. Appendix B provides a separately generated listening set; those clips are not benchmark samples.

Why Numeric TTS Is a Semantic Problem

Reading a number correctly is not the same as pronouncing its digits clearly. A TTS model first has to infer the number’s role:

A fluent sentence can therefore contain a serious numeric error. For IDs, money, measurements, and formulas, this can be more damaging than a small pronunciation defect.

Benchmark at a Glance

Experiment Design

1. Build Five Balanced Prompt Groups

We used 75 Mandarin sentences per model-and-voice condition: 15 prompts in each of five numeric dimensions.

1. Decimals tested internal zeros, repeated digits, and trailing fractional zeros.
2. Identifiers used synthetic, non-attributable long digit strings with leading and repeated zeros that must not be collapsed into cardinal numbers.
3. Currencies covered USD, CNY, JPY, CAD, AUD, SGD, and HKD, including ambiguous $ and ¥ symbols.
4. Formulas covered multiplication and division, including equivalent readings such as 二分之三 and 三除以二.
5. Scientific and engineering units covered cm, µm, and nm.

Every carrier sentence was individually authored as a plausible business, engineering, logistics, or operations message rather than produced by swapping targets into one template.

2. Hold the Voice References Constant

All four model families cloned the same two Mandarin reference voices from our internal dataset.

Scores combine both references, producing 150 evaluated samples per model family.

Holding these references constant reduces one source of variation: a model is not being given a different speaker from its competitors. It does not isolate the effect of reference quality or establish performance across Mandarin accents. The combined result also does not show whether one of the two references was easier to clone. Treat the voices as two controlled conditions, not a representative speaker survey.

3. Send the Display Sentence Exactly as Written

The benchmark deliberately did not expand digits before synthesis.

For example:

1

维修平台确认，本次设备校验码为001000000066。

We did not replace the number with the expected reading:

1

维修平台确认，本次设备校验码为零零一零零零零零零零六六。

This isolates the model’s text-normalization behavior. In production, pre-expanding critical numbers is often the safer design; here it would have removed the question we wanted to measure.

4. Generate and Normalize the Audio

The open-weight models ran from pinned checkpoints with deterministic seeds where supported. Every Fish Audio S2.1 request used normalize=true. The service ran in normal latency mode, so its scores describe Fish Audio S2.1 with the provider normalizer enabled.

Provider outputs were normalized to mono, 24 kHz, 16-bit PCM WAV, then verified for channel count, sample rate, sample width, and decodability.

All 600 generated samples passed this format check.

This check establishes that the files can enter the same downstream transcription pipeline. It does not verify the spoken content, naturalness, or similarity to the reference speaker. In particular, a valid WAV file can still contain a missing digit or an incorrect currency; those failures are evaluated separately in the next step.

5. Transcribe and Score Numeric Meaning

Each generated file was transcribed with the same pinned Qwen/Qwen3-ASR-1.7B revision. We converted the ASR transcript and target into category-aware canonical values, then required an exact match. A missing zero, changed operator, conflicting currency, or lost exponent therefore failed even when the surrounding sentence was fluent. The category-specific rules are documented in Appendix A .

Results

The results aggregate the two internal reference voices. Currency outputs were rescored from the stored validator transcripts using the ambiguity-aware policy in Appendix A. Each combined score is based on 150 samples; each category score is based on 30.

Fish Audio S2.1 led overall at 79.3%, but no model won every category. The more useful result is how differently the models failed depending on whether the same digits represented an identifier, a decimal, a currency amount, a formula, or a unit.

Identifiers: The Largest Model Gap

Identifiers produced the widest score gap. The main failure was not pronunciation quality but sequence integrity: models grouped digits, omitted repeated zeros, or inserted an extra digit.

For an order code or equipment identifier, dropping a leading zero changes the target even if the resulting number sounds natural. The relevant test is therefore whether every position survives, not whether a listener understands the approximate magnitude. The 30-item category combines 15 prompts and two reference voices; it does not cover every possible identifier length or pattern.

Decimals: Broadly Reliable

Decimals were the most reliable category. Their familiar spoken structure made them easier than identifiers, which require strict preservation of every position.

The scoring policy still demands more than an approximately correct value: fractional digits and trailing zeros must survive. IndexTTS2’s 30/30 result is a clean pass on this test set, not a guarantee for all measurement readouts. Longer fractional parts, different sentence contexts, and additional reference voices would need their own checks.

Currency: Separate the Amount from the Symbol

We judged the spoken amount and currency meaning separately. A bare $ does not uniquely specify a regional currency, while an explicit CAD or 加拿大元 requires a Canadian-dollar reading. In the scored samples, Fish Audio S2.1 read an explicit HK$100 as US dollars, while most CosyVoice3-RL failures changed the amount itself.

This policy makes the score deliberately narrower than full business-context understanding. A dollar reading can pass for bare $ even when surrounding context suggests a particular country. For payment messages, supply the currency explicitly and verify both it and the amount. Otherwise, an acceptable benchmark reading may still leave the recipient with an ambiguous instruction.

Formulas: Operators Are Easily Lost

A correct-sounding number could conceal a semantic failure: when the multiplication operator disappeared, 9*14 became 94.

We test whether the expression is read, not whether the model calculates it. Saying the result of a division therefore fails when the target requires the operands and operator. This distinction matters for dictating settings, dimensions, or formulas that another person must enter exactly; a numerically equivalent answer is not an equivalent instruction.

Scientific and Engineering Units: Exponents Must Survive

The base value and unit could remain intelligible even when the squared exponent disappeared, changing an area measurement into a different meaning.

The validator checks the value, base unit, and exponent together. A model cannot compensate for losing “square” by pronouncing the number and “centimeter” clearly. Qwen’s 26/30 result led this category, but four samples still failed the automatic check. Technical narration should preserve these elements explicitly instead of treating units as optional pronunciation details.

Model Takeaways

1. Fish Audio S2.1

Fish Audio is the hosted synthesis service in this comparison, evaluated as S2.1 with normalize=true in normal latency mode. Its result therefore includes the provider’s text-normalization behavior as well as speech generation. The developer documentation describes the service; current product names or defaults should not be assumed to reproduce this recorded configuration.

It achieved the highest combined exactness, 79.3% (119/150). Decimals and currencies both reached 93.3%, but identifiers fell to 66.7%, equivalent to 20 passes out of 30. Its errors were not limited to long codes: the scored HK$100 example changed an explicit Hong Kong-dollar amount into a US-dollar reading. The strongest aggregate score still left 31 samples non-exact under the automatic judge.

For mixed business narration, this result makes Fish a useful candidate to evaluate, while leaving clear work for the application. Expand critical identifiers digit by digit and specify regional currencies unambiguously before synthesis, then verify the resulting audio. That preprocessing is a proposed production safeguard, not an improvement measured in this raw-notation benchmark. Price, latency, and voice similarity were not ranked here.

2. CosyVoice3-RL

CosyVoice3-RL is the reinforcement-learning variant identified in the pinned checkpoint table. The official model card distinguishes the base and RL variants; this article’s results belong to the RL configuration, not every CosyVoice release. Like the other systems, it cloned the same two Mandarin references and received the unexpanded display sentences.

Its 74.7% combined exactness (112/150) concealed a sharp difference between categories: identifiers reached 90.0%, the best of the four models, while currency exactness was only 46.7%. That is 27 identifier passes versus 14 currency passes, each out of 30. Most currency failures changed the amount itself, so simply adding an explicit currency code would not address every observed failure.

An application dominated by order codes should investigate this identifier result, but a payment or invoice reader needs a separate acceptance test. Do not choose it for both tasks solely from its position in the combined ranking: it exceeded IndexTTS2 overall by only one passing sample. Keep numeric categories visible in evaluation and inspect failed audio, because the automatic ASR judge can also introduce transcription errors.

3. IndexTTS2

IndexTTS2 is a reference-conditioned speech-generation model whose official card describes zero-shot TTS with expressive and duration-control capabilities. Those features are outside this test’s scoring scope. Here, the question is whether its pinned voice-cloning configuration preserves the numeric meaning of the input when given the same references as the other systems.

Combined exactness was 74.0% (111/150). All 30 decimal samples passed, and currencies reached 93.3%, but identifiers passed only 12 of 30 cases, or 40.0%. Formulas reached 70.0% and units 66.7%. This pattern shows why a successful demonstration reading ordinary measurements is insufficient evidence for a logistics application that also reads long shipment or equipment codes.

For a measurement-heavy workflow, preserve the promising decimal result as a reason to run a broader evaluation. Include new values, longer fractional parts, additional reference voices, and units with exponents. Handle identifiers as an explicitly different input type. The perfect decimal score establishes what happened in these 30 samples; it does not measure the reliability of arbitrary numbers, emotional delivery, or the separately generated public demo clips below.

4. Qwen3-TTS Base

The tested checkpoint is Qwen3-TTS-12Hz-1.7B-Base , used for reference-based voice cloning. It is distinct from the Qwen CustomVoice and VoiceDesign variants. The 1.7B size and Base designation matter for reproduction; a result from another checkpoint or voice-generation mode should not be substituted for this condition.

Combined exactness was 70.7% (106/150). Qwen led formulas and scientific units at 86.7% each, or 26/30, and also reached 86.7% on currencies. It did not lead currencies: Fish and IndexTTS2 both scored 93.3%. All 30 identifier samples failed, making the combined percentage especially uninformative for anyone building a code-reading workflow.

This is a candidate to investigate for symbol-heavy technical narration, with explicit safeguards around digit sequences. The identifier result applies to raw notation in these prompts and reference conditions; it does not prove that every possible identifier will fail or that explicit spoken-digit input cannot work. That alternative was not tested. Use a separate identifier path and evaluate its output, while retaining the formula operator and unit exponent in the expected answer.

Reproducibility Snapshot

Limitations

• The judge is an ASR model, not a native-listener panel. ASR errors can be mistaken for TTS errors.
• The two internal reference voices do not represent all Mandarin speakers, accents, pitches, or recording conditions.
• The benchmark tests raw display notation. A production text-normalization layer can improve results substantially.
• Percentages come from 30 samples per model family and dimension after aggregating two voices; small differences should not be over-interpreted.
• Provider services and model implementations can change after the pinned evaluation date.

Conclusion

There was no universal winner for Mandarin numeric TTS.

Fish Audio S2.1 was the combined leader at 79.3%. CosyVoice3-RL led identifiers, and IndexTTS2 passed all 30 decimal samples. Qwen3-TTS Base led formulas and scientific units while failing every identifier item. Fish and IndexTTS2 tied for the highest currency score.

The practical lesson is broader than the ranking: numeric meaning is a separate reliability dimension. If a spoken number can change a payment, measurement, route, identity, or access decision, normalize it explicitly and verify it after synthesis.

Appendix A: Category-Specific Normalization Rules

Appendix B: Public Audio Examples

Why include listening examples? Aggregate scores show how often a model produced the intended reading, but not what its errors sound like. These examples make the main failure modes concrete: a missing or grouped digit, a changed decimal structure, an omitted operator, or a dropped exponent. Use them only as listening examples: this separate 20-clip demonstration set was not part of the 600-sample benchmark and did not contribute to the reported percentages.

Synthetic Reference Used for the Demos

All four systems used the same AI-generated Qwen3-TTS Serena clip as their voice reference. Listen to it first to establish the common voice that the demonstrations below attempt to reproduce.

Reference text: 今天我们将讨论项目进度、客户反馈与后续安排。请各位确认会议内容，并在结束后整理需要继续跟进的事项，感谢大家的配合。

Identifier Example

Input: 维修平台确认，本次设备校验码为001000000066。

Target: 零零一零零零零零零零六六

Decimal Example

Input: 值班工程师在电话会上报告：“主回路当前读数是3168.846。”

Target: 三千一百六十八点八四六

Currency Example

Input: 设计负责人提醒大家，加拿大外包插画的尾款还有$640未付。

Accepted meaning: the amount 640 plus a dollar-denominated reading

Formula Example

Input: 生成标签底稿时，把画布网格参数填写为9*14。

Target: 九乘以十四

Scientific and Engineering Unit Example

Input: 封装测试要求导热贴的接触面积达到12cm^2。

Target: 十二平方厘米