Offline Speech Transcription Benchmark: 16 Models Across Android, iOS, macOS, and Windows

Source Code:

• android-offline-transcribe — Android benchmark app with 16 ASR models and 6 inference engines
• ios-mac-offline-transcribe — iOS and macOS benchmark app with 17 ASR models and 9 inference engines
• windows-offline-transcribe — Windows benchmark app with 12 ASR models, CPU-only

Abstract

We benchmark 16 on-device speech-to-text models across 9 inference engines on Android, iOS, macOS, and Windows, measuring inference speed (tok/s), real-time factor (RTF), and memory footprint. This benchmark measures speed only — transcription accuracy (WER/CER) is not evaluated. Key findings: the choice of inference engine can change performance by 51x for the same model (sherpa-onnx vs whisper.cpp on Android); Moonshine Tiny and SenseVoice Small achieve the fastest inference across platforms; and WhisperKit CoreML crashes on 4 GB iOS devices for models above Whisper Tiny. All benchmark apps and results are open-source.

Motivation

Developers building voice-enabled edge applications face a combinatorial selection problem: dozens of ASR models (from 31 MB Whisper Tiny to 1.8 GB Qwen3 ASR), multiple inference engines (ONNX Runtime, CoreML, whisper.cpp, MLX), and 4+ target platforms — each combination producing vastly different speed and memory characteristics. Published model benchmarks typically report results on server GPUs, not the consumer mobile and laptop hardware where these models actually deploy.

This benchmark addresses the deployment choice problem directly: which model + engine combination delivers real-time transcription on each target platform, and what are the memory constraints? The results enable developers to select a model/engine pair based on measured data from their target device class, rather than extrapolating from GPU benchmarks.

Methodology

Android, iOS, and macOS benchmarks use the same 30-second WAV file containing a looped segment of the JFK inaugural address (16 kHz, mono, PCM 16-bit). Windows benchmarks use an 11-second excerpt from the same source (see Windows section for details).

Metrics (speed-only — this benchmark does not measure transcription accuracy/WER):

• Inference: Wall-clock time from engine call to result
• tok/s: Output words per second (higher = faster)
• RTF: Real-Time Factor — ratio of processing time to audio duration (below 1.0 = faster than real-time)

Devices:

Android Results

Device: Samsung Galaxy S10, Android 12, API 31

15/16 PASS — 0 OOM conditions. The sole failure (Omnilingual 300M) is a known model limitation with English language detection.

Android Engine Comparison: Same Model, Different Backends

The Whisper Tiny model shows dramatic performance differences depending on the inference backend:

sherpa-onnx is 51x faster than whisper.cpp for the same Whisper Tiny model on Android — a critical finding for developers choosing an inference runtime.

iOS Results

Device: iPad Pro 3rd gen, A12X Bionic, 4 GB RAM

WhisperKit OOM warning: On 4 GB devices, WhisperKit CoreML crashes (OOM) for Whisper Base and above. The tok/s values shown for OOM models were measured before the crash occurred and do not represent complete successful runs. whisper.cpp handles the same models without OOM, though at lower throughput.

macOS Results

Device: MacBook Air M4, 32 GB RAM

macOS has no OOM issues — with 32 GB RAM, all models including Whisper Large v3 Turbo run successfully via WhisperKit CoreML.

Windows Results

Device: Intel Core i5-1035G1 @ 1.00 GHz (4C/8T), 8 GB RAM, CPU-only

Tested with an 11-second JFK inauguration audio excerpt (22 words). All models run CPU-only on x86_64 (i5-1035G1, 4C/8T) — no GPU acceleration. Parakeet TDT v2 is notable for combining fast inference (17.8 words/s) with full punctuation output.

Limitations

• Speed only: This benchmark measures inference speed, not transcription accuracy (WER/CER). Accuracy varies by model, language, and audio conditions — developers should evaluate accuracy separately for their target use case.
• Single audio sample: All platforms use a single JFK inaugural address recording. Results may differ with other audio characteristics (noise, accents, domain-specific vocabulary).
• Windows clip length: Windows uses an 11-second audio clip vs 30 seconds on other platforms, so cross-platform speed comparisons should account for this difference.
• iOS OOM values: The tok/s values shown for OOM-marked iOS models were measured before the crash and do not represent complete successful runs.

Further Research

• Add accuracy benchmarks: Pair speed metrics with WER/CER across multilingual datasets and noisy speech conditions.
• Expand audio conditions: Add long-form audio, overlapping speakers, and domain vocabulary (meetings, call-center, industrial) instead of one speech sample.
• Quantization sweep: Benchmark INT8/INT4 and mixed-precision variants across engines to map memory/speed/accuracy trade-offs.
• Smaller Qwen3-ASR variants: Include Qwen3-ASR-0.6B and any future sub-1B checkpoints (for example, a potential 0.8B release) to test whether speed gains justify quality loss.
• Power and thermal profiling: Add battery drain and sustained-performance measurements for continuous on-device transcription workloads.

Conclusion

On-device speech transcription achieves faster-than-real-time speeds across all major mobile and desktop platforms. The choice of inference engine matters as much as model selection — sherpa-onnx on Android and CoreML on Apple deliver 10–50x speedups over naive CPU inference. Among the fastest models, Moonshine Tiny (English) or Whisper Tiny (multilingual) with the right engine achieves real-time transcription at minimal memory cost. Model selection should also consider transcription accuracy (WER), which was not measured in this benchmark.

References

Our Repositories:

• android-offline-transcribe — Android benchmark app (Apache 2.0)
• ios-mac-offline-transcribe — iOS/macOS benchmark app (Apache 2.0)
• windows-offline-transcribe — Windows benchmark app (Apache 2.0)

Models:

• Moonshine Tiny/Base — Useful Sensors, English-only
• Whisper Tiny/Base/Small/Turbo — OpenAI, 99 languages
• SenseVoice Small — FunAudioLLM, 5 languages
• Parakeet TDT 0.6B v3 — NVIDIA NeMo, 25 European languages
• Qwen3 ASR 0.6B — Alibaba Qwen, 30 languages
• Zipformer Streaming — Next-gen Kaldi, English
• Omnilingual 300M — MMS, 1,600+ languages

Inference Engines:

• sherpa-onnx — Next-gen Kaldi ONNX Runtime
• whisper.cpp — C/C++ port of OpenAI Whisper
• WhisperKit — CoreML Whisper for Apple
• MLX — Apple Machine Learning framework