Whisper Model Comparison: Speed vs Accuracy Trade-offs for Dictation

Many dictation users pick the largest Whisper model available, assume it gives the best possible transcription, and never revisit the choice. That assumption is understandable. Bigger models usually are more accurate. But "most accurate on a benchmark" and "best for the way you dictate" are different questions, and the answer depends on your content, your Mac, and how long you are willing to wait for text.

This guide covers what OpenAI actually publishes about each Whisper model size, what changes as you move up the ladder, and how to run a small benchmark of your own so the decision rests on your audio rather than someone else's.

What OpenAI Publishes About Each Whisper Size

OpenAI's Whisper repository lists six model sizes, from tiny to large, plus a newer turbo model. For each one it gives the parameter count, the approximate memory required, and a speed figure relative to the large model, measured by OpenAI on an A100 GPU transcribing English speech.^[1]

Two notes from the same source matter for dictation. First, the English-only ".en" models "tend to perform better, especially for the tiny.en and base.en models," so an English-only dictation workflow can get more from a small model by choosing its .en variant.^[1] Second, the turbo model "is not trained for translation tasks," so if you rely on speech translation you need a multilingual model instead.^[1]

The relative speed column is a ratio, not a promise about your Mac. It tells you that tiny is roughly ten times faster than large on the same hardware, but not how many seconds a paragraph will take on a MacBook Air. That depends on the runtime and how it uses the chip, which is why measuring on your own machine matters.

How the Models Actually Differ Beyond Parameter Count

Parameter count is the number people fixate on, but what you feel during dictation is the combination of accuracy, latency, and memory pressure.

Tiny and base are small enough to run almost anywhere and respond quickly. They are useful for short commands, quick notes, and testing a workflow, but expect to correct more words on longer or more technical passages.

Small is often the first size where everyday conversational dictation becomes comfortable, particularly with the English-only small.en variant.

Medium adds considerably more capacity at roughly twice the memory of small in OpenAI's table. For many people it is the point where uncommon vocabulary and accents start to land more reliably.

Large is the most capable classic size and the slowest. OpenAI's model card for large-v3 says it "shows improved performance over a wide variety of languages, showing 10% to 20% reduction of errors compared to Whisper large-v2."^[2] The same card notes that large-v3 uses 128 mel frequency bins instead of 80 and was trained on a much larger mixture of weakly labeled and pseudo-labeled audio.^[2]

Turbo sits between medium and large in parameter count and is listed as about eight times faster than large in OpenAI's comparison, with the translation caveat above.^[1]

Memory on Apple Silicon: Why Runtime Matters

Apple Silicon Macs share one pool of unified memory between the CPU and GPU, so there is no separate graphics memory budget to fit a model into. What matters is whether the model, your other apps, and macOS fit comfortably in the memory you have.

The memory figures in OpenAI's table describe the reference PyTorch implementation. Other runtimes use less. The whisper.cpp project, for example, publishes its own figures: about 852 MB of memory for small, about 2.1 GB for medium, and about 3.9 GB for large.^[3] whisper.cpp also supports integer quantization, which it says lets models use "less memory and disk space."^[3] On Apple Silicon it can run the encoder on the Apple Neural Engine through Core ML, which the project says can be "more than x3 faster compared with CPU-only execution."^[3]

The practical takeaway: two apps running "the large model" can feel very different on the same Mac, because one may use a quantized or accelerated runtime and the other may not. Compare tools on your own hardware rather than assuming model size alone determines speed.

When the Larger Model Is Worth the Wait

Larger models generally earn their latency when the audio is harder for a smaller model to get right:

• Specialized vocabulary. Drug names, legal terms, product names, and code identifiers are where smaller models tend to substitute a common word for the one you said.
• Accents and mixed languages. If you speak with a strong regional or non-native accent, or switch languages mid-sentence, a larger multilingual model is more likely to keep up.
• Imperfect recording conditions. Background noise, distance from the microphone, and room echo all make recognition harder.

For short, plain-language emails and notes dictated with a good microphone in a quiet room, a smaller and faster model may produce text you only need to skim. The only way to know where your own threshold sits is to measure it.

Run Your Own Speed and Accuracy Benchmark

Published benchmarks usually use clean, read-aloud datasets. Your dictation has pauses, false starts, jargon, and whatever noise your room produces. A one-day personal benchmark tells you more than any headline number.

1. Write a reference passage. Create a passage of about 200 words that looks like what you actually dictate: a typical email paragraph, a few domain terms, two or three names, and a couple of numbers.

2. Record it once. Use the microphone and room you normally work in. Reusing one recording keeps the comparison fair across models.

3. Transcribe it with each candidate model. Time how long each run takes from start to finished text. Divide the audio length by the processing time to get a real-time multiplier for your Mac.

4. Score accuracy. Calculate word error rate (substitutions plus deletions plus insertions, divided by the number of words in your reference). Our guide to measuring word error rate for dictation walks through the calculation. Mark critical errors, such as a wrong number or a dropped "not," separately.

5. Note memory pressure. Open Activity Monitor's Memory tab while each model runs alongside your usual apps. If memory pressure turns yellow or red, that model is a poor fit for daily use on that machine.

A Decision Framework for Choosing a Model

Use three variables: what you dictate, what hardware you have, and how much latency you will tolerate.

Plain-language writing on a lighter Mac

You write emails, posts, and notes in standard English on a Mac with modest memory. Start with small or small.en and move up to medium only if your correction count stays high. A model that fits comfortably in memory keeps the rest of the machine responsive.

Clinical, legal, or technical dictation

Your content is dense with terminology where a single wrong word matters. Start with the largest model your Mac runs comfortably, add a custom vocabulary for recurring terms, and review every number and name before the text leaves your draft. Our explainers on how on-device AI models run on Apple Silicon and the Mac dictation apps that process speech on device cover the hardware and tool side of this choice.

Developer documentation

Code comments and READMEs are mostly plain English with occasional identifiers. A mid-size model plus a custom vocabulary for function names and product terms often covers the narrative well, and you can spell out the rare identifier that still trips it up.

Multilingual work

If you switch languages or use speech translation, choose a multilingual model rather than an English-only variant, and avoid turbo for translation tasks.^[1]

Auditory lets you choose among local models, so you can use a lighter model for quick notes and a larger one for terminology-heavy work rather than committing to a single choice.

What to Do This Week

Stop reading benchmarks and start collecting your own data.

Day 1: Pick one model and use it for all dictation. At the end of each session, count the corrections you made.

Days 2 to 5: Keep tracking corrections per session, and note the content type (email, notes, technical, casual).

Day 6: Repeat your reference-passage test with the next model size up, and compare correction counts by content type.

Day 7: Set your default. Keep the faster model for content where the larger one did not reduce corrections, and switch up only where it clearly did.

The metric to watch going forward is corrections per 500 words. If it drops after a change, the change is working. If it does not move, the larger model is costing you time without buying accuracy.

References

[1] OpenAI, Whisper repository README (available models and languages). https://github.com/openai/whisper

[2] OpenAI, whisper-large-v3 model card on Hugging Face. https://huggingface.co/openai/whisper-large-v3

[3] ggml-org, whisper.cpp README (memory usage, quantization, and Core ML support). https://github.com/ggml-org/whisper.cpp