Apple's Neural Engine Explained: Why Your M-Series Mac Runs Whisper Locally

Ask most people how their Mac can transcribe speech without touching the internet, and they will shrug. Part of the answer is a piece of silicon most users have never heard of: the Apple Neural Engine. It is a dedicated machine-learning accelerator on the same chip as your CPU and GPU, and with the right software it can run parts of Whisper, OpenAI's speech-recognition model, on your device.

The other part of the answer is the runtime. The same Whisper model can run on the CPU, the GPU, or the Neural Engine depending on how the software was built, and that choice has as much effect on speed as the chip itself. Here is what happens inside your Mac when you dictate, which runtimes use which hardware, and how to measure the result yourself.

What the Neural Engine Actually Does When You Dictate

The Neural Engine is built for the repetitive multiply-and-add operations that neural networks are made of. The CPU is a generalist, the GPU is good at parallel graphics and compute, and the Neural Engine is tuned for running neural-network layers efficiently.

When you dictate, the audio moves through a pipeline. First, the waveform is converted into a mel spectrogram, a representation of frequencies over time. This stage is light. Then Whisper's encoder, a stack of transformer layers, turns that spectrogram into a numerical representation of the sound. The decoder takes that representation and predicts text tokens one at a time. Both the encoder and decoder are heavy matrix math, which is the kind of work accelerators are designed for.

The reason this matters for privacy is architectural. When your selected runtime and model execute locally, the transcription step does not require sending audio to a remote server. A cloud dictation service, by comparison, must transmit your voice for remote processing and return the text. Review cloud fallbacks, integrations, transcript destinations, backups, and device security separately, because local inference only covers the transcription step. Our article on where cloud transcription liability sits covers the rest of the workflow.

M1 to M5: What Apple Says About Each Generation

Apple publishes Neural Engine figures in its chip announcements. The numbers below are Apple's own peak figures, quoted from its newsroom.

Peak operations per second are a ceiling, not a transcription speed. A Whisper runtime that never sends work to the Neural Engine gets none of that capacity, and the M5's change moves more machine-learning capacity into the GPU, which benefits runtimes built on Metal. For short notes and emails, any M-series Mac can run small Whisper models locally. If you routinely transcribe long recordings, newer chips and larger memory configurations give you more headroom. Measure before you upgrade.

Whisper.cpp vs MLX vs Stock Whisper: Which Hardware Each Uses

Here is the part most hardware reviews miss: the same Whisper model runs on different parts of the chip depending on the runtime you use.

whisper.cpp describes Apple Silicon as a "first-class citizen," optimized via ARM NEON, the Accelerate framework, Metal, and Core ML.^[6] Its README says that on Apple Silicon "the Encoder inference can be executed on the Apple Neural Engine (ANE) via Core ML," which it says can be "more than x3 faster compared with CPU-only execution."^[6] It also supports integer quantization, which reduces memory and disk use.^[6]

MLX is Apple's open-source array framework for machine learning on Apple Silicon. Its README says operations "can run on any of the supported devices (currently the CPU and the GPU)," and that arrays live in shared memory so work can move between devices without copying.^[7] MLX Whisper builds therefore accelerate transcription on the GPU rather than the Neural Engine.

Stock openai-whisper is the Python reference implementation. It is a correct baseline, but it is not built around Apple's accelerators.

```python

# MLX Whisper: runs on the Apple Silicon GPU through Metal

import mlx_whisper

result = mlx_whisper.transcribe(

"meeting.wav",

path_or_hf_repo="mlx-community/whisper-large-v3-mlx"

)

print(result["text"])

```

```python

# Reference implementation (openai-whisper): a correct baseline

import whisper

model = whisper.load_model("large-v3")

result = model.transcribe("meeting.wav")

print(result["text"])

```

Both snippets aim to produce a transcript of the same file. How long each takes on your Mac is something to measure, not assume. If transcription feels slow on a fast Mac, check which runtime and model your app uses before blaming the hardware.

Measuring On-Device Speed Against Your Alternatives

You do not need published benchmarks to know how your setup performs. Two simple measurements tell you most of what matters.

Real-time multiplier. Record a two-minute clip at your normal dictation pace. Transcribe it and time the run. Divide two minutes by the processing time. A multiplier above 1 means the model processes audio faster than you speak it. The higher the number, the less you wait.

End-of-speech wait. For live dictation, what you feel is the delay between finishing a sentence and seeing text. Time it with a stopwatch over ten short utterances and take the median.

Cloud services add a network round trip and depend on your connection. Local inference removes the network from the transcription step, so it keeps working on a plane or behind a restrictive corporate network. Which is faster for you depends on your chip, model, runtime, and connection, so test both on the same clip if speed is your deciding factor.

Why Built-In Dictation Can Fall Short for Heavy Use

Apple's built-in Dictation is useful for a text message or a quick search, and on Apple silicon you can keep typing while you speak.^[8] It can still fall short for heavier work in predictable places.

The first is long passages, where punctuation and paragraph structure need more correction as sentences stack up. The second is technical vocabulary. Dictate "we deployed the Kubernetes ingress controller with a mutual TLS sidecar" and see how many words survive. Every wrong word is a manual correction, and the cost is easy to miss until you count it.

A useful way to decide whether you have outgrown built-in tools is to count your post-edit minutes per day for a week. If that number keeps growing, a dedicated tool with model choice and custom vocabulary may pay back the setup time.

A dedicated local Whisper app closes the gap in the places built-in tools struggle. Auditory wraps local Whisper models with cleanup and custom vocabulary so product names and jargon land correctly more often, while the transcription step stays on your Mac.

Choosing the Right Whisper Model for Your Mac

OpenAI lists Whisper sizes with approximate memory requirements for its reference implementation.^[9] Other runtimes, including quantized ones, can use considerably less.^[6]

Two scenarios show the trade-off. A developer dictating code comments is mostly writing plain English with occasional identifiers, so a smaller model may handle it well and return text quickly. A clinician dictating notes full of drug names and dosages benefits from the largest model their Mac runs comfortably, plus a custom vocabulary and careful review.

Frequently Asked Questions: Neural Engine and Local Whisper

Does Whisper actually use the Neural Engine?

It depends on the runtime. whisper.cpp can run the encoder on the Neural Engine through Core ML.^[6] MLX runs on the CPU and GPU.^[7] The stock Python implementation is not built around Apple's accelerators. If you are unsure what your app uses, ask the developer.

How does local transcription work?

With local inference, the pipeline from mel spectrogram to encoder to decoder executes on your Mac. If you enable a cloud fallback or another networked integration, review that path separately.

Which M-series chip do I need for real-time dictation?

Any M-series chip can run smaller Whisper models locally. Larger models and long recordings benefit from newer chips and more memory. Run the real-time multiplier test above on your own Mac to see where your threshold sits.

Why is my transcription slow even on a fast Mac?

Usually a runtime and model mismatch. A large model on a CPU-only implementation will be slow on any chip. Try an accelerated runtime or a smaller model, then measure again.

Next Steps: Test Your Mac's Local AI Today

Here is a concrete 30-minute test. Record a two-minute clip of yourself talking at your normal dictation pace, transcribe it with your current tool using a small model, and calculate your real-time multiplier. Then try a larger model on the same clip and compare accuracy on any technical terms.

Track one metric this week: your post-edit minutes per day. If it keeps climbing, try a larger model or an accelerated runtime and measure again.

For more on how Apple Silicon handles this workload, see our Apple Silicon on-device AI explainer, and if dictation is part of a plan to type less, our guide on voice input and RSI is worth a read.

References

[1] Apple Newsroom, Apple unleashes M1 (November 2020). https://www.apple.com/newsroom/2020/11/apple-unleashes-m1/

[2] Apple Newsroom, Apple unveils M2 (June 2022). https://www.apple.com/newsroom/2022/06/apple-unveils-m2-with-breakthrough-performance-and-capabilities/

[3] Apple Newsroom, Apple unveils M3, M3 Pro, and M3 Max (October 2023). https://www.apple.com/newsroom/2023/10/apple-unveils-m3-m3-pro-and-m3-max-the-most-advanced-chips-for-a-personal-computer/

[4] Apple Newsroom, Apple introduces M4 chip (May 2024). https://www.apple.com/newsroom/2024/05/apple-introduces-m4-chip/

[5] Apple Newsroom, Apple unleashes M5 (October 2025). https://www.apple.com/newsroom/2025/10/apple-unleashes-m5-the-next-big-leap-in-ai-performance-for-apple-silicon/

[6] ggml-org, whisper.cpp README. https://github.com/ggml-org/whisper.cpp

[7] ml-explore, MLX README. https://github.com/ml-explore/mlx

[8] Apple Support, Dictate messages and documents on Mac. https://support.apple.com/guide/mac-help/mh40584/mac

[9] OpenAI, Whisper repository README. https://github.com/openai/whisper