How On-Device AI Models Actually Work on Apple Silicon

Every Apple Silicon Mac contains a processor designed to run neural networks. It sits next to the CPU and GPU on the same chip, and most Mac users never think about it.

That processor is the Apple Neural Engine, and understanding it, along with the unified memory and GPU around it, explains how a laptop can run a speech model like Whisper locally. This is not magic. It is architecture, plus software that knows how to use it. Here is a plain-language breakdown of the hardware, the speech pipeline, and the choices that matter for anyone dictating on a Mac.

Your Mac Has a Dedicated Machine-Learning Engine

Apple introduced the Neural Engine in the A11 Bionic chip for iPhone. When Apple moved the Mac to its own silicon with the M1 in November 2020, the Neural Engine came along. Apple described the M1's Neural Engine as a "16-core architecture capable of 11 trillion operations per second."^[1] By the M4, Apple quoted 38 trillion operations per second, which it called "60x faster than the first Neural Engine in A11 Bionic."^[2]

Unlike a GPU, which handles graphics, display work, and general compute, the Neural Engine is specialized for the multiply-and-add operations at the heart of neural networks. That specialization is what makes it efficient, and also what limits it: it runs the kinds of operations it was designed for, and other work falls back to the GPU or CPU.

There is a catch that matters for dictation. The Neural Engine only helps if the software sends work to it. Apps built on frameworks that target it can use it. Apps built on frameworks that do not will run entirely on the CPU and GPU, however capable the Neural Engine is.

Unified Memory: Why Model Size Is a Memory Question

On a typical PC with a discrete graphics card, the GPU has its own memory, and model weights must be copied from system memory to graphics memory before the GPU can use them. Apple Silicon takes a different approach: the CPU, GPU, and Neural Engine share one pool of unified memory.

Apple's MLX framework describes the benefit directly: arrays "live in shared memory," and operations can run on any supported device "without transferring data."^[3] For a speech model, that means the audio buffer, the spectrogram, and the model's intermediate results can all live in one place that every compute block can read.

The practical consequence is that "will this model fit?" becomes a question about your Mac's total memory and what else you have open. OpenAI lists approximate memory needs for its reference implementation, from about 1 GB for the tiny and base models to about 10 GB for the large model.^[4] Leaner runtimes need less: whisper.cpp publishes about 3.9 GB of memory for its large model.^[5] If you run a big model alongside a browser, an IDE, and a video call, watch memory pressure in Activity Monitor.

Neural Engine vs GPU vs CPU: Which Runs What

Apple Silicon packs three compute blocks onto one chip, and on-device inference can use all of them.

• CPU cores handle general work: capturing audio, preparing it for the model, coordinating the pipeline, and turning the model's output back into text.
• The Neural Engine is efficient at the regular, repetitive matrix operations found in neural-network layers.
• The GPU, programmed through Apple's Metal API, handles large parallel compute, including operations a given runtime chooses not to send to the Neural Engine.

Core ML, Apple's on-device machine-learning framework, decides how to split the work. Apple says Core ML "optimizes on-device performance by leveraging the CPU, GPU, and Neural Engine while minimizing its memory footprint and power consumption," and that running a model strictly on the device "removes any need for a network connection."^[6]

Different Whisper runtimes make different choices. whisper.cpp can run Whisper's encoder on the Neural Engine through Core ML, which its README says can be "more than x3 faster compared with CPU-only execution."^[5] MLX runs on the CPU and GPU.^[3] Neither is wrong. They are different routes to the same goal.

From Sound Wave to Text: How Whisper Processes Your Voice

The journey from spoken words to text involves four stages.

Stage 1: Audio preparation. Your microphone captures audio. The Whisper paper says all audio "is re-sampled to 16,000 Hz, and an 80-channel log-magnitude Mel spectrogram representation is computed on 25-millisecond windows with a stride of 10 milliseconds."^[7] A spectrogram is a picture of sound: time on one axis, frequency on the other, and brightness for energy. The newer large-v3 model uses 128 mel frequency bins instead of 80.^[8] Whisper processes audio in 30-second windows.^[4]

Stage 2: The encoder. Whisper's encoder is a stack of transformer layers that turns the spectrogram into a numerical representation of the sound. The paper's architecture table lists 32 layers with a width of 1,280 for the large model.^[7] This is the heavy, parallel part of the work, and it is the stage whisper.cpp can send to the Neural Engine.^[5]

Stage 3: The decoder. The decoder generates text one token at a time, each step looking at the encoder output and the tokens produced so far. That sequential dependency limits how much the work can be parallelized, which is one reason longer outputs take longer.

Stage 4: Post-processing. Tokens become text with timestamps. Application-level cleanup, such as removing filler words or fixing punctuation, happens after the model has produced its transcript.

Core ML, Metal, and the Runtime Choice

Hardware capabilities mean little without software that uses them. Two Apple frameworks do most of the work for local models.

Core ML runs models that have been converted into its format. Tools such as Apple's coremltools convert models from other frameworks, and the Core ML runtime then decides which compute block runs each part of the model.^[6]

Metal is Apple's low-level GPU programming interface. Frameworks such as MLX use it to run model math on the GPU.

For a dictation app, the runtime decides two things you will notice: how long you wait for text, and how much memory the model takes. When you compare apps, ask which runtime each one uses and test both on the same audio clip, rather than assuming the same model size means the same experience.

What Quantization Does to a Model

Quantization stores model weights at lower numerical precision, for example 8-bit integers instead of 16-bit or 32-bit floating point. Lower precision means a smaller file and lower memory use, which can make a larger model practical on a Mac with less memory.

whisper.cpp supports integer quantization and says quantized models need "less memory and disk space" and, depending on the hardware, "can be processed more efficiently."^[5] Its published memory figures range from about 273 MB for tiny to about 3.9 GB for large.^[5]

The trade-off is accuracy. Lower precision can introduce errors, and how much depends on the model, the quantization method, and your audio. Rather than relying on general claims, test a quantized model and a full-precision one on the same passage of your own dictation, paying attention to specialized terms and numbers.

Measuring Local Performance on Your Own Mac

Published numbers depend on the chip, runtime, model, and audio used. Your own measurement is more useful.

Real-time multiplier. Record a two-minute clip at your normal dictation pace. Transcribe it and divide the clip length by the processing time. A value above 1 means the model keeps ahead of your speech.

Wait after speaking. For live dictation, time the gap between finishing a sentence and seeing text over ten short utterances, and take the median.

Memory headroom. Note memory pressure during transcription with your normal apps open.

Repeat the test when you change models, runtimes, or hardware. Newer chips raise the ceiling. Apple says the M3 family's Neural Engine is "up to 60 percent faster than in the M1 family of chips,"^[9] and the M5 adds "a dedicated Neural Accelerator in each" GPU core.^[10] Whether your app benefits depends on which hardware it uses.

What This Means for Your Workflow

The architecture translates into three benefits you notice when you dictate.

No network dependency for transcription. A local model does not need Wi-Fi to turn speech into text, so it keeps working on a plane or behind a restrictive network. Apple notes that running a model strictly on the device "helps keep a person's data private and your app responsive."^[6]

A clearer processing boundary. When transcription happens on the Mac, the audio does not need to go to a cloud speech service for that step. You still need to review where the finished text goes, including sync, backups, and integrations.

No per-minute meter for local transcription. Many cloud speech APIs bill by audio duration. A local model uses your Mac's compute instead.

The practical limit is memory. If you plan to run large models alongside demanding apps, more unified memory gives you room to work.

Here is your next step: run the real-time multiplier test with your current dictation tool and one model size up. If the larger model fits in memory and cuts your corrections, use it for the content where accuracy matters most. Auditory lets you choose among local models so you can make that trade-off per task.

References

[1] Apple Newsroom, Apple unleashes M1 (November 2020). https://www.apple.com/newsroom/2020/11/apple-unleashes-m1/

[2] Apple Newsroom, Apple introduces M4 chip (May 2024). https://www.apple.com/newsroom/2024/05/apple-introduces-m4-chip/

[3] ml-explore, MLX README. https://github.com/ml-explore/mlx

[4] OpenAI, Whisper repository README. https://github.com/openai/whisper

[5] ggml-org, whisper.cpp README. https://github.com/ggml-org/whisper.cpp

[6] Apple Developer Documentation, Core ML. https://developer.apple.com/documentation/coreml

[7] Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (2022). https://arxiv.org/abs/2212.04356

[8] OpenAI, whisper-large-v3 model card on Hugging Face. https://huggingface.co/openai/whisper-large-v3

[9] Apple Newsroom, Apple unveils M3, M3 Pro, and M3 Max (October 2023). https://www.apple.com/newsroom/2023/10/apple-unveils-m3-m3-pro-and-m3-max-the-most-advanced-chips-for-a-personal-computer/

[10] Apple Newsroom, Apple unleashes M5 (October 2025). https://www.apple.com/newsroom/2025/10/apple-unleashes-m5-the-next-big-leap-in-ai-performance-for-apple-silicon/