Forty tokens per second is an attractive number, but the more interesting news is the code path that makes local generation practical on a familiar computer.
The initial Metal inference pull request for llama.cpp, merged on June 4, brings LLaMA generation to Apple silicon GPUs. Its early implementation has specific limits, including Q4_0 support and a different path for prompt ingestion. It is worth reading the actual change before treating a shared performance demonstration as a description of every stage or model.
Two workloads inside one conversation
Processing the user's prompt and generating the answer are related but different operations. The prompt contains many known tokens that can be processed with substantial parallel work. During ordinary autoregressive generation, each new token depends on the tokens already produced. That creates a repeated decoding loop.
A single tokens-per-second figure can hide this distinction. A model may respond quickly to a short prompt but take noticeably longer to begin answering a large document. Time to the first token and speed after generation begins tell different parts of the story.
Memory also has more than one component. Model weights must fit, intermediate work needs space, and attention state accumulates with context. Quantizing weights reduces one major cost, but it does not make every other allocation disappear. The application needs room to operate alongside the rest of the machine.
Apple silicon's shared memory architecture is relevant because CPU and GPU work draw from the same physical memory pool. That can avoid some transfers associated with separate graphics memory, while also making the total available capacity a shared constraint. A large model competes with everything else using that memory.
Quantization introduces its own choices. Fewer bits can make a model smaller and reduce bandwidth demands, but the representation has to be supported by the implementation. Quality should be evaluated on the intended task, rather than inferred from file size or a single fluent answer.
What I would want from a local assistant
The attraction is not only avoiding a server call. A local tool can remain close to personal documents, work without a network after setup, and make its availability less dependent on a hosted service. Those are useful properties for a small, bounded task even when the model is less capable than a larger remote one.
I would begin with something that can be checked: rewriting a short passage while preserving specified facts, extracting a few fields, or answering from supplied material. A model that writes smoothly can still lose a negation or invent a detail. Keeping the task concrete makes the limitations visible.
The application should also be honest about resource use. Starting a model can take time, and keeping it resident occupies memory. A good interface gives the user control over that lifecycle rather than silently treating the machine as a dedicated inference server.
The Metal work is exciting because it turns local language models into a more ordinary software possibility. The next useful questions are practical: which model fits, which workload feels responsive, and which task benefits from staying on the device? Those answers will determine whether a fast demonstration becomes a tool that stays open.