Apple’s Foundation Model is not the most capable AI model you can run on an iPhone, according to a new DeviceMark evaluation. But it has one major advantage over the other models tested: speed. Apple’s on-device model can generate responses considerably faster than the competing LLMs tested on an iPhone 17 Pro.
DeviceMark tested Apple’s Foundation Model alongside several other local models, including LFM2.5-1.2B, Qwen3.5-2B, Gemma 4 E2B, Granite-4.0-H-1B and NVIDIA’s Nemotron-3-Nano-4B. The evaluation uses 596 questions covering instruction following, general knowledge and mathematics, giving a broader picture of how these models perform on an iPhone.
Apple’s Foundation Model scored 55.8% on DeviceMark’s overall intelligence score. That puts it behind LFM2.5-1.2B at 68.2%, Qwen3.5-2B at 62.1% and Nemotron-3-Nano-4B at 61.4%.
Where Apple’s model stands out is generation speed. DeviceMark estimates that the Foundation Model generates between 70 and 86 tokens per second on the iPhone 17 Pro. LFM2.5-1.2B reaches 45.5 tokens per second, while Gemma 4 E2B reaches 57 tokens per second. Qwen3.5-2B manages 29 tokens per second, Granite-4.0-H-1B reaches 31 tokens per second and Nemotron-3-Nano-4B reaches just 14.7 tokens per second. That gives Apple’s model a substantial advantage when generating text locally.
There is an important qualification to the Apple result. The Foundation Models framework does not expose token counts, so DeviceMark cannot directly measure tokens per second in the same way as the other models. Instead, the benchmark estimates Apple’s token rate from the amount of text generated and the character-to-token ratios observed from the other models. DeviceMark therefore reports a range of 70 to 86 tokens per second rather than a single number. Even allowing for that difference in measurement, Apple’s on-device model is clearly optimized for fast responses.
The intelligence results show a different picture. Apple’s Foundation Model scores 82.1% on IFEval, which measures how well an AI model follows instructions. LFM2.5-1.2B leads with 88.4%, while Gemma 4 E2B scores 76.3%, Qwen3.5-2B scores 69.2% and Nemotron-3-Nano-4B scores 43.4%.
Apple’s model performs less well on tasks requiring broader knowledge and mathematical reasoning. It scores 41.3% on MMLU-Pro and 44% on MATH. Nemotron-3-Nano-4B, for example, scores 57.6% on MMLU-Pro and 83% on MATH.
So Apple is not trying to make the most powerful general-purpose LLM available for the iPhone. That makes sense when looking at how Apple intends its Foundation Model to be used.
The model is built into Apple’s operating systems and accessed by developers through the Foundation Models framework. Developers do not have to package a multi-gigabyte LLM with their applications. The DeviceMark testing lists Apple’s model as OS-resident, while the other models tested require between 1.7GB and 4.6GB of model data. This is an important advantage for on-device AI.
An app that needs to rewrite text, summarize information, extract details or perform another relatively small AI task does not necessarily need a huge model with advanced mathematical reasoning. It needs a model that can understand the request and return an answer quickly. Apple’s Foundation Model appears to be designed around exactly those requirements.
The DeviceMark results also show why model size alone is not a good way to judge an on-device LLM. NVIDIA’s Nemotron-3-Nano-4B has four billion parameters and is much stronger at mathematics than Apple’s model, but it generates text at only 14.7 tokens per second in the test.
LFM2.5-1.2B is another interesting comparison. It has considerably fewer parameters than some of the other models but achieves the highest overall intelligence score in DeviceMark’s comparison while generating 45.5 tokens per second.
The hardware and software used to run a model matter just as much as the model itself. Apple has control over the entire stack since it designs the processors in the iPhone, controls iOS, controls the Foundation Models framework and develops the model specifically for its own hardware. That gives Apple considerably more room to optimize how the model runs than a developer simply taking a general-purpose open model and deploying it on a phone.
There is also a difference in how much of the benchmark each model completed. DeviceMark reports that Apple’s Foundation Model produced answers for 60% of the benchmark questions. LFM2.5-1.2B answered 100%, Nemotron-3-Nano-4B answered 91%, Qwen3.5-2B answered 86%, Gemma 4 E2B answered 80% and Granite-4.0-H-1B answered 97%.
Questions that do not produce an answer within the benchmark’s allowed generation budget are counted as incorrect, which affects the overall intelligence score. This helps explain why Apple’s Foundation Model does not rank particularly highly when looking only at the final intelligence percentage.
The comparison with cloud-based models is even more significant. Large cloud LLMs still perform considerably better on complex reasoning, knowledge and mathematics. A model running entirely on an iPhone has far less memory, processing power and energy available than a model running across a large data center.
Apple is making a different trade-off. The Foundation Model needs to be available locally, respond quickly and operate within the hardware and memory limits of an iPhone. It also needs to be small enough that Apple can integrate it into the operating system rather than requiring every developer to distribute their own AI model.
DeviceMark’s results show that Apple has prioritized those requirements. Its Foundation Model does not lead the benchmark for overall intelligence, but its estimated generation speed is significantly higher than every other model tested on the iPhone 17 Pro. That could be more important for the everyday AI features Apple is building into iOS than simply having the highest benchmark score.
For Apple, the goal is not necessarily to put the biggest LLM on an iPhone. It is to make AI features feel like another part of the operating system. And based on DeviceMark’s testing, Apple has optimized its Foundation Model to do one part of that particularly well: respond quickly without leaving the iPhone.
