<p>Local inference using llama.cpp on Apple Silicon hardware. Tested on
M2 Air (24GB unified memory).</p>
<h2 id="tested-models">Tested Models</h2>
<p>Models evaluated on M2 Air with 24GB RAM:</p>
<table>
<thead>
<tr>
<th>Model</th>
<th>Size</th>
<th>Speed</th>
<th>Quality</th>
</tr>
</thead>
<tbody>
<tr>
<td>Qwen3 8B</td>
<td>8B params</td>
<td>Fast</td>
<td>Good for coding tasks</td>
</tr>
<tr>
<td>DeepSeek-R1-Distill-Qwen-7B</td>
<td>7B params</td>
<td>Fast</td>
<td>Reasoning-focused</td>
</tr>
<tr>
<td>Mistral Nemo 12B</td>
<td>12B params</td>
<td>Medium</td>
<td>Balanced</td>
</tr>
<tr>
<td>Mistral Small 3.1 24B</td>
<td>24B params</td>
<td>Slow</td>
<td>Best quality, memory-limited</td>
</tr>
</tbody>
</table>
<h2 id="hardware-constraints">Hardware Constraints</h2>
<ul>
<li>24GB unified memory limits model size to ~24B parameters (Q4 quantization)</li>
<li>M2 Air GPU cores handle inference but thermal throttling occurs on long runs</li>
<li>SSD swap enables larger models but with significant speed penalty</li>
</ul>
<h2 id="setup">Setup</h2>
<p>llama.cpp with Metal acceleration. Models in GGUF format from
HuggingFace. Server mode exposes OpenAI-compatible API on localhost.</p>
<h2 id="related-blog-posts">Related Blog Posts</h2>
<ul>
<li><a href="/local-llm-m2-air.html">Local LLM Bakeoff on Apple M2 Air</a>:
comparative benchmarks</li>
</ul>


Running language models locally on Apple Silicon using llama.cpp. Covers model selection, performance benchmarks, and hardware constraints.

Kyle Pericak

"It works in my environment"

Local LLMs

Tested Models

Hardware Constraints

Setup

Model	Size	Speed	Quality
Qwen3 8B	8B params	Fast	Good for coding tasks
DeepSeek-R1-Distill-Qwen-7B	7B params	Fast	Reasoning-focused
Mistral Nemo 12B	12B params	Medium	Balanced
Mistral Small 3.1 24B	24B params	Slow	Best quality, memory-limited