<p>No API costs, no data leaving the machine, works offline. The catch is you're
limited to models that fit in RAM. At 16GB, you can run most 7-8B models at
good quality, or push into 12B at a lower quant.</p>
<p>This runs three models locally and compares them:</p>
<table>
<thead>
<tr>
<th>Model</th>
<th>Params</th>
<th>Style</th>
<th>Q4_K_L Size</th>
</tr>
</thead>
<tbody>
<tr>
<td>Qwen3 8B</td>
<td>8B</td>
<td>Instruction</td>
<td>5.49 GB</td>
</tr>
<tr>
<td>DeepSeek-R1-Distill-Qwen-7B</td>
<td>7B</td>
<td>Reasoning</td>
<td>5.09 GB</td>
</tr>
<tr>
<td>Mistral Nemo Instruct 2407</td>
<td>12B</td>
<td>Instruction</td>
<td>7.98 GB</td>
</tr>
</tbody>
</table>
<p>All three fit comfortably in 16GB with room for context. To browse other models,
<a href="https://huggingface.co/bartowski">bartowski's HuggingFace page</a> is a good
starting point for well-tested GGUF quantizations.</p>
<p><strong>Params</strong> (parameters) are the learned weights that make up the model. More
parameters generally means more capable, but also more RAM and slower inference.
7-8B is a practical sweet spot for 16GB machines.</p>
<p><strong>Quantization</strong> is how a full-precision model gets compressed to fit on
consumer hardware. The original weights are 16 or 32-bit floats; quantization
rounds them down to fewer bits, trading a small amount of accuracy for a large
reduction in size and RAM usage. <strong>Q4_K_L</strong> specifically uses 4-bit weights
with the embedding and output layers kept at 8-bit to preserve quality. It's a
good default: much smaller than the lossless formats, better output than plain
Q4.</p>
<p><strong>Instruction</strong> models are trained to follow directions and answer questions
directly. You ask, they answer. <strong>Reasoning</strong> models think out loud first,
working through the problem step by step before giving a final answer. Slower,
but more reliable on anything involving logic, math, or code.</p>
<h1>Hardware</h1>
<ul>
<li>MacBook Air M2, 16GB unified memory</li>
<li>macOS 26 (Tahoe)</li>
</ul>
<p>No active cooling on the Air, so it'll throttle during long runs. Fine for
interactive use.</p>
<h1>Setup</h1>
<p>These steps are shared across all three models.</p>
<h2>1. Install llama.cpp</h2>
<p><a href="https://github.com/ggerganov/llama.cpp">llama.cpp</a> is the runtime that loads
and runs GGUF model files. It handles the actual inference: tokenizing your
input, running it through the model weights, and streaming tokens back to you.
Think of it as the engine; the GGUF file is the model it runs. It also handles
Metal GPU offloading on Apple Silicon, which is what makes local inference fast
enough to be usable.</p>
<pre><code class="language-bash">brew install llama.cpp
</code></pre>
<p>Gives you <code>llama-cli</code> for interactive chat and <code>llama-server</code> for an
OpenAI-compatible HTTP API.</p>
<h2>2. Install the Hugging Face CLI</h2>
<p>Installing Python tools with plain <code>pip</code> can stomp on system Python or conflict
with other packages. <code>pipx</code> puts each CLI tool in its own isolated environment
and adds it to your PATH. No venv to activate.</p>
<pre><code class="language-bash">brew install pipx
pipx ensurepath
pipx install "huggingface_hub[cli]"
</code></pre>
<p><code>pipx ensurepath</code> adds pipx's bin dir to your PATH. Restart your terminal after
running it.</p>
<p>Note: the installed command is <code>hf</code>, not <code>huggingface-cli</code>. The name changed in
v1.x.</p>
<h1>The Models</h1>
<h2>Qwen3 8B</h2>
<p>Alibaba's Qwen3 8B (released April 2025) is a dense instruction-tuned model
that also supports a thinking mode. By default it answers directly. You can
enable step-by-step reasoning with <code>/think</code> in the chat, or disable it with
<code>/no_think</code>.</p>
<pre><code class="language-bash">hf download bartowski/Qwen_Qwen3-8B-GGUF Qwen_Qwen3-8B-Q4_K_L.gguf \
  --local-dir ~/models/qwen3-8b/
</code></pre>
<h2>DeepSeek-R1-Distill-Qwen-7B</h2>
<p>Note: despite the name, there is no 8B Qwen-based DeepSeek distill. The 8B
distill uses Llama architecture. This one is 7B on Qwen architecture, released
January 2025.</p>
<p>DeepSeek distilled the chain-of-thought behaviour from their R1 reasoning model
into this smaller base. It always shows its thinking inside <code>&#x3C;think>...&#x3C;/think></code>
tags before the final answer. Verbose for simple questions, but useful for
anything involving logic or code.</p>
<pre><code class="language-bash">hf download bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF \
  DeepSeek-R1-Distill-Qwen-7B-Q4_K_L.gguf \
  --local-dir ~/models/deepseek-r1-qwen-7b/
</code></pre>
<h2>Mistral Nemo Instruct 12B</h2>
<p>Mistral's 12B model (released July 2024, jointly with NVIDIA). Bigger than the
other two, which shows in the quality of longer-form responses. At Q4_K_L it's
7.98 GB, leaving ~6GB free for macOS and context.</p>
<pre><code class="language-bash">hf download bartowski/Mistral-Nemo-Instruct-2407-GGUF \
  Mistral-Nemo-Instruct-2407-Q4_K_L.gguf \
  --local-dir ~/models/mistral-nemo-12b/
</code></pre>
<h1>Running Them</h1>
<p>The flags are the same for all three. Swap the <code>-m</code> path.</p>
<h2>Interactive chat</h2>
<pre><code class="language-bash"># Qwen3 8B
llama-cli \
  -m ~/models/qwen3-8b/Qwen_Qwen3-8B-Q4_K_L.gguf \
  -ngl 99 -c 8192

# DeepSeek-R1-Distill-Qwen-7B
llama-cli \
  -m ~/models/deepseek-r1-qwen-7b/DeepSeek-R1-Distill-Qwen-7B-Q4_K_L.gguf \
  -ngl 99 -c 8192

# Mistral Nemo 12B
llama-cli \
  -m ~/models/mistral-nemo-12b/Mistral-Nemo-Instruct-2407-Q4_K_L.gguf \
  -ngl 99 -c 8192
</code></pre>
<p>Flags:</p>
<ul>
<li><code>-ngl 99</code>: offload all layers to Metal. Without this you're CPU-only and it's
very slow. <code>--fit on</code> instead might be better, not sure.</li>
<li><code>-c 8192</code>: context window. Comfortable with 16GB for the smaller two. Fine
for Mistral Nemo too given the headroom.</li>
</ul>
<p>Interactive mode is auto-enabled when the model has a chat template, which all
three do. The old <code>--interactive-first</code> flag was removed in newer llama.cpp.</p>
<h2>OpenAI-compatible server</h2>
<pre><code class="language-bash">llama-server \
  -m ~/models/qwen3-8b/Qwen_Qwen3-8B-Q4_K_L.gguf \
  -ngl 99 -c 8192 --port 8080
</code></pre>
<p>Exposes <code>http://localhost:8080/v1/chat/completions</code>. Swap the model path for
whichever you want to serve.</p>
<h1>Conclusion</h1>
<h2>Notes on Each</h2>
<p><strong>Qwen3 8B</strong>: Pretty fast. The thinking thing was a bit slow but not bad. It mostly
stuck to the instructions I gave it, like "max 3 paragraphs", and was reasonably
creative. It was also kind of bland and generic, for better or worse.</p>
<p><strong>DeepSeek-R1-Distill-Qwen-7B</strong>: Also ok speed. It had a hard time following
instructions and sticking to the constraints that I'd tried to put in place, like that
same 3 paragraph limit. Overall I was pretty dissapointed given its reputation.</p>
<p><strong>Mistral Nemo 12B</strong> The direct responses felt fast even though it ran at about half
the tokens per second, due to it not working in Reasoning mode. The writing style of
Nemo was by far the best, and it adhered much better to the prompts.</p>
<h2>Not enough tokens</h2>
<p>The <code>8192</code> tokens they all use got to feeling pretty small, pretty fast. I tried to
push the continuity with a "Choose your own adventure" style conversation and it did
well until it just kind of stopped. I think I've seen claude crushing context down when
it gets near the limit, I wonder if there's a way to do that with this too, or get it
to do some sort of context shifting where it restarts with just the later context when
it runs low. Maybe I can run it in server mode and have a better client manage that,
but that seems like its own post.</p>
<h2>Winner</h2>
<p>Nemo was the pretty clear winner. I'm not sure if that's because of the size of the
model or something else, but it was a significant step up in its ability to stick to
the prompt's instructions, and the writing style was better too.</p>


Kyle Pericak

"It works in my environment"

Categories

Tags