<p>I'm building a tabletop RPG game that generates images on the fly: character portraits,
enemy illustrations, scene art. Cloud APIs work but get expensive when every combat
encounter wants several images. More importantly, I need character consistency. If I
generate a portrait of "Aria the rogue" in session one, future images of her should
look like the same character, not a random new person.</p>
<p>ComfyUI solves both. It runs locally on Apple Silicon, exposes a REST API, and
supports IP-Adapter for using an existing portrait as a visual reference for new
generations.</p>
<h1 id="hardware-reality">Hardware Reality</h1>
<p>MacBook Air M2, 16GB unified memory. The Air throttles under sustained load since
it has no active cooling. Fine for a game that generates at natural breaks (between
scenes, at combat start) rather than continuously.</p>
<p>16GB comfortably fits:</p>
<ul>
<li><strong>SDXL</strong> (fp16, ~6.5GB): mature IP-Adapter ecosystem, good portrait quality</li>
<li><strong>FLUX.2-klein-4B</strong> (fp8, ~8-9GB): Apache 2.0, released November 2025, 4-step
distilled generation, supersedes FLUX.1-schnell. IP-Adapter support less
mature than SDXL.</li>
<li><strong>FLUX.1-dev</strong> (fp8, ~8GB): previous generation but still the best quality
option that fits 16GB. FLUX.2-dev exists but won't fit.</li>
</ul>
<p>For this post I'm using SDXL. Its IP-Adapter tools are battle-tested for exactly
this use case, and 6.5GB leaves plenty of headroom for additional models.</p>
<h1 id="install-comfyui">Install ComfyUI</h1>
<p>ComfyUI has a <a href="https://github.com/Comfy-Org/desktop">Desktop app</a> that handles
everything automatically. Use that if you just want the UI. For API access and
full control, manual install is cleaner.</p>
<pre><code class="language-bash">git clone https://github.com/comfyanonymous/ComfyUI
cd ComfyUI
python3.13 -m venv venv
source venv/bin/activate
python --version  # must say 3.13 — if it says 3.9 you have an old venv, rm -rf venv and redo
pip install -r requirements.txt
</code></pre>
<p>On Apple Silicon, the requirements file pulls a PyTorch build with MPS support
automatically. No CUDA, no special flags.</p>
<p>Start it:</p>
<pre><code class="language-bash">source venv/bin/activate
python main.py --listen 0.0.0.0 --port 8188
</code></pre>
<p><code>--listen 0.0.0.0</code> binds to all interfaces instead of localhost only. Needed for
the API to be reachable from other processes or devices on your network.</p>
<p>I keep a <code>start-comfyui.sh</code> in <code>~/models/</code> next to the model README so there's
one place for everything model-related:</p>
<pre><code class="language-bash">#!/bin/bash
source /path/to/ComfyUI/venv/bin/activate
cd /path/to/ComfyUI
python main.py --listen 0.0.0.0 --port 8188
</code></pre>
<p>Open <code>http://localhost:8188</code>. You should see the node graph UI with a default
workflow loaded. No models needed for the server to start — the UI comes up
regardless.</p>
<h1 id="model-storage">Model Storage</h1>
<p>Models don't need to live inside the repo. ComfyUI will look in additional
directories if you tell it to via <code>extra_model_paths.yaml</code>:</p>
<pre><code class="language-bash">cp extra_model_paths.yaml.example extra_model_paths.yaml
</code></pre>
<p>Then add an entry pointing at wherever you keep models. For example, if you
store them under <code>~/models/</code>:</p>
<pre><code class="language-yaml">my_models:
  base_path: ~/models/
  checkpoints: sdxl/
  ipadapter: ipadapter/
  clip_vision: clip_vision/
</code></pre>
<p>Adjust the subdirectory names to match your layout. ComfyUI merges these with
its own paths and shows everything in the UI dropdowns.</p>
<h1 id="models">Models</h1>
<p>Download SDXL (~6.5GB):</p>
<pre><code class="language-bash">hf download stabilityai/stable-diffusion-xl-base-1.0 \
  sd_xl_base_1.0.safetensors \
  --local-dir ~/models/sdxl/
</code></pre>
<p>For IP-Adapter, clone the custom node into ComfyUI's <code>custom_nodes/</code> directory:</p>
<pre><code class="language-bash">cd /path/to/ComfyUI/custom_nodes
git clone https://github.com/cubiq/ComfyUI_IPAdapter_plus
</code></pre>
<p>No pip install needed — it has no extra dependencies beyond what ComfyUI
already installs. ComfyUI picks it up automatically on next startup.</p>
<p>Then download the IP-Adapter weights:</p>
<pre><code class="language-bash"># IP-Adapter for SDXL
hf download h94/IP-Adapter \
  sdxl_models/ip-adapter_sdxl.bin \
  --local-dir ~/models/ipadapter/

# CLIP vision encoder (required by IP-Adapter — must be ViT-bigG, not ViT-L)
hf download laion/CLIP-ViT-bigG-14-laion2B-39B-b160k \
  --local-dir ~/models/clip_vision/CLIP-ViT-bigG-14-laion2B-39B-b160k/
</code></pre>
<p>Restart ComfyUI after installing custom nodes. It rescans the nodes directory on
startup.</p>
<h1 id="quick-test">Quick Test</h1>
<p>Before touching the API, confirm generation works at all.</p>
<p>The built-in Templates target newer models and won't work with SDXL. ComfyUI
ships a test workflow in the repo — open it with <strong>File > Open</strong>:</p>
<pre><code>/path/to/ComfyUI/tests/inference/graphs/default_graph_sdxl1_0.json
</code></pre>
<p><img src="/images/comfyui-web-interface.png" alt="ComfyUI web interface"></p>
<p>It's a base+refiner workflow with two Load Checkpoint nodes. The top one
should already show <code>sd_xl_base_1.0.safetensors</code>. The bottom one defaults to
<code>sd_xl_refiner_1.0.safetensors</code> — change it to <code>sd_xl_base_1.0.safetensors</code>
too. We're not doing base+refiner, just running base twice.</p>
<p>Drop a prompt into one of the CLIP Text Encode nodes and hit <strong>Run</strong>:</p>
<pre><code>a knight in plate armor holding a mace and shield, fantasy RPG portrait,
detailed face, dramatic lighting, painterly style, 4k, highly detailed
</code></pre>
<p>First run is slow while the model loads. On M2 Air, SDXL at 20 steps takes
90-120 seconds. Subsequent runs with the model cached are faster but still
in that range — the Air throttles under sustained load with no active cooling.</p>
<p><img src="/images/oss-img-gen-knight.png" alt="Knight portrait generated with SDXL"></p>
<h1 id="img2img-post-battle">Img2Img: Post-Battle</h1>
<p>Txt2img generates from noise. Img2img starts from an existing image and
nudges it toward a new prompt. Good way to stay close to a character while
changing context or condition.</p>
<p>Add two nodes to the current workflow. Right-click empty canvas space (not
on a node) to get the picker:</p>
<ul>
<li><strong>image → Load Image</strong> — then select your knight from <code>ComfyUI/output/</code>
using the dropdown in the node, or upload it from there</li>
<li><strong>latent → VAE Encode</strong> — converts a pixel image into the latent space
the sampler works in, replacing the EmptyLatentImage</li>
</ul>
<p>Then make three connections. To draw a wire: click and drag from an output
dot (right side of a node) to an input dot (left side of another node).
The dots are color-coded by type and will only connect to compatible ports.</p>
<ol>
<li>Drag from the <strong>IMAGE</strong> dot (blue, right side of Load Image) to the
<strong>pixels</strong> dot on VAE Encode</li>
<li>Drag from either <strong>Load Checkpoint VAE</strong> dot → <strong>VAE Encode vae</strong> dot</li>
<li>Drag from <strong>VAE Encode LATENT</strong> dot → <strong>KSampler latent_image</strong> dot
(disconnects EmptyLatentImage automatically)</li>
</ol>
<p>The test workflow uses <strong>KSampler Advanced</strong>, which has no <code>denoise</code> slider.
Instead set <strong>start_at_step</strong> to <code>6</code>. With 20 total steps that runs the last
14, equivalent to about 70% denoise. Lower start_at_step = more faithful to
original, higher = more change.</p>
<p>Update the positive prompt — that's the CLIP Text Encode node wired to the
KSampler's <strong>positive</strong> input, as opposed to the negative one (which holds
your "ugly, blurry..." text):</p>
<pre><code>a knight in plate armor, damaged armor, battle-worn, dented helmet,
scratches and dents, mud-stained, exhausted, fantasy RPG portrait,
detailed face, dramatic lighting, painterly style, 4k, highly detailed
</code></pre>
<p>Hit <strong>Run</strong>. If it's changing too much, drop start_at_step to 4. If the
damage isn't reading, push to 8.</p>
<p><img src="/images/img2image-knight.png" alt="Knight after img2img battle damage pass"></p>
<p>Img2img works, but it's coarse. Fine details like face structure get lost
as you increase denoise, and at low denoise the prompt has limited
influence. Good for "same scene, different state." Not what you want for
generating the same character across completely different scenes.</p>
<p>That's what IP-Adapter is for. Instead of perturbing the reference image,
it encodes it through CLIP vision into an embedding that gets injected
alongside the text conditioning during generation. The sampler still starts
from pure noise, but the attention layers are nudged toward the character's
appearance at every step. Face and body survive across very different
scenes because the conditioning runs throughout, not just at the start.</p>
<h1 id="the-api">The API</h1>
<p>ComfyUI's API accepts a workflow as JSON and queues the generation. The workflow
you build in the UI can be exported in API format: go to Settings, enable Dev
Mode Options, then use the "Save (API Format)" button on the workflow.</p>
<p>The API format represents each node as a dict entry keyed by node ID. Here's a
minimal SDXL workflow:</p>
<pre><code class="language-json">{
  "4": {
    "class_type": "CheckpointLoaderSimple",
    "inputs": { "ckpt_name": "sd_xl_base_1.0.safetensors" }
  },
  "5": {
    "class_type": "EmptyLatentImage",
    "inputs": { "batch_size": 1, "height": 1024, "width": 768 }
  },
  "6": {
    "class_type": "CLIPTextEncode",
    "inputs": {
      "clip": ["4", 1],
      "text": "a fantasy rogue in a dark alley, RPG portrait, detailed"
    }
  },
  "7": {
    "class_type": "CLIPTextEncode",
    "inputs": {
      "clip": ["4", 1],
      "text": "ugly, blurry, watermark, text, bad anatomy"
    }
  },
  "3": {
    "class_type": "KSampler",
    "inputs": {
      "model": ["4", 0],
      "positive": ["6", 0],
      "negative": ["7", 0],
      "latent_image": ["5", 0],
      "seed": 42,
      "steps": 20,
      "cfg": 7,
      "sampler_name": "dpmpp_2m",
      "scheduler": "karras",
      "denoise": 1
    }
  },
  "8": {
    "class_type": "VAEDecode",
    "inputs": { "samples": ["3", 0], "vae": ["4", 2] }
  },
  "9": {
    "class_type": "SaveImage",
    "inputs": { "filename_prefix": "rpg", "images": ["8", 0] }
  }
}
</code></pre>
<p>Submit it:</p>
<pre><code class="language-bash">curl -X POST http://localhost:8188/prompt \
  -H "Content-Type: application/json" \
  -d '{"prompt": &#x3C;workflow_json>}'
# Returns: {"prompt_id": "abc123...", "number": 1, "node_errors": {}}
</code></pre>
<p>Poll for completion:</p>
<pre><code class="language-bash">curl http://localhost:8188/history/abc123
# When done: history[prompt_id]["status"]["completed"] == true
</code></pre>
<p>Fetch the image:</p>
<pre><code class="language-bash">curl "http://localhost:8188/view?filename=rpg_00001_.png" \
  --output character.png
</code></pre>
<p>Images also land in <code>ComfyUI/output/</code> on disk if you'd rather read them
directly.</p>
<h1 id="image-consistency-with-ip-adapter">Image Consistency with IP-Adapter</h1>
<p>IP-Adapter encodes a reference image using CLIP vision and injects that embedding
alongside the text prompt during generation. The result is a new image that
respects the prompt but is visually pulled toward the reference: same hair, same
armor, similar face structure.</p>
<p>It's not a perfect clone. Think of it as a strong nudge. Consistent text prompts
help too. If every prompt for Aria includes "tall elf rogue, purple hair, leather
armor," you're conditioning from both directions.</p>
<p>Add these nodes to the workflow above:</p>
<pre><code class="language-json">"10": {
  "class_type": "IPAdapterModelLoader",
  "inputs": { "ipadapter_file": "ip-adapter_sdxl.bin" }
},
"11": {
  "class_type": "CLIPVisionLoader",
  "inputs": { "clip_name": "CLIP-ViT-bigG-14-laion2B-39B-b160k/open_clip_model.safetensors" }
},
"12": {
  "class_type": "LoadImage",
  "inputs": { "image": "aria_portrait.png" }
},
"13": {
  "class_type": "IPAdapterAdvanced",
  "inputs": {
    "model": ["4", 0],
    "ipadapter": ["10", 0],
    "image": ["12", 0],
    "clip_vision": ["11", 0],
    "weight": 0.6,
    "weight_type": "linear",
    "combine_embeds": "concat",
    "start_at": 0.0,
    "end_at": 1.0,
    "embeds_scaling": "V only"
  }
}
</code></pre>
<p>Then change node <code>"3"</code>'s <code>model</code> input from <code>["4", 0]</code> to <code>["13", 0]</code>.</p>
<p>The prompt here asks for the same knight in a tavern — completely different
scene, same character used as reference.</p>
<p><img src="/images/oss-img-gen-ipadapter.png" alt="Knight in a tavern, generated with IP-Adapter using the portrait as reference"></p>
<p><code>weight</code> is the main knob. At 0.5 you get style and general appearance influence.
At 0.8 it starts fighting the text prompt. For character identity, 0.6-0.7 is
the range to start with.</p>
<p>The <code>image</code> input in node <code>"12"</code> is a filename relative to <code>ComfyUI/input/</code>.
Copy your reference images there, or upload them via the API:</p>
<pre><code class="language-bash">curl -X POST http://localhost:8188/upload/image \
  -F "image=@aria_portrait.png"
# Returns: {"name": "aria_portrait.png", "subfolder": "", "type": "input"}
</code></pre>
<h2 id="face-consistency">Face Consistency</h2>
<p>Standard IP-Adapter works well for overall style and body type. Faces drift more
than you'd like, especially across very different scenes. IP-Adapter FaceID handles
this by extracting face embeddings specifically rather than encoding the whole image.</p>
<p>Download:</p>
<pre><code class="language-bash">hf download h94/IP-Adapter-FaceID \
  ip-adapter-faceid-plusv2_sdxl.bin \
  --local-dir ~/models/ipadapter/
</code></pre>
<p>FaceID also requires InsightFace, which isn't on HuggingFace. Download the
<code>antelopev2</code> model package from the
<a href="https://github.com/deepinsight/insightface/tree/master/model_zoo">InsightFace model zoo</a>
and place it under <code>~/models/insightface/models/antelopev2/</code> (or wherever your
<code>extra_model_paths.yaml</code> maps <code>insightface</code>).</p>
<p>Install the Python package into the ComfyUI venv:</p>
<pre><code class="language-bash">source /path/to/ComfyUI/venv/bin/activate
pip install insightface onnxruntime
</code></pre>
<p>With FaceID installed, use <code>IPAdapterFaceIDAdvanced</code> instead of <code>IPAdapterAdvanced</code>.
It takes the same inputs but runs the face embedding extraction automatically.
Faces stay recognizable across very different scenes much more reliably than with
the standard adapter.</p>
<p>You can combine both: use <code>IPAdapterAdvanced</code> for overall character appearance
and <code>IPAdapterFaceIDAdvanced</code> for face locking, chaining them in sequence.</p>
<h1 id="game-integration">Game Integration</h1>
<p>The pattern for a TTRPG character system:</p>
<pre class="mermaid">
graph TD
    A[Player describes character] --> B[Generate initial portrait\nno reference image]
    B --> C[Store portrait in character record]
    C --> D[Game event triggers image]
    D --> E[Build scene + character prompt]
    E --> F[Upload portrait, generate\nwith IP-Adapter]
    F --> G[Display in game]
    G --> D
</pre>


<p>No API costs, no data leaving the machine, works offline. The catch is you're
limited to models that fit in RAM. At 16GB, you can run most 7-8B models at
good quality, or push into 12B at a lower quant.</p>
<p>This runs three models locally and compares them:</p>
<table>
<thead>
<tr>
<th>Model</th>
<th>Params</th>
<th>Style</th>
<th>Quant</th>
<th>Size</th>
</tr>
</thead>
<tbody>
<tr>
<td>Qwen3 8B</td>
<td>8B</td>
<td>Instruction</td>
<td>Q4_K_L</td>
<td>5.49 GB</td>
</tr>
<tr>
<td>DeepSeek-R1-Distill-Qwen-7B</td>
<td>7B</td>
<td>Reasoning</td>
<td>Q4_K_L</td>
<td>5.09 GB</td>
</tr>
<tr>
<td>Mistral Nemo Instruct 2407</td>
<td>12B</td>
<td>Instruction</td>
<td>Q4_K_L</td>
<td>7.98 GB</td>
</tr>
<tr>
<td>Mistral Small 3.1 24B</td>
<td>24B</td>
<td>Instruction</td>
<td>Q4_K_S</td>
<td>13.5 GB</td>
</tr>
</tbody>
</table>
<p>The first three fit comfortably in 16GB with room for context. Mistral Small
3.1 at 13.5 GB leaves very little headroom on a 16GB machine. To browse other models,
<a href="https://huggingface.co/bartowski">bartowski's HuggingFace page</a> is a good
starting point for well-tested GGUF quantizations.</p>
<p><strong>Params</strong> (parameters) are the learned weights that make up the model. More
parameters generally means more capable, but also more RAM and slower inference.
7-8B is a practical sweet spot for 16GB machines.</p>
<p><strong>Quantization</strong> is how a full-precision model gets compressed to fit on
consumer hardware. The original weights are 16 or 32-bit floats; quantization
rounds them down to fewer bits, trading a small amount of accuracy for a large
reduction in size and RAM usage. <strong>Q4_K_L</strong> uses 4-bit weights with the
embedding and output layers kept at higher precision (8-bit) to preserve
quality. <strong>Q4_K_S</strong> is the smaller sibling: same 4-bit compression throughout,
including the embedding layers, shaving off a bit of quality for a smaller
file. For Mistral Small 3.1 that's the difference between 13.5 GB (K_S) and
14.8 GB (K_L). On a 16GB machine, that 1.3 GB matters, so K_S is the right
pick there.</p>
<p><strong>Instruction</strong> models are trained to follow directions and answer questions
directly. You ask, they answer. <strong>Reasoning</strong> models think out loud first,
working through the problem step by step before giving a final answer. Slower,
but more reliable on anything involving logic, math, or code.</p>
<h1 id="hardware">Hardware</h1>
<ul>
<li>MacBook Air M2, 16GB unified memory</li>
<li>macOS 26 (Tahoe)</li>
</ul>
<p>No active cooling on the Air, so it'll throttle during long runs. Fine for
interactive use.</p>
<h1 id="setup">Setup</h1>
<p>These steps are shared across all three models.</p>
<h2 id="1-install-llamacpp">1. Install llama.cpp</h2>
<p><a href="https://github.com/ggerganov/llama.cpp">llama.cpp</a> is the runtime that loads
and runs GGUF model files. It handles the actual inference: tokenizing your
input, running it through the model weights, and streaming tokens back to you.
Think of it as the engine; the GGUF file is the model it runs. It also handles
Metal GPU offloading on Apple Silicon, which is what makes local inference fast
enough to be usable.</p>
<pre><code class="language-bash">brew install llama.cpp
</code></pre>
<p>Gives you <code>llama-cli</code> for interactive chat and <code>llama-server</code> for an
OpenAI-compatible HTTP API.</p>
<h2 id="2-install-the-hugging-face-cli">2. Install the Hugging Face CLI</h2>
<p>Installing Python tools with plain <code>pip</code> can stomp on system Python or conflict
with other packages. <code>pipx</code> puts each CLI tool in its own isolated environment
and adds it to your PATH. No venv to activate.</p>
<pre><code class="language-bash">brew install pipx
pipx ensurepath
pipx install "huggingface_hub[cli]"
</code></pre>
<p><code>pipx ensurepath</code> adds pipx's bin dir to your PATH. Restart your terminal after
running it.</p>
<p>Note: the installed command is <code>hf</code>, not <code>huggingface-cli</code>. The name changed in
v1.x.</p>
<h1 id="the-models">The Models</h1>
<h2 id="qwen3-8b">Qwen3 8B</h2>
<p>Alibaba's Qwen3 8B (released April 2025) is a dense instruction-tuned model
that also supports a thinking mode. By default it answers directly. You can
enable step-by-step reasoning with <code>/think</code> in the chat, or disable it with
<code>/no_think</code>.</p>
<pre><code class="language-bash">hf download bartowski/Qwen_Qwen3-8B-GGUF Qwen_Qwen3-8B-Q4_K_L.gguf \
  --local-dir ~/models/qwen3-8b/
</code></pre>
<h2 id="deepseek-r1-distill-qwen-7b">DeepSeek-R1-Distill-Qwen-7B</h2>
<p>Note: despite the name, there is no 8B Qwen-based DeepSeek distill. The 8B
distill uses Llama architecture. This one is 7B on Qwen architecture, released
January 2025.</p>
<p>DeepSeek distilled the chain-of-thought behaviour from their R1 reasoning model
into this smaller base. It always shows its thinking inside <code>&#x3C;think>...&#x3C;/think></code>
tags before the final answer. Verbose for simple questions, but useful for
anything involving logic or code.</p>
<pre><code class="language-bash">hf download bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF \
  DeepSeek-R1-Distill-Qwen-7B-Q4_K_L.gguf \
  --local-dir ~/models/deepseek-r1-qwen-7b/
</code></pre>
<h2 id="mistral-small-31-24b">Mistral Small 3.1 24B</h2>
<p>Released March 17, 2025. Mistral's most capable small model at the time.
24B parameters, multimodal (supports image input), 128k context window,
Apache 2.0. Mistral officially recommends 32GB RAM for this one, so running
it on a 16GB M2 Air is pushing it. It loads and runs, but context has to stay
small.</p>
<p>Note: bartowski's repo for this model has a <code>mistralai_</code> prefix in the name,
unlike the others.</p>
<pre><code class="language-bash">hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF \
  Mistral-Small-3.1-24B-Instruct-2503-Q4_K_S.gguf \
  --local-dir ~/models/mistral-small-3.1-24b/
</code></pre>
<p>K_S over K_L here because at 13.5 GB vs 14.8 GB, the difference determines
whether you have any RAM left for context and the OS.</p>
<h2 id="mistral-nemo-instruct-12b">Mistral Nemo Instruct 12B</h2>
<p>Mistral's 12B model (released July 2024, jointly with NVIDIA). Bigger than the
other two, which shows in the quality of longer-form responses. At Q4_K_L it's
7.98 GB, leaving ~6GB free for macOS and context.</p>
<pre><code class="language-bash">hf download bartowski/Mistral-Nemo-Instruct-2407-GGUF \
  Mistral-Nemo-Instruct-2407-Q4_K_L.gguf \
  --local-dir ~/models/mistral-nemo-12b/
</code></pre>
<h1 id="running-them">Running Them</h1>
<p>The flags are the same for all three. Swap the <code>-m</code> path.</p>
<h2 id="interactive-chat">Interactive chat</h2>
<pre><code class="language-bash"># Qwen3 8B
llama-cli \
  -m ~/models/qwen3-8b/Qwen_Qwen3-8B-Q4_K_L.gguf \
  -ngl 99 -c 8192

# DeepSeek-R1-Distill-Qwen-7B
llama-cli \
  -m ~/models/deepseek-r1-qwen-7b/DeepSeek-R1-Distill-Qwen-7B-Q4_K_L.gguf \
  -ngl 99 -c 8192

# Mistral Nemo 12B
llama-cli \
  -m ~/models/mistral-nemo-12b/Mistral-Nemo-Instruct-2407-Q4_K_L.gguf \
  -ngl 99 -c 8192

# Mistral Small 3.1 24B — tight on RAM, keep context small
llama-cli \
  -m ~/models/mistral-small-3.1-24b/Mistral-Small-3.1-24B-Instruct-2503-Q4_K_S.gguf \
  -ngl 99 -c 2048
</code></pre>
<p>Flags:</p>
<ul>
<li><code>-ngl 99</code>: offload all layers to Metal. Without this you're CPU-only and it's
very slow. <code>--fit on</code> instead might be better, not sure.</li>
<li><code>-c 8192</code>: context window. Comfortable with 16GB for the smaller two. Fine
for Mistral Nemo too given the headroom.</li>
</ul>
<p>Interactive mode is auto-enabled when the model has a chat template, which all
three do. The old <code>--interactive-first</code> flag was removed in newer llama.cpp.</p>
<h2 id="openai-compatible-server">OpenAI-compatible server</h2>
<pre><code class="language-bash">llama-server \
  -m ~/models/qwen3-8b/Qwen_Qwen3-8B-Q4_K_L.gguf \
  -ngl 99 -c 8192 --port 8080
</code></pre>
<p>Exposes <code>http://localhost:8080/v1/chat/completions</code>. Swap the model path for
whichever you want to serve.</p>
<h1 id="conclusion">Conclusion</h1>
<h2 id="notes-on-each">Notes on Each</h2>
<p><strong>Qwen3 8B</strong>: Pretty fast. The thinking thing was a bit slow but not bad. It mostly
stuck to the instructions I gave it, like "max 3 paragraphs", and was reasonably
creative. It was also kind of bland and generic, for better or worse.</p>
<p><strong>DeepSeek-R1-Distill-Qwen-7B</strong>: Also ok speed. It had a hard time following
instructions and sticking to the constraints that I'd tried to put in place, like that
same 3 paragraph limit. Overall I was pretty dissapointed given its reputation.</p>
<p><strong>Mistral Nemo 12B</strong> The direct responses felt fast even though it ran at about half
the tokens per second, due to it not working in Reasoning mode. The writing style of
Nemo was by far the best, and it adhered much better to the prompts.</p>
<p><strong>Mistral Nemo 24B</strong> Not enough RAM. I'd figured I could close things and make it work
but it's not worth it. Maybe on a better computer.</p>
<h2 id="not-enough-tokens">Not enough tokens</h2>
<p>The <code>8192</code> tokens they all use got to feeling pretty small, pretty fast. I tried to
push the continuity with a "Choose your own adventure" style conversation and it did
well until it just kind of stopped. I think I've seen claude crushing context down when
it gets near the limit, I wonder if there's a way to do that with this too, or get it
to do some sort of context shifting where it restarts with just the later context when
it runs low. Maybe I can run it in server mode and have a better client manage that,
but that seems like its own post. If you want to run models through an API
rather than locally, <a href="/openrouter-ai-tools.html">OpenRouter</a> is worth a look.</p>
<h2 id="winner">Winner</h2>
<p>Nemo was the pretty clear winner. I'm not sure if that's because of the size of the
model or something else, but it was a significant step up in its ability to stick to
the prompt's instructions, and the writing style was better too.</p>


Kyle Pericak

"It works in my environment"

Categories

Tags