<p>The <a href="/bot-wiki.html">bot-wiki post</a> introduced a structured wiki
designed for machine consumption. The whole point was to build
something a RAG pipeline could actually use. This is that pipeline.</p>
<h2 id="the-stack">The stack</h2>
<p>The wiki has 21 markdown pages with structured frontmatter:
title, summary, keywords, scope. The RAG pipeline is a single
Python script that:</p>
<ol>
<li>Embeds all 21 pages into vectors using OpenRouter's API</li>
<li>Stores them in a FAISS flat index (129 KB, committed to git)</li>
<li>At query time, embeds the question, finds the nearest pages,
and feeds them to an LLM for a grounded answer</li>
</ol>
<p>Dependencies: <code>faiss-cpu</code>, <code>openai</code> (for the OpenAI-compatible
API), and <code>pyyaml</code>. That's it.</p>
<h2 id="why-faiss">Why FAISS</h2>
<p><a href="https://github.com/facebookresearch/faiss">FAISS</a> is Meta's
library for similarity search over dense vectors. It runs
in-process, no server needed. You give it vectors, it gives
you nearest neighbors. The index is a flat file you can
commit to git.</p>
<p>I picked it because this is 21 vectors. There's no need for
a database, a server, or a managed service. <code>pip install faiss-cpu</code>, build an index, save it, done. The index is
129 KB. It loads in milliseconds.</p>
<p>If you're working at larger scale or want more features,
the main alternatives are:</p>
<ul>
<li><a href="https://github.com/chroma-core/chroma">Chroma</a>: embedded
vector database with a nice Python API. Good if you want
metadata filtering or persistence without managing files
yourself. Still runs locally.</li>
<li><a href="https://github.com/qdrant/qdrant">Qdrant</a>: vector search
engine written in Rust. Runs as a server, supports
filtering and payloads. Better fit when you have millions
of vectors or need a proper API.</li>
<li><a href="https://www.pinecone.io/">Pinecone</a>: fully managed cloud
vector database. Zero ops, but you're paying per query and
your data lives on someone else's servers.</li>
</ul>
<p>If you're already running PostgreSQL,
<a href="https://github.com/pgvector/pgvector">pgvector</a> is worth
a look. It adds vector columns and similarity search to
your existing database. No new service to run, and your
vectors live next to your relational data.</p>
<p>For 21 pages committed to a git repo, FAISS is the obvious
choice. No dependencies beyond the pip package, no running
services, no accounts.</p>
<h2 id="why-whole-page-chunks">Why whole-page chunks</h2>
<p>The wiki pages are short, 18 to 65 lines each. Splitting them
by section would lose context without gaining anything. The
frontmatter was designed to enrich embeddings, so each chunk
looks like this:</p>
<pre><code>Title: Claude Code
Summary: Anthropic's CLI-based AI coding assistant. Runs
  in terminal, edits files directly, supports MCP tool
  integrations and custom hooks.
Keywords: claude-code, anthropic, cli, ai-coding-assistant,
  mcp, hooks
Scope: Covers Claude Code setup, configuration, MCP
  integration, hooks, and usage patterns.

Claude Code is Anthropic's CLI tool for AI-assisted software
engineering. It operates directly in the terminal...
</code></pre>
<p>Prepending the summary, keywords, and scope to the body gives
the embedding model more signal about what the page covers.
21 chunks, 5,434 tokens total.</p>
<h3 id="chunking-in-general">Chunking in general</h3>
<p>Most RAG guides recommend chunks of 256 to 512 tokens. That's
a reasonable default for long documents where you want to
retrieve a specific paragraph, not the whole thing. Smaller
chunks give more precise retrieval. Larger chunks give the
LLM more context per result.</p>
<p>The tradeoffs:</p>
<ul>
<li><strong>Too small</strong> (under 100 tokens): chunks lose meaning on
their own. "See above" or "as mentioned" with no referent.
Retrieval finds fragments, not answers.</li>
<li><strong>Too large</strong> (over 1000 tokens): embeddings become diluted.
A chunk about five topics matches all five weakly instead
of one strongly. You also burn more of the LLM's context
window per retrieved result.</li>
<li><strong>Overlap</strong> helps when chunks split mid-thought. A common
pattern is 512-token chunks with 50-100 tokens of overlap.
This adds redundancy but reduces the chance of cutting a
relevant passage in half.</li>
</ul>
<p>The wiki pages average about 260 tokens each, which lands
right in the sweet spot. They're also self-contained by
design, each page covers one topic with clear boundaries.
No reason to split them further.</p>
<p>If I were indexing longer documents (blog posts, docs, book
chapters), I'd split by section headers first, then by token
count within sections. Semantic boundaries beat arbitrary
ones.</p>
<h2 id="building-the-index">Building the index</h2>
<pre><code class="language-bash">python3 -m venv .venv
.venv/bin/pip install -r bin/requirements-rag.txt
.venv/bin/python bin/wiki-rag.py build
</code></pre>
<pre><code>Embedding 21 wiki pages...
Indexed 21 pages, 1536 dimensions, 5434 tokens
Index: 129,069 bytes, metadata: 29,663 bytes
Saved to blog/markdown/wiki/.index
</code></pre>
<p>The embedding model is <code>text-embedding-3-small</code> via OpenRouter.
1536 dimensions per vector. The FAISS index uses <code>IndexFlatIP</code>
(inner product on normalized vectors = cosine similarity). No
approximate search needed at this scale.</p>
<p>The index and a <code>metadata.json</code> file (slug, title, summary,
full chunk text for each page) get saved to
<code>blog/markdown/wiki/.index/</code>. Both files are committed to git.
Any tool that wants to query the wiki can just read the index
directly without needing the embedding API.</p>
<p>The build is a manual step, not part of <code>npm run build</code>. Wiki
content doesn't change often, and there's no reason to pay for
embeddings on every deploy.</p>
<h2 id="querying">Querying</h2>
<p>A query hits two models. First, the question gets embedded
into a vector using the same embedding model that indexed
the wiki pages. FAISS compares that vector against the 21
stored vectors and returns the closest matches. Those
matches are wiki pages that are semantically similar to the
question, not keyword matches, but meaning matches.</p>
<p>Then the retrieved pages get passed to an LLM (Claude
Sonnet) as context, along with the original question. The
LLM reads the pages and writes an answer grounded in what
they actually say. Without the retrieval step, the LLM
would just guess from its training data. With it, the LLM
answers from your specific documents.</p>
<p>That's the "retrieval-augmented" part of RAG: find relevant
documents first, then generate an answer from them.</p>
<pre class="mermaid">
graph LR
    Q[Question] --> E[Embed question]
    E --> F[FAISS search]
    F --> R[Top 3 wiki pages]
    R --> P[Build prompt]
    P --> LLM[Claude Sonnet]
    LLM --> A[Answer + sources]
</pre>
<pre><code class="language-bash">.venv/bin/python bin/wiki-rag.py query \
  "What MCP servers does this project use?"
</code></pre>
<pre><code>Question: What MCP servers does this project use?

Top 3 matches:
  1. [0.644] MCP Integrations (wiki/mcp)
  2. [0.563] Linear MCP (wiki/mcp/linear)
  3. [0.524] GA4 MCP (wiki/mcp/ga4)

Answer:
Based on the provided context, this project uses three MCP
servers:

1. Playwright, for browser automation
2. Linear, for project management
3. GA4, for accessing website traffic data and reports

All MCP servers run as subprocesses communicating via JSON-RPC
over stdio, and are configured in ~/.claude.json or a
project-level .claude.json file.
</code></pre>
<p>The retrieval nailed the right pages. Here's a different query:</p>
<pre><code class="language-bash">.venv/bin/python bin/wiki-rag.py query "How do I run security scans?"
</code></pre>
<pre><code>Question: How do I run security scans?

Top 3 matches:
  1. [0.484] Security Toolkit (wiki/devops/security-toolkit)
  2. [0.439] DevOps &#x26; Security (wiki/devops)
  3. [0.343] OpenClaw Kubernetes Security (wiki/openclaw/kubernetes)
</code></pre>
<p>The answer included the actual docker commands from the security
toolkit wiki page, copy-paste ready. The third result
(Kubernetes security) is a reasonable stretch, the word
"security" overlaps, but the top two are exactly right.</p>
<h2 id="the-script">The script</h2>
<p>The interesting parts are small. Embedding is a single API call
since all 21 pages fit in one batch:</p>
<pre><code class="language-python">client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)
response = client.embeddings.create(
    model="openai/text-embedding-3-small",
    input=texts,
)
</code></pre>
<p>OpenRouter's embedding endpoint is OpenAI-compatible, so the
<code>openai</code> Python package works with just a base URL swap.</p>
<p>FAISS search returns similarity scores and indices:</p>
<pre><code class="language-python">scores, indices = index.search(query_vec, k)
for score, idx in zip(scores[0], indices[0]):
    page = metadata[idx]
    # score is cosine similarity (0 to 1)
</code></pre>
<p>The retrieved pages get stuffed into a prompt with instructions
to only use the provided context and cite sources. The LLM
(Claude Sonnet via OpenRouter) generates the answer.</p>
<h2 id="cost">Cost</h2>
<p>Indexing 21 pages used 5,434 tokens at
<code>text-embedding-3-small</code> rates ($0.02/1M tokens). That's
$0.0001. Each query embeds a short question (maybe 20 tokens)
plus one chat completion. A query costs roughly $0.002 to
$0.01 depending on the response length.</p>
<p>Building the index and running test queries cost $0.03
total on OpenRouter. That covers the embedding API calls
and the chat completions for query-mode answers. The
development itself (writing the script, iterating on the
blog post) ran through Claude Code on an Anthropic Max
subscription, so that cost isn't on the OpenRouter bill.</p>


<p>I added a new top-level section to this site: the
<a href="/wiki.html">Bot-Wiki</a>.</p>
<p>It looks like a wiki. It reads like a wiki. But it's not really for
humans. It's a structured knowledge base designed for AI agents and
RAG pipelines to consume.</p>
<h2 id="why-a-wiki-when-i-already-have-a-blog">Why a wiki when I already have a blog?</h2>
<p>Blog posts are stories. They have a narrative arc, opinions, tangents.
They're written for people who want to understand how I got from point
A to point B. That's great for learning from my experience but
terrible for a robot trying to answer "what MCP servers does this
project use?"</p>
<p>The wiki is the opposite. Each page is a factual reference card.
Structured metadata, explicit scope declarations, keyword lists for
embedding search, and cross-references for graph traversal. The tone
is robots writing notes for other robots.</p>
<h2 id="whats-in-it">What's in it</h2>
<p>The wiki covers the same ground as my recent blog posts but organized
by topic rather than chronology:</p>
<ul>
<li><strong>AI Tools</strong>: Claude Code, OpenCode, OpenRouter, local LLMs, image gen</li>
<li><strong>MCP Integrations</strong>: Playwright, Linear, GA4</li>
<li><strong>OpenClaw</strong>: the agent, its skills, Kubernetes security</li>
<li><strong>DevOps</strong>: security toolkit, lint toolkit, Ruler</li>
<li><strong>Blog Architecture</strong>: how this site is built, the markdown pipeline,
testing setup</li>
</ul>
<p>Each page has frontmatter designed for machine consumption:</p>
<pre><code class="language-yaml">title: "Claude Code"
summary: "CLI AI coding assistant by Anthropic with MCP support"
keywords:
  - claude-code
  - anthropic
  - mcp
related:
  - wiki/ai-tools/opencode
  - playwright-mcp
scope: "Setup, config, MCP integration, and usage patterns"
last_verified: 2026-03-10
</code></pre>
<p>The <code>summary</code> and <code>scope</code> fields help a RAG system decide whether to
retrieve the full page. The <code>keywords</code> list feeds embedding search.
The <code>related</code> links let an agent traverse the knowledge graph.
<code>last_verified</code> tells the agent how stale the content might be.</p>
<h2 id="how-this-connects-to-rag">How this connects to RAG</h2>
<p>This is step one of a larger plan. Next I'm building a RAG pipeline
that indexes the wiki content, generates embeddings, and lets agents
query it during conversations. The wiki's structured metadata makes
chunking and retrieval much cleaner than trying to RAG over blog
posts directly.</p>
<p>The directory structure maps to the URL structure which maps to the
knowledge hierarchy. An agent asking about MCP servers gets pointed
to <code>wiki/mcp/</code> and its children, not to a 2000-word blog post that
mentions MCP in passing.</p>
<h2 id="how-its-built">How it's built</h2>
<p>The wiki reuses the blog's existing infrastructure. A new <code>WikiService</code>
reads markdown from <code>markdown/wiki/</code> instead of <code>markdown/posts/</code>,
builds a tree from the directory structure, and generates navigation
automatically. Each <code>index.md</code> file gets a tree of its children
rendered above its own content.</p>
<p>The routing plugs into the same Next.js catch-all that handles blog
posts. Wiki pages get the same layout, sidebar, and responsive
behavior. They just have different content and metadata.</p>
<p>No images in the wiki. It's all text, all the time. Robots don't
need hero images.</p>
<p>You can find it at <a href="/wiki.html">/wiki.html</a>, or use the new Wiki
link in the header.</p>


Kyle Pericak

"It works in my environment"

Categories

Tags