<h2 id="table-of-contents">Table of contents</h2>
<ul>
<li><a href="#why-build-my-own">Why build my own?</a></li>
<li><a href="#key-features">Key Features</a>
<ul>
<li><a href="#memory">Memory</a>
<ul>
<li><a href="#storage">Storage</a></li>
<li><a href="#search">Search</a></li>
<li><a href="#active-recall">Active recall</a></li>
</ul>
</li>
<li><a href="#discord-gateway">Discord gateway</a>
<ul>
<li><a href="#gateway-webhook">Gateway webhook</a></li>
<li><a href="#mention-detection">Mention detection</a></li>
<li><a href="#thread-tracking">Thread tracking</a></li>
<li><a href="#catchup-on-reconnect">Catchup on reconnect</a></li>
</ul>
</li>
<li><a href="#scheduling">Scheduling</a>
<ul>
<li><a href="#commitment-tracking">Commitment tracking</a></li>
<li><a href="#cronjobs-out-of-process">CronJobs (out-of-process)</a></li>
</ul>
</li>
<li><a href="#persona">Persona</a></li>
<li><a href="#self-improvement-and-observability">Self-improvement and observability</a>
<ul>
<li><a href="#pai-self-improver-cron">pai-self-improver cron</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<h1 id="why-build-my-own">Why build my own?</h1>
<p>OpenClaw is really feature rich, popular, and arguably useful. It's also just... not
mine. I rolled it out <a href="/openclaw-mvp.html">for a prior post</a> as a test and while it
was neat, I learned very little and didn't trust it since I didn't understand what
it was doing or why.</p>
<p>I decided it'd be fun to build some of the features I find most useful from it into my
own AI assistant tool, and keep the scope much more narrow to my own specific usecases.</p>
<p>I named it Pai, for "Pericak AI".</p>
<h1 id="key-features">Key Features</h1>
<h2 id="memory">Memory</h2>
<p>It's nice when talking to Pai on discord, for it to remember things. ChatGPT and similar
tools have been doing it for so long now it's kind of expected.</p>
<h3 id="storage">Storage</h3>
<p>Three plain-markdown files are stored in a k8s PVC that I back up to Google cloud.</p>
<ul>
<li><code>MEMORY.md</code>: sectioned by <code>##</code> headers</li>
<li><code>daily/YYYY-MM-DD.md</code>: rolling notes</li>
<li><code>COMMITMENTS.md</code>: YAML-fenced follow-ups</li>
</ul>
<p>A typed MCP wraps the files and exposes <code>memory_save</code>,
<code>memory_search</code>, <code>memory_recall</code>, <code>memory_get</code>, <code>memory_list</code>,
<code>memory_commitment_due</code>, <code>memory_commitment_done</code>, <code>memory_promote</code>.</p>
<p>Example of a <code>COMMITMENTS.md</code> block:</p>
<pre><code class="language-markdown">---
id: c-2026-05-08-001
status: pending
precision: precise
due: 2026-05-08T19:00:00Z
scope: channel:1482815120000000000
---
Remind Kyle about the dentist appointment at 3 PM.
</code></pre>
<p>The MCP exists rather than letting Pai use Read/Glob/Grep because
the typed contract reduces malformed entries and the commitment
lifecycle wants <code>due</code> / <code>done</code> semantics in the protocol. The files
themselves are plain markdown, auditable without the MCP running.</p>
<p>This tradeoff of using
an MCP tool kind of sucks, I'm not totally sold on it over a skill. My claude startup
time got kind of bad and I had to tune some settings, so now the MCP servers may not be
ready by the time it tries to use them.</p>
<h3 id="search">Search</h3>
<p>Builtin Python <a href="https://en.wikipedia.org/wiki/Okapi_BM25">BM25 (Best Matching 25</a>
over the markdown. Pai runs it on-demand to try and remember things.</p>
<p><code>memory_search</code> returns ranked hits with provenance: <code>{path, line, snippet, score}</code>. Pai calls it mid-turn, when it decides it needs to look something up.
No embeddings, no API keys.</p>
<h3 id="active-recall">Active recall</h3>
<p>A subagent calls BM25 proactively before Pai tries to respond, enriching the chat context
with memory details. This slows things down but gets better answers, I'm not sold on
it actually being better than active search yet but it's conceptually neat.</p>
<p>This is directly inspired by openclaw's Active Memory plugin pattern.</p>
<h2 id="discord-gateway">Discord gateway</h2>
<h3 id="gateway-webhook">Gateway webhook</h3>
<p>To watch for Discord activity there's a long-lived outbound websocket to
Discord's <code>gateway.discord.gg</code> from a K8s Deployment.</p>
<p>The gateway's got some other bells and whistles:</p>
<ul>
<li>A per-session queue with a serialization lock handled by an asyncio coroutine
<ul>
<li>A "session" = a Discord channel or thread ID</li>
<li>If you message @Pai 3 times, it doesn't fire 3 responses off at once due to the lock. It answers #1, then probably merges #2 and #2.</li>
</ul>
</li>
<li>Stored transcripts: per-session rolling records of recent dialogue, basically allows look-back to prior messages</li>
<li>Idle thread sweeper: Pai always replies in threads, never in the main channel. The threads accumulate.
<ul>
<li>It's a coroutine that runs every 5 minutes, if something is idle for an hour post a farewell, unbind the thread, and delete the local transcript</li>
</ul>
</li>
<li>Periodic review every 15 minutes for unmentioned messages: Just polls for anything it classifies as ~"should respond to" when not mentioned directly</li>
<li>Health server on :8080/healthz - tiny aiohttp web app used for k8s liveness probe</li>
</ul>
<p>There's no service or ingress pointing at it. Only K8s itself talks to it.</p>
<h3 id="mention-detection">Mention detection</h3>
<p>I had to fix this up myself, it handles three forms of mentions</p>
<ol>
<li>user mentions (<code>&#x3C;@id></code>) via <code>msg.mentions</code></li>
<li>literal <code>&#x3C;@bot_user_id></code> substring</li>
<li>role mentions matching any role the bot itself holds.</li>
</ol>
<p>The role was needed in my case because the role is also called Pai, it kept geting mixed up.</p>
<h3 id="thread-tracking">Thread tracking</h3>
<p>Threads bind on mention OR on Pai's own posts. Pai creates a thread to reply to a
parent-channel mention; without auto-bind, follow-ups in that thread silently get ignored.</p>
<h3 id="catchup-on-reconnect">Catchup on reconnect</h3>
<p>Helps avoid orphaned messages from things like mentions sent on pod reset.
<code>_catchup</code> runs once per <code>on_ready</code>.</p>
<h2 id="scheduling">Scheduling</h2>
<p>Two layers: in-process for items requested soon, and creates K8s CronJobs
for daily or recurring schedules.</p>
<h3 id="commitment-tracking">Commitment tracking</h3>
<p>Every 60 seconds it polls <code>COMMITMENTS.md</code> for entries with <code>status: pending AND due &#x3C;= now</code>, spawns Pai with a small tool list (<code>send_message</code>, <code>create_thread</code>,
<code>memory_commitment_done</code>) to execute on the commitments each.</p>
<p>Each commitment carries a <code>precision</code> field, either <code>precise</code> (for
explicit "remind me at...") or <code>soft</code> (for inferred follow-ups),
which Pai uses to phrase the message. Same delivery path either
way.</p>
<p>Inspired by openclaw's inferred-commitments.</p>
<h3 id="cronjobs-out-of-process">CronJobs (out-of-process)</h3>
<p>Per-task YAML in <code>infra/ai-agents/cronjobs/helm/templates/</code>. The
<a href="/cron-event-triggered-ai-agents-k8s.html">K8s cron agent post</a> covers how I built
that framework.</p>
<p>At the time of writing this I have it summarizing the news, writing curated
horroscopes for my wife and kids, making nightly SEO improvements to the site, and
syncing any new blog posts to twitter bluesky and mastadon. I've also got an experimental
one called "autolearn" that is meant to look into emerging technologies, but it's kind
of bad so far.</p>
<h2 id="persona">Persona</h2>
<p><code>.claude/agents/pai.md</code> is one inline definition: voice, Discord
behaviour, security rules (Discord messages are untrusted input),
tool list, memory directives, browser scope. Repo-level <code>CLAUDE.md</code>
adds cross-project rules.</p>
<p>Per-user factual content (Kyle, Kara, family, projects) lives in
<code>MEMORY.md</code> and surfaces via recall, not in a separate persona
file. That's closer to what openclaw's <code>USER.md</code> is supposed to do
without the auto-injection token cost.</p>
<h2 id="self-improvement-and-observability">Self-improvement and observability</h2>
<p>A PostToolUse hook emits structured JSON per tool call. A DaemonSet ships
container stdout to OpenObserve's <code>k8s_logs</code> stream.</p>
<p>OpenObserve MCP exposes a bunch of tools that let Pai interact with it.</p>
<h3 id="pai-self-improver-cron">pai-self-improver cron</h3>
<p>Daily CronJob runs at 09:00 UTC. It queries the last 24h of failures,
clusters by normalized signature, threshold 3+ recurrences, and caps 5
proposals per run. Each run produces one Linear issue (label
<code>pai-self-improver</code>) and one Discord summary.</p>


<h2 id="table-of-contents">Table of contents</h2>
<ul>
<li><a href="#the-goal">The goal</a></li>
<li><a href="#the-journalist-agent">The journalist agent</a></li>
<li><a href="#how-the-pieces-connect">How the pieces connect</a></li>
<li><a href="#controller-and-runtime">Controller and runtime</a></li>
<li><a href="#agenttask-crd">AgentTask CRD</a></li>
<li><a href="#openrouter-for-api-auth">OpenRouter for API auth</a></li>
<li><a href="#permissions-with-allowedtools">Permissions with allowedTools</a></li>
<li><a href="#webhook-trigger-on-demand">Webhook: trigger on demand</a></li>
<li><a href="#helm-chart-and-deployment">Helm chart and deployment</a></li>
</ul>
<h2 id="the-goal">The goal</h2>
<p>I want AI agents that run autonomously on a cron schedule or in
response to events, managed by Kubernetes.</p>
<p>The demo task is a daily AI news digest. Every morning at 8am,
an agent searches the web for yesterday's AI news, writes a
digest to the wiki, and commits. It's also triggerable on demand
via webhook.</p>
<p>K8s handles the scheduling. A custom controller watches a CRD
and creates Jobs. The agent runs in a container, writes to a
shared volume, and the controller tracks the result. Agents
commit locally but don't push. I review and push manually.</p>
<h2 id="the-journalist-agent">The journalist agent</h2>
<p>I wrote a new Claude Code agent for this:
<a href="https://github.com/kylep/multi/blob/main/.claude/agents/journalist.md">journalist</a>.
Sonnet, not Opus. This runs daily, and the task is search and
summarize. Sonnet is cheaper and good enough.</p>
<p>The agent searches for yesterday's AI news, writes a digest to
<code>wiki/journal/YYYY-MM-DD/ai-news.md</code> with source URLs for every
item, and commits. It determines the date itself via <code>date</code>.</p>
<p>Early versions used <code>WebSearch</code>, which burned through context
with HTML scraping. I built a small
<a href="/wiki/custom-tools/google-news-mcp.html">GNews MCP server</a>
that wraps the GNews API. Structured JSON responses, fewer
tokens. The agent runs six parallel queries (<code>OpenAI</code>,
<code>Anthropic</code>, <code>Google AI</code>, <code>NVIDIA AI</code>, <code>AI startup</code>, plus
tech headlines) and deduplicates the results.</p>
<p>I also wrote a
<a href="/wiki/custom-tools/discord-mcp.html">Discord MCP server</a>
so the agent posts its digest to the <code>#news</code> channel in my
Discord bot server.</p>
<h2 id="how-the-pieces-connect">How the pieces connect</h2>
<pre class="mermaid">
graph TD
    AT[daily-ai-news AgentTask] -->|watched by| CTRL[Controller]
    CTRL -->|8am cron| JOB[K8s Job]
    JOB -->|init| GIT[git fetch + reset]
    GIT -->|then| AGENT[Journalist Agent]
    AGENT -->|writes| WIKI[wiki/journal/YYYY-MM-DD/ai-news.md]
    AGENT -->|commits| PVC[Shared PVC]
    AGENT -->|posts via MCP| DISCORD[Discord #news]
    WH[Webhook :8080] -->|creates| AT
</pre>
<p>The controller is a Go binary running as a Deployment. It watches
<code>AgentTask</code> custom resources in the <code>ai-agents</code> namespace.</p>
<p>Each Job has two containers:</p>
<ol>
<li><strong>Init container</strong> (<code>alpine/git</code>): fetches the repo, resets
to the configured branch, then <code>chown</code>s the workspace to
uid 1000 (the agent user).</li>
<li><strong>Main container</strong> (<code>ai-agent-runtime</code>): writes MCP config,
then runs <code>claude --mcp-config ... --agent journalist -p "..." --allowedTools ...</code></li>
</ol>
<h2 id="controller-and-runtime">Controller and runtime</h2>
<p>The
<a href="https://github.com/kylep/multi/tree/main/infra/ai-agents/agent-controller/pkg/controller">controller</a>
is a Go reconcile loop that lists all AgentTasks every 30 seconds.
For scheduled tasks, it compares <code>lastRunTime</code> against the cron
expression and creates a Job when due. For manual and webhook
tasks, it creates a Job immediately when phase is <code>Pending</code>.</p>
<p>The runtime image is an Alpine container with Claude Code and
OpenCode baked in (<code>infra/ai-agents/ai-agent-runtime/Dockerfile</code>). Runs
as uid 1000.</p>
<h2 id="agenttask-crd">AgentTask CRD</h2>
<p>The custom resource definition gives <code>kubectl get agenttasks</code>
for free. Here's the daily news task:</p>
<pre><code class="language-yaml">apiVersion: agents.kyle.pericak.com/v1alpha1
kind: AgentTask
metadata:
  name: daily-ai-news
spec:
  agent: journalist
  runtime: claude
  prompt: >-
    Search for yesterday's most notable AI news...
  schedule: "0 8 * * *"
  trigger: scheduled
  readOnly: false
  allowedTools: >-
    WebSearch,WebFetch,Read,Glob,Grep,Write,
    Bash(mkdir *),Bash(git add *),
    Bash(git commit *),Bash(date *)
</code></pre>
<p>The controller splits <code>allowedTools</code> on commas and passes each
one as a separate <code>--allowedTools</code> flag to Claude Code. This is
how you run Claude Code autonomously without
<code>--dangerously-skip-permissions</code>.</p>
<table>
<thead>
<tr>
<th>Field</th>
<th>Purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>agent</code></td>
<td>Name from <code>.claude/agents/</code></td>
</tr>
<tr>
<td><code>runtime</code></td>
<td><code>claude</code> or <code>opencode</code></td>
</tr>
<tr>
<td><code>prompt</code></td>
<td>The <code>-p</code> argument</td>
</tr>
<tr>
<td><code>schedule</code></td>
<td>Cron expression (empty = one-shot)</td>
</tr>
<tr>
<td><code>trigger</code></td>
<td><code>manual</code>, <code>scheduled</code>, or <code>webhook</code></td>
</tr>
<tr>
<td><code>readOnly</code></td>
<td>Whether this agent modifies the repo</td>
</tr>
<tr>
<td><code>allowedTools</code></td>
<td>Comma-separated tool permissions</td>
</tr>
</tbody>
</table>
<h2 id="openrouter-for-api-auth">OpenRouter for API auth</h2>
<p>Claude Code normally uses <code>ANTHROPIC_API_KEY</code> to talk to
Anthropic's API directly. In a container, I route it through
<a href="/openrouter-ai-tools.html">OpenRouter</a> instead. One API key
for all model providers, unified billing.</p>
<p>The Helm chart's Secret injects these env vars into every
agent container:</p>
<pre><code class="language-yaml">ANTHROPIC_BASE_URL: https://openrouter.ai/api
ANTHROPIC_AUTH_TOKEN: &#x3C;openrouter-key>
ANTHROPIC_API_KEY: ""
CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: "1"
</code></pre>
<p><code>ANTHROPIC_API_KEY</code> must be empty or Claude Code tries to use
it and the auth conflicts.</p>
<h2 id="permissions-with-allowedtools">Permissions with allowedTools</h2>
<p>Claude Code prompts for permission before using tools. In a
container with no TTY, it just hangs. The fix is
<code>--allowedTools</code>, which auto-approves specific tools.</p>
<p>Each tool needs its own flag. Comma-separated doesn't work:</p>
<pre><code class="language-bash"># wrong
claude -p "..." --allowedTools "WebSearch,Write,Read"

# right
claude -p "..." \
  --allowedTools "WebSearch" \
  --allowedTools "Write" \
  --allowedTools "Read"
</code></pre>
<p>Tools with specifiers use parentheses for scoping:</p>
<pre><code class="language-bash">--allowedTools 'Bash(mkdir *)'    # only mkdir
--allowedTools 'Bash(git add *)'  # only git add
--allowedTools 'Write'            # any file
</code></pre>
<p>I tried scoping Write to a specific path with
<code>Write(apps/blog/blog/markdown/wiki/journal/**)</code> but it didn't
match inside the container. The pattern works locally but fails
when the working directory is <code>/workspace/repo</code>. Unrestricted
<code>Write</code> works. The agent definition and Bash scoping still
constrain what gets written.</p>
<h2 id="webhook-trigger-on-demand">Webhook: trigger on demand</h2>
<p>The controller exposes <code>:8080/webhook</code> for on-demand runs.
Requires a bearer token via <code>AI_WEBHOOK_TOKEN</code> env var (skipped
if empty for local dev):</p>
<pre><code class="language-bash">curl -X POST http://localhost:8080/webhook \
  -H "Authorization: Bearer $AI_WEBHOOK_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "agent": "journalist",
    "prompt": "Search for today AI news...",
    "allowedTools": "WebSearch,WebFetch,Read,Write,..."
  }'
</code></pre>
<p>The handler creates an AgentTask CR with the allowedTools, and
the reconcile loop picks it up.</p>
<h2 id="helm-chart-and-deployment">Helm chart and deployment</h2>
<p>The
<a href="https://github.com/kylep/multi/tree/main/infra/ai-agents/agent-controller/helm">Helm chart</a>
packages everything: CRD, controller Deployment, ServiceAccount,
RBAC, Secrets, and PVC. hostPath volume, single node.</p>
<pre><code class="language-bash">helm install agent-controller ./helm \
  -n ai-agents --create-namespace \
  -f values-override.yaml
</code></pre>
<p>The OpenRouter key has special characters that <code>--set</code> mangles.
Use a values file instead:</p>
<pre><code class="language-yaml">secrets:
  openrouterApiKey: "&#x3C;your-openrouter-key>"
  discordBotToken: "&#x3C;your-discord-bot-token>"
  discordGuildId: "&#x3C;your-guild-id>"
repo:
  branch: kyle/blog-k8s-autonomous-agents-mvp
</code></pre>


Kyle Pericak

Pai: A Personalized Openclaw-Inspired Assistant

My First Cron/Event-Triggered AI Agents on K8s

Trying to Secure OpenClaw using Kubernetes