<p>After building a <a href="/ai-security-toolkit.html">security scanning toolkit</a>, I wanted the
same pattern for linting. Tedious rules that slow developers down are trivial for a
coding agent to stick to, and it'll help provide some baseline quality.</p>
<p>I also don't want the agent to ever make a mess, and knowing they love to git commit
constantly, I'm leaning again on pre-commit hooks to keep things deterministic.</p>
<h1 id="megalinter-is-slow">MegaLinter is SLOW</h1>
<p>I tried <a href="https://megalinter.io/">MegaLinter</a> first. It's a
popular containerized linter that bundles 100+ tools. The full
image is 17GB. The "cupcake" flavor with common languages is
10GB. Even the language-specific flavors are 5GB on disk.</p>
<p>The scan took over five minutes on a monorepo with a few
thousand files. That's not something I want AI agents running
on demand. So I built my own.</p>
<h1 id="introducing-kpericakai-lint-toolkit">Introducing kpericak/ai-lint-toolkit</h1>
<p>Cool thing about agentic coding is you can roll your own anything with basically no
effort. I asked it to make me my own faster one, and to run it then fix up my own
project.</p>
<p>Ended up with an image containing four linters, each the fastest in its category:</p>
<table>
<thead>
<tr>
<th>Language</th>
<th>Tool</th>
<th>Written in</th>
</tr>
</thead>
<tbody>
<tr>
<td>Python</td>
<td><a href="https://docs.astral.sh/ruff/">ruff</a></td>
<td>Rust</td>
</tr>
<tr>
<td>JS / TS</td>
<td><a href="https://biomejs.dev/">biome</a></td>
<td>Rust</td>
</tr>
<tr>
<td>Dockerfile</td>
<td><a href="https://github.com/hadolint/hadolint">hadolint</a></td>
<td>Haskell</td>
</tr>
<tr>
<td>Terraform</td>
<td><a href="https://github.com/terraform-linters/tflint">tflint</a></td>
<td>Go</td>
</tr>
</tbody>
</table>
<p>All four are single compiled binaries. No Python runtime, no
Node.js, no JVM. The final image is <strong>282MB</strong>.</p>
<p>Ruff replaces flake8, pylint, isort, and black in one tool.
Biome replaces eslint and prettier. Both are written in Rust
and finish in under a second on this monorepo.</p>
<h1 id="the-dockerfile">The Dockerfile</h1>
<p>Multi-stage build. Copy hadolint and tflint from their
official images, download ruff and biome as release binaries.</p>
<pre><code class="language-dockerfile">FROM hadolint/hadolint:v2.14.0-alpine AS hadolint
FROM ghcr.io/terraform-linters/tflint:v0.61.0 AS tflint

FROM alpine:3.21

ARG RUFF_VERSION=0.15.5
ARG BIOME_VERSION=2.4.6
ARG TARGETARCH

RUN apk upgrade --no-cache

COPY --from=hadolint /bin/hadolint /usr/local/bin/hadolint
COPY --from=tflint /usr/local/bin/tflint /usr/local/bin/tflint

# ruff: Python linter (Rust binary)
RUN if [ "$TARGETARCH" = "arm64" ]; then \
      RUFF_ARCH="aarch64"; else RUFF_ARCH="x86_64"; fi \
    &#x26;&#x26; wget -qO- \
    "https://github.com/astral-sh/ruff/releases/download/\
${RUFF_VERSION}/ruff-${RUFF_ARCH}-unknown-linux-musl.tar.gz" \
    | tar xz --strip-components=1 -C /usr/local/bin

# biome: JS/TS linter (Rust binary)
RUN if [ "$TARGETARCH" = "arm64" ]; then \
      BIOME_ARCH="linux-arm64-musl"; \
      else BIOME_ARCH="linux-x64-musl"; fi \
    &#x26;&#x26; wget -qO /usr/local/bin/biome \
    "https://github.com/biomejs/biome/releases/download/\
%40biomejs%2Fbiome%40${BIOME_VERSION}/biome-${BIOME_ARCH}" \
    &#x26;&#x26; chmod +x /usr/local/bin/biome

RUN addgroup -S linter &#x26;&#x26; adduser -S linter -G linter
RUN mkdir -p /workspace &#x26;&#x26; chown linter:linter /workspace

WORKDIR /workspace
USER linter
ENTRYPOINT ["/bin/sh"]
</code></pre>
<p>The <code>TARGETARCH</code> arg is set automatically by Docker BuildKit,
so the same Dockerfile works on both amd64 and arm64. A
non-root <code>linter</code> user runs the tools. The image builds in
under 10 seconds because every tool is a precompiled binary.</p>
<p>Source:
<a href="https://github.com/kylep/multi/tree/main/infra/ai-lint-toolkit">GitHub</a></p>
<h1 id="asking-ai-agents-to-lint">Asking AI agents to lint</h1>
<p>Same approach as the
<a href="/ai-security-toolkit.html">security toolkit</a>. A <a href="/ruler-cross-tool-ai-rules.html">Ruler</a>
file provides the commands, then <code>ruler apply</code> injects them into CLAUDE.md and any
other tool config.</p>
<p>Note that this is nondetermistic and such not dependable.</p>
<pre><code class="language-markdown"># Linting

The Docker image `kpericak/ai-lint-toolkit:0.1` bundles
ruff, biome, hadolint, and tflint in one Alpine container.

Run from the repo root. Mount the project directory
as /workspace.

## Python (ruff)
docker run --rm -v "$(pwd):/workspace:ro" \
  kpericak/ai-lint-toolkit:0.1 \
  -c "ruff check /workspace --no-cache"

## JavaScript / TypeScript (biome)
docker run --rm -v "$(pwd):/workspace:ro" \
  kpericak/ai-lint-toolkit:0.1 \
  -c "biome lint /workspace/path/to/src"

## Dockerfiles (hadolint)
docker run --rm -v "$(pwd):/workspace:ro" \
  kpericak/ai-lint-toolkit:0.1 \
  -c "find /workspace -name 'Dockerfile*' \
  -not -path '*/node_modules/*' | xargs hadolint"

## Terraform (tflint)
docker run --rm -v "$(pwd):/workspace:ro" \
  kpericak/ai-lint-toolkit:0.1 \
  -c "cd /workspace/path/to/tf &#x26;&#x26; tflint"
</code></pre>
<p>The volume is mounted read-only. Linters don't need to write
to your source tree. Ruff needs <code>--no-cache</code> to skip writing
a cache directory since the mount is read-only.</p>
<p>For biome, point it at specific source directories rather
than the repo root. Otherwise it crawls <code>node_modules</code> and
takes forever.</p>
<h1 id="running-the-linters">Running the linters</h1>
<p>Here's what each tool found on the first scan of this
monorepo.</p>
<h2 id="ruff">ruff</h2>
<pre><code>46  F401  [*] unused-import
13  F841  [-] unused-variable
11  F541  [*] f-string-missing-placeholders
 1  F821  [ ] undefined-name
Found 71 errors.
[*] 59 fixable with the `--fix` option.
</code></pre>
<p>71 findings across all the Python in the monorepo. Most are
in older projects (kytrade, mods) that haven't been touched
in a while. The <code>[*]</code> markers mean ruff can auto-fix them.
Unused imports and f-string placeholders are the bulk of it.</p>
<p>Ruff scanned the entire repo in 0.5 seconds.</p>
<h2 id="biome">biome</h2>
<pre><code>7 lint/suspicious/noDoubleEquals
6 lint/style/noNonNullAssertion
4 lint/style/useConst
1 lint/correctness/useParseIntRadix
1 lint/correctness/noUnusedImports
</code></pre>
<p>19 findings. The double-equals warnings are in the blog's
route handler (<code>==</code> instead of <code>===</code>). The non-null
assertions are in test files for a game project. All fixable.</p>
<p>Biome scanned the JS/TS source in 0.7 seconds.</p>
<h2 id="hadolint">hadolint</h2>
<p>Hadolint found 14 findings across 5 Dockerfiles. The
interesting ones:</p>
<ul>
<li><strong>DL3020</strong>: Use COPY instead of ADD (2 errors in kytrade)</li>
<li><strong>DL3018</strong>: Pin versions in <code>apk add</code></li>
<li><strong>DL3013</strong>: Pin versions in <code>pip install</code></li>
<li><strong>DL4006</strong>: Set <code>SHELL</code> with <code>-o pipefail</code> before piped RUN</li>
</ul>
<p>The security toolkit's own Dockerfile got flagged for the
pipefail issue in its <code>wget | tar</code> pipe. Fair catch.</p>
<h2 id="tflint">tflint</h2>
<pre><code>Warning: terraform "required_version" attribute is required
Warning: Missing version constraint for provider "google"
Warning: [Fixable] data "aws_caller_identity" "current"
  is declared but not used
</code></pre>
<p>4 warnings across 2 Terraform directories. Missing version
constraints and an unused data source. Standard Terraform
hygiene stuff.</p>
<h1 id="size-comparison">Size comparison</h1>
<table>
<thead>
<tr>
<th>Image</th>
<th>Size on disk</th>
<th>Run time</th>
</tr>
</thead>
<tbody>
<tr>
<td>MegaLinter (full)</td>
<td>17 GB</td>
<td>5+ min</td>
</tr>
<tr>
<td>MegaLinter (cupcake)</td>
<td>10 GB</td>
<td>untested</td>
</tr>
<tr>
<td>MegaLinter (javascript)</td>
<td>5 GB</td>
<td>untested</td>
</tr>
<tr>
<td>ai-lint-toolkit</td>
<td>282 MB</td>
<td>&#x3C; 5 sec</td>
</tr>
</tbody>
</table>
<p>The custom image is 60x smaller than MegaLinter's full image
and runs in seconds. The tradeoff is that it only covers four
languages. But those are the four I need, and adding more is
one <code>wget</code> or <code>COPY</code> away.</p>
<h1 id="pre-commit-hooks-as-lazy-ci">Pre-commit hooks as lazy CI</h1>
<p>I already had semgrep and gitleaks running as pre-commit
hooks from the
<a href="/ai-security-toolkit.html">security toolkit</a>. Adding
ruff and biome follows the same pattern.</p>
<pre><code class="language-yaml">repos:
- repo: local
  hooks:
    - id: ruff
      name: Ruff
      entry: >-
        bash -c 'docker run
        -v "$(pwd):/workspace:ro" --rm
        kpericak/ai-lint-toolkit:0.1
        -c "ruff check /workspace --no-cache"'
      language: system
      stages: ["pre-commit"]
    - id: biome
      name: Biome
      entry: >-
        bash -c 'docker run
        -v "$(pwd):/workspace:ro" --rm
        kpericak/ai-lint-toolkit:0.1
        -c "biome lint /workspace/apps/blog/blog/pages
        /workspace/apps/blog/blog/components
        /workspace/apps/blog/blog/utils"'
      language: system
      stages: ["pre-commit"]
    - id: semgrep
      name: Semgrep
      entry: >-
        bash -c 'docker run
        -v "$(pwd):/workspace:ro" --rm
        kpericak/ai-security-toolkit-1:0.2
        -c "semgrep scan --config auto
        --error /workspace"'
      language: system
      stages: ["pre-commit", "pre-push"]
    - id: gitleaks
      name: Gitleaks
      entry: >-
        bash -c 'docker run
        -v "$(pwd):/workspace:ro" --rm
        kpericak/ai-security-toolkit-1:0.2
        -c "cd /workspace &#x26;&#x26; gitleaks detect
        --source ."'
      language: system
      stages: ["pre-commit", "pre-push"]
</code></pre>
<p>Every commit now gets linted for Python and JS/TS issues,
scanned for security bugs by semgrep, and checked for
leaked secrets by gitleaks. Four tools, two Docker images,
no CI server required.</p>
<p>Hadolint and tflint aren't in the hooks. Dockerfiles and
Terraform configs change rarely enough that running them
on demand through the AI agent is fine.</p>


<p>AI coding agents can write code, review PRs, and run tests. They
can't run security scans unless you give them the tools. I built a
Docker image that bundles three scanners into one container, then
wired it into my repo so any AI tool can use them.</p>
<h1 id="whats-in-the-image">What's in the image</h1>
<p>The image <code>kpericak/ai-security-toolkit-1</code> has 3 tools:</p>
<table>
<thead>
<tr>
<th>Category</th>
<th>Tool</th>
</tr>
</thead>
<tbody>
<tr>
<td>SAST (static analysis)</td>
<td><a href="https://semgrep.dev/">semgrep</a></td>
</tr>
<tr>
<td>SCA (dependency vulns)</td>
<td><a href="https://trivy.dev/">trivy</a></td>
</tr>
<tr>
<td>Secrets (files on disk)</td>
<td><a href="https://trivy.dev/">trivy</a></td>
</tr>
<tr>
<td>Secrets (git history)</td>
<td><a href="https://gitleaks.io/">gitleaks</a></td>
</tr>
<tr>
<td>Misconfig (Dockerfile, Terraform)</td>
<td><a href="https://trivy.dev/">trivy</a></td>
</tr>
<tr>
<td>Container image scanning</td>
<td><a href="https://trivy.dev/">trivy</a></td>
</tr>
</tbody>
</table>
<h1 id="the-dockerfile">The Dockerfile</h1>
<p>Multi-stage build. Copy the Go binaries from the official images,
install semgrep via pip on Alpine.</p>
<pre><code class="language-dockerfile">FROM aquasec/trivy:0.69.3 AS trivy
FROM zricethezav/gitleaks:v8.30.0 AS gitleaks

FROM python:3.12.13-alpine

RUN apk upgrade --no-cache &#x26;&#x26; apk add --no-cache nodejs git gcc musl-dev

COPY --from=trivy /usr/local/bin/trivy /usr/local/bin/trivy
COPY --from=gitleaks /usr/bin/gitleaks /usr/local/bin/gitleaks

RUN pip install --no-cache-dir semgrep

COPY .trivyignore /workspace/.trivyignore

RUN addgroup -S scanner &#x26;&#x26; adduser -S scanner -G scanner
RUN mkdir -p /workspace &#x26;&#x26; chown scanner:scanner /workspace

WORKDIR /workspace
USER scanner
ENTRYPOINT ["/bin/sh"]
</code></pre>
<p>Trivy and gitleaks are single Go binaries, so the COPY stages are all they need.
Semgrep needs Python and a C compiler for its native
extensions, which is why <code>gcc</code> and <code>musl-dev</code> are there. The base
images are pinned to specific versions for reproducible builds. A
non-root <code>scanner</code> user runs the tools so the container doesn't
operate as root. The image ends up around 800MB. Not small, but
it's a tooling image, not a production service.</p>
<p>Source:
<a href="https://github.com/kylep/multi/tree/main/infra/ai-security-toolkit-1">GitHub</a></p>
<h1 id="teaching-ai-agents-to-scan">Teaching AI agents to scan</h1>
<p>The scans are just shell commands. The objective is getting robot to run them.
I use <a href="/ruler-cross-tool-ai-rules.html">Ruler</a> to maintain a single
<code>.ruler/security.md</code> file that gets injected into CLAUDE.md,
AGENTS.md, and any other tool-specific config.</p>
<p>Here's the ruler file:</p>
<pre><code class="language-markdown"># Security Scanning

The Docker image `kpericak/ai-security-toolkit-1:0.2` bundles
semgrep, trivy, and gitleaks in one Alpine container.

Run from the repo root. Mount the project directory as /workspace.

## Static analysis (semgrep)
docker run --rm -v "$(pwd):/workspace:ro" \
  kpericak/ai-security-toolkit-1:0.2 \
  -c "semgrep scan --config auto /workspace"

## Vulnerability scanning (trivy)
docker run --rm -v "$(pwd):/workspace:ro" \
  kpericak/ai-security-toolkit-1:0.2 \
  -c "trivy fs --scanners vuln,secret,misconfig /workspace"

## Container image scanning (trivy)
docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  kpericak/ai-security-toolkit-1:0.2 \
  -c "trivy image &#x3C;image:tag>"

## Secret scanning (gitleaks)
docker run --rm -v "$(pwd):/workspace:ro" \
  kpericak/ai-security-toolkit-1:0.2 \
  -c "cd /workspace &#x26;&#x26; gitleaks detect --source ."
</code></pre>
<p>After running <code>ruler apply</code>, every AI tool in the repo knows how to
run security scans. I can tell Claude Code "scan this repo for
vulnerabilities" and it has the commands ready.</p>
<h1 id="pre-commit-hooks">Pre-commit hooks</h1>
<p>Both semgrep and gitleaks run as
<a href="https://pre-commit.com/">pre-commit</a> hooks. Every commit gets
a static analysis scan and a secret scan before it lands.</p>
<h2 id="installing-pre-commit">Installing pre-commit</h2>
<p>You need <a href="https://pre-commit.com/">pre-commit</a> installed, plus
Docker for the scanner container.</p>
<pre><code class="language-bash"># macOS
brew install pre-commit

# Linux (pip)
pip install pre-commit

# Then activate hooks in the repo
cd your-repo
pre-commit install
</code></pre>
<p>That wires <code>.pre-commit-config.yaml</code> into <code>.git/hooks/pre-commit</code>
so the hooks run automatically on every commit.</p>
<h2 id="the-hook-config">The hook config</h2>
<pre><code class="language-yaml">- id: semgrep
  name: Semgrep
  description: Static analysis scan using the security toolkit image.
  entry: bash -c 'docker run -v "$(pwd):/workspace:ro" --rm
    kpericak/ai-security-toolkit-1:0.2
    -c "semgrep scan --config auto --error /workspace"'
  language: system
  stages: ["pre-commit", "pre-push"]
- id: gitleaks
  name: Gitleaks
  description: Scan for leaked secrets using the security toolkit image.
  entry: bash -c 'docker run -v "$(pwd):/workspace:ro" --rm
    kpericak/ai-security-toolkit-1:0.2
    -c "cd /workspace &#x26;&#x26; gitleaks detect --source ."'
  language: system
  stages: ["pre-commit", "pre-push"]
</code></pre>
<p>The <code>--error</code> flag on semgrep makes it exit non-zero on findings,
which blocks the commit. Gitleaks catches patterns like API keys,
tokens, and credentials.</p>
<h1 id="running-the-scans">Running the scans</h1>
<p>Here's what each scanner produces when pointed at this monorepo.</p>
<h2 id="semgrep">semgrep</h2>
<p>The first scan found 43 findings. Most were false positives:
<code>dangerouslySetInnerHTML</code> in the blog's SSG rendering (the HTML
comes from my own markdown), <code>path.join</code> with internal filenames,
K8s manifests in old sample projects, and vendored Prism.js.</p>
<p>Two changes got it to zero:</p>
<ol>
<li>A <code>.semgrepignore</code> file to skip directories that aren't active
code: <code>samples/</code>, old infra, vendored JS, and a legacy project.</li>
<li>Inline <code>// nosemgrep</code> comments on the few remaining lines where
the finding is a known false positive, like the SSG innerHTML.</li>
</ol>
<pre><code>┌──────────────┐
│ Scan Summary │
└──────────────┘
Ran 717 rules on 485 files: 0 findings.
</code></pre>
<p>Zero findings means semgrep can run as a pre-commit hook with
<code>--error</code> and block on any new issue. No triage fatigue.</p>
<h2 id="trivy">trivy</h2>
<p>Trivy's default scanners are <code>vuln</code> and <code>secret</code>. It also has
<code>misconfig</code> and <code>license</code> scanners. Adding <code>--scanners vuln,secret,misconfig</code> enables Dockerfile and Terraform
misconfiguration checks on top of the dependency and secret
scanning.</p>
<p>The first scan found 7 npm vulnerabilities. 3 HIGH in Next.js
14 (DoS in server components), plus medium/low hits in
dompurify, lodash-es, and diff. I bumped Next.js from 14 to 15
and ran <code>npm audit fix</code> for the transitive deps. The upgrade
required no code changes. Pages Router, <code>output: 'export'</code>, and
React 18 all still work on Next.js 15.</p>
<p>With misconfig enabled, trivy also flagged Dockerfile issues:
no non-root USER, <code>cd</code> instead of <code>WORKDIR</code>, missing
HEALTHCHECK. The kind of things you'd catch in a code review
but might miss if nobody's looking.</p>
<pre><code>Report Summary
┌────────────────────────────────────────┬────────────┬───────┬─────────┬───────────────────┐
│                 Target                 │    Type    │ Vulns │ Secrets │ Misconfigurations │
├────────────────────────────────────────┼────────────┼───────┼─────────┼───────────────────┤
│ apps/blog/blog/package-lock.json       │    npm     │   0   │    -    │         -         │
├────────────────────────────────────────┼────────────┼───────┼─────────┼───────────────────┤
│ apps/blog/Dockerfile                   │ dockerfile │   -   │    -    │         3         │
├────────────────────────────────────────┼────────────┼───────┼─────────┼───────────────────┤
│ apps/blog/tf                           │ terraform  │   -   │    -    │         0         │
├────────────────────────────────────────┼────────────┼───────┼─────────┼───────────────────┤
│ infra/ai-security-toolkit-1/Dockerfile │ dockerfile │   -   │    -    │         1         │
└────────────────────────────────────────┴────────────┴───────┴─────────┴───────────────────┘
</code></pre>
<p>Zero dependency vulns. The Dockerfile misconfigs are real
findings to fix next. The policy going forward: keep SCA vulns
at 0 in active projects. Major package upgrades are acceptable
to get there. Playwright tests catch regressions from the
upgrades.</p>
<h2 id="trivy-image">trivy image</h2>
<p><code>trivy fs</code> scans source code and lock files. <code>trivy image</code>
scans a built Docker image, catching OS-level CVEs in the base
image and vulnerabilities in installed packages that don't show
up in any lock file.</p>
<p>Running it against the first version of the toolkit image found
25 vulnerabilities that <code>trivy fs</code> never saw: Alpine OS
packages, Python dependencies, and Go stdlib CVEs baked into
the trivy and gitleaks binaries.</p>
<p>Fixes for v0.2:</p>
<ol>
<li>Pinned <code>python:3.12.13-alpine</code> and added <code>apk upgrade</code> to
get the latest Alpine packages. Fixed the CRITICAL zlib CVE.</li>
<li>The Go stdlib CVEs live inside the gitleaks binary, compiled
with Go 1.25.4. No newer gitleaks release exists. Added a
<code>.trivyignore</code> with comments explaining each suppression.</li>
</ol>
<p>The policy: no HIGH or CRITICAL findings. MEDIUM and below from
upstream binaries we don't compile are accepted and documented.</p>
<h2 id="gitleaks">gitleaks</h2>
<pre><code>144 commits scanned.
no leaks found.
</code></pre>
<p>Clean. Gitleaks scans git history by default, so it catches
secrets that were committed and later deleted. Trivy also has
a secret scanner, but it only checks files on disk. If someone
committed an API key and removed it in the next commit, trivy
wouldn't see it. Gitleaks would, because the key is still in
the git log.</p>
<p>That's why both are in the image. Trivy covers dependency vulns
and current-file secrets. Gitleaks covers the git history angle.
I originally had <code>npm audit</code> in the image too, but dropped it
since trivy already scans <code>package-lock.json</code> for the same CVEs.</p>
<h1 id="the-workflow">The workflow</h1>
<p>With all this wired up, security scanning happens at three levels:</p>
<ol>
<li><strong>Every commit</strong>: semgrep and gitleaks run as pre-commit hooks.
Any new static analysis finding or leaked secret blocks the
commit.</li>
<li><strong>Scheduled</strong>: a security agent runs <code>trivy image</code> against
built container images on a schedule, triages the results,
and files issues in
<a href="/linear-mcp.html">Linear</a> for anything that needs attention.
Image scanning is too slow for pre-commit hooks but catches
OS-level CVEs that filesystem scans miss.</li>
<li><strong>On demand</strong>: I tell the AI agent "run security scans" or
include it in task instructions. The agent runs the docker
commands from its CLAUDE.md context and triages the output.</li>
</ol>
<p>The AI agent is good at triaging scan results. It can reason about
which dependency vulnerabilities actually matter for a static
export vs a running server. That's the real value of combining
scanners with AI agents: the agent reads the output instead of
just dumping a report.</p>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<h1 id="workflow">Workflow</h1>
<ol>
<li>Commit code to AWS CodeCommit</li>
<li>CloudWatch rule triggers CodePipeline</li>
<li>CodePipeline uses CodeBuild to build the Docker image</li>
<li>CodePipeline pushes the image to ECR</li>
</ol>
<hr>
<h1 id="deploy-code-to-codecommit">Deploy Code to CodeCommit</h1>
<ol>
<li>Open <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codecommit/repositories">CodeCommit</a>.</li>
<li>Create a repository</li>
<li>Go to your IAM page and edit your user.
<ol>
<li>Look for "HTTPS Git credentials for AWS CodeCommit"</li>
<li>click Generate Credentials</li>
<li>Save your credentials.</li>
</ol>
</li>
<li>Grab the HTTPS URL for your repository and <code>git clone</code> it. Use your new
credentials.</li>
<li>Commit your code (including a Dockerfile) to the CodeCommit repository.</li>
</ol>
<hr>
<h1 id="write-buildspecyml">Write buildspec.yml</h1>
<p>AWS CodeBuild will look for a <code>buildspec.yml</code> file to dictate its actions.
This is where we define how the image will be built. Put it in your CodeCommit
project's root directory.</p>
<p>Here's mine. This will build a docker image and tag it with a timestamp, then
push it to ECR as the timestamp and latest. The variables are defined inside
the build as Environment Variables.</p>
<pre><code class="language-yml"># buildspec.yml
version: 0.2

phases:
  install:
    runtime-versions:
      docker: 18
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - $(aws ecr get-login --no-include-email --region ca-central-1)
      - IMAGE_TAG=$(date +%Y.%m.%d.%I.%M)
  build:
    commands:
      - echo Build started on `date`
      - echo "Building the Docker image."
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest
</code></pre>
<hr>
<h1 id="define-the-pipeline">Define the Pipeline</h1>
<h2 id="cloud-build-role">Cloud Build Role</h2>
<p><strong>Bad Practice Alert</strong>: These steps don't apply the principal of least
privilege. You should lock down your role to be allowed to do only the minimum
that it needs to do.</p>
<ol>
<li>In the IAM page, create a new role.</li>
<li>It will be used by CodeBuild.</li>
<li>Assing the following policies (again, more than it needs)
<ul>
<li>CloudWatchLogsFullAccess</li>
<li>AWSCodeBuildAdminAccess</li>
<li>AmazonS3FullAccess</li>
<li>AmazonEC2ContainerRegistryFullAccess</li>
<li>AWSCodeCommitFullAccess</li>
</ul>
</li>
<li>Name: Assign a name, such as <code>CloudBuild-Admin</code></li>
</ol>
<h2 id="define-cloudbuild-project">Define CloudBuild project</h2>
<ol>
<li>Navigate to the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codebuild/projects">CloudBuild Console</a></li>
<li>Create build project</li>
<li>No build badge</li>
<li>Source provider: CodeCommit</li>
<li>Choose your repository</li>
<li>Choose your branch</li>
<li>Managed Image</li>
<li>Operating System: Ubuntu</li>
<li>Runtime: Standard</li>
<li>Image: aws/codebuild/standard:3.0</li>
<li>Image version: latest</li>
<li>Environment type: Linux</li>
<li>Privileged: true</li>
<li>Service Role: Existing service role. Use the one you just made.</li>
<li>Allow CodeBuild to modify this role: False (we gave it admin)</li>
<li>Additional Configuration - Environment variables (click Add to get more)
- IMAGE_REPO_NAME: (your image name)
- AWS_ACCOUNT_ID: (your numeric AWS account iD)
- AWS_DEFAULT_REGION: ca-central-1</li>
<li>Buildspec: Use a buildspec file</li>
<li>Artifacts: No artifacts</li>
<li>Logs: Give a group and stream name for CloudWatch</li>
<li>Create Build Project</li>
</ol>
<p>Now test the build by clicking "Start build". It should create your image.
You'll need to disable artifacts.</p>
<p>Once the build works and your image is pushed successfully to ECR, you can move
on to setting up a pipeline so this happens automatically.</p>
<h2 id="create-pipeline-role">Create Pipeline Role</h2>
<p><strong>Bad Practice Alert</strong>: These steps don't apply the principal of least
privilege. You should lock down your role to be allowed to do only the minimum
that it needs to do.</p>
<ol>
<li>In the IAM page, create a new role.</li>
<li>It will be used by CodeBuild.</li>
<li>Assing the following policies (again, more than it needs)
<ul>
<li>CloudWatchLogsFullAccess</li>
<li>AWSCodeBuildAdminAccess</li>
<li>AmazonS3FullAccess</li>
<li>AmazonEC2ContainerRegistryFullAccess</li>
<li>AWSCodeCommitFullAccess</li>
</ul>
</li>
<li>Name: Assign a name, such as <code>CloudBuild-Admin</code></li>
</ol>
<h2 id="define-the-pipeline-1">Define the pipeline</h2>
<ol>
<li>Open the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codepipeline/pipelines">CodePipeline console</a></li>
<li>Click Create Pipeline</li>
<li>Name the pipeline</li>
<li>Service Role: New service role</li>
<li>Role Name: Pick a name</li>
<li>Allow AWS CodePipeline to create a role</li>
<li>Next</li>
<li>Source Provider: AWS CodeCommit</li>
<li>Repository Name: Select the repository name</li>
<li>Branch name: master</li>
<li>Change detection: Amazon CloudWatch Events</li>
<li>Next</li>
<li>Build Provider: AWS CodeBuild</li>
<li>Region: Canada (Central)</li>
<li>Project Name: Create Project
<ol>
<li>Project Name: Choose a name</li>
<li>Description: Add a description</li>
<li>Environment image: Managed Image</li>
<li>Operating System: Ubuntu</li>
<li>Runtimes: Standard</li>
<li>Image: aws/codebuild/standard:3.0</li>
<li>Image version: Always use the latest</li>
<li>Environment type: Linux</li>
<li>Privileged: True</li>
<li>Service Role: New service role</li>
<li>Role Name: default name is fine</li>
<li>Advanced Configuration - Environment variables (click Add to get more)
<ul>
<li>IMAGE_REPO_NAME: (your image name)</li>
<li>AWS_ACCOUNT_ID: (your numeric AWS account iD)</li>
<li>AWS_DEFAULT_REGION: ca-central-1</li>
</ul>
</li>
<li>Build specifications: Use a buildspec file</li>
<li>Buildspec name: Leave it blank so it uses buildspec.yml</li>
<li>CloudWatch logs: True</li>
<li>Group name: pick a name</li>
<li>Stream name: pick a name</li>
<li>S3 logs: false</li>
<li>Continue to CodePipeline</li>
</ol>
</li>
<li>Next</li>
<li>Skip deploy stage, pushing to ECR is enough</li>
<li>Create pipeline</li>
</ol>
<p>This will re-run the build that you tested before. It should pass again.</p>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page contains operations-related mini-guides and minor posts.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</a></li>
<li><a href="#convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</a></li>
<li><a href="#journalctl-commands">Journalctl commands</a></li>
<li><a href="#get-real-ram-use-of-processes">Get real RAM use of processes</a>
<ul>
<li><a href="#of-ceph">of ceph</a></li>
<li><a href="#of-vms">of VMs</a></li>
</ul>
</li>
<li><a href="#bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</a></li>
<li><a href="#create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</a>
<ul>
<li><a href="#use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</a></li>
</ul>
</li>
<li><a href="#test-mtu-from-ubuntu">Test MTU from Ubuntu</a></li>
<li><a href="#find-big-files-on-mac-os">Find big files on Mac OS</a></li>
<li><a href="#renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</a></li>
<li><a href="#create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</a></li>
<li><a href="#passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</a></li>
<li><a href="#vim-tasks">VIM Tasks</a>
<ul>
<li><a href="#find--replace">Find &#x26; Replace</a></li>
<li><a href="#spell-check-commands">Spell-Check commands</a></li>
</ul>
</li>
<li><a href="#use-an-insecure-docker-registry">Use an Insecure Docker Registry</a></li>
<li><a href="#ansible-ignore-errors">Ansible ignore errors</a></li>
<li><a href="#how-to-tail--f-on-dmesg">How to tail -f on dmesg</a></li>
<li><a href="#ubuntu-downgrade-a-package">Ubuntu Downgrade a package</a></li>
<li><a href="#make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</a></li>
<li><a href="#bash-print-all-but-first-line">Bash: Print all but first line</a></li>
<li><a href="#bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</a></li>
</ul>
<h1 id="find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</h1>
<pre><code>cat /proc/net/bonding/bond0
</code></pre>
<hr>
<h1 id="convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</h1>
<pre><code class="language-bash">openssl pkcs7 -print_certs -in certnew.p7b -out cert.pem
</code></pre>
<hr>
<h1 id="journalctl-commands">Journalctl commands</h1>
<p>Watch the logs as they come in:</p>
<pre><code class="language-bash">journalctl -f
</code></pre>
<p>When journalctl logs too much and you need to pair it down:</p>
<pre><code class="language-bash"># to 30 days
journalctl --vacuum-time=30d

# to 500MB
journalctl --vacuum-size=500M
</code></pre>
<p>Show for a specific service</p>
<pre><code class="language-bash">journalctl ceph*
</code></pre>
<hr>
<h1 id="get-real-ram-use-of-processes">Get real RAM use of processes</h1>
<p>You can always use top or htop or whatever. I use this to specifically get the
info about those processes.</p>
<h2 id="of-ceph">of ceph</h2>
<pre><code>ps aux | grep ceph | grep -v -e grep -e qemu -e "\[" | while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); id=$(echo $line | awk '{print $16}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tid: $id,    \tRAM: ($size)"; done
</code></pre>
<h2 id="of-vms">of VMs</h2>
<p>for KVM</p>
<pre><code>ps aux | grep qemu | grep -v -e grep| while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); guest=$(echo $line | awk '{print $13}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tguest: $guest,   \t\tRAM: ($size)" ; done
</code></pre>
<h1 id="bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</h1>
<p>As of Mac Catalina, developing a site using self-signed certs is a bigger pain
than before. Yeah you can go get the key and add it, but that's a nuisance.
It looks like the Chrome team built something they call an
"interstitial bypass keyword" to deal with that. That's one cool feature name.</p>
<p><strong>Obvious disclaimer</strong> If you're not a developer doing this to access your own
site, don't do this.</p>
<p>To bypass the warning, type</p>
<pre><code>thisisunsafe
</code></pre>
<p>Apparently it used to be <code>badidea</code> but they rotated it since it got too well
known. They'll probably do it again someday. I suspect people are using this
on sites they "trust" without really knowing any better and getting hurt by
it.</p>
<hr>
<h1 id="create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</h1>
<p>Useful for simulating a real drive when building a test environment. I use this to test OpenStack in a VM as a Ceph or LVM back-end.</p>
<pre><code class="language-bash"># create empty file of all zeros
sudo dd if=/dev/zero of=/root/virtual-disk bs=1M count=512
# show list of loopback devices. Note its not there
losetup --list
# create the loopback device
losetup /dev/loop0 /root/virtual-disk
# confirm it worked
losetup --list
</code></pre>
<p>You can now treat it like a normal volume. Mount it, put a filesystem on it, whatever. Example:</p>
<pre><code class="language-bash">pvcreate /dev/loop0
vgcreate cinder-volumes /dev/loop0
</code></pre>
<h2 id="use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</h2>
<p>I found <a href="https://unix.stackexchange.com/questions/418322/persistent-lvm-device-with-loopback-devices-by-fstab">this useful systemd unit example</a> that didn't quite work for me, but was close. Here's mine:</p>
<p><code>vi /etc/systemd/system/loops-setup.service</code></p>
<pre><code class="language-text">[Unit]
Description=Setup loopback devices

DefaultDependencies=no
Conflicts=umount.target

Requires=lvm2-lvmetad.service mnt-host.mount
Before=local-fs.target umount.target
After=lvm2-lvmetad.service mnt-host.mount

[Service]
ExecStart=/sbin/losetup /dev/loop0 /root/virtual-disk
ExecStop=/sbin/losetup -d /dev/loop0

RemainAfterExit=yes
Type=oneshot

[Install]
WantedBy=local-fs-pre.target
</code></pre>
<p>Then enable it</p>
<pre><code class="language-bash">systemctl enable loops-setup
</code></pre>
<p>Reboot, then check <code>vgdisplay</code> (if that's your use case). It'll be there.</p>
<hr>
<h1 id="test-mtu-from-ubuntu">Test MTU from Ubuntu</h1>
<p>You would not believe the strange problems that an MTU issue can be the root cause of.</p>
<pre><code class="language-bash">ping -M do -s 1472 &#x3C;remote server>
ping -M do -s 8900 &#x3C;remote server>
</code></pre>
<hr>
<h1 id="find-big-files-on-mac-os">Find big files on Mac OS</h1>
<p>My SSD is always almost full. Here are useful commands for cleaning up:</p>
<pre><code class="language-bash"># Find all the files over 1G
sudo find / -type f -size +1G -exec ls -lh {} \; | awk '{ print $9 ": " $5 }' 2>/dev/null

# List files in Downloads directory by size
ls -lhS ~/Downloads
</code></pre>
<h1 id="renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</h1>
<pre><code class="language-bash">service ntp stop
ntpd -gq
service ntp start
</code></pre>
<hr>
<h1 id="create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</h1>
<pre><code class="language-bash">aws ecr describe-repositories --repository-names $repo_name \
    || aws ecr create-repository --repository-name $repo_name
</code></pre>
<hr>
<h1 id="passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</h1>
<p>Run <code>visudo</code> as root.</p>
<pre><code class="language-bash"># as root
export EDITOR=vim
visudo
</code></pre>
<p>Add the user (replace <code>exampleUser</code>) at the end.</p>
<pre><code class="language-text">exampleUser ALL=(ALL) NOPASSWD: ALL
</code></pre>
<p>For those who don't use vim, a quick reminder:
<code>i</code> enter to insert mode , <code>[esc]</code> to exit insert mode, <code>:wq</code> to save and quit.</p>
<hr>
<h1 id="vim-tasks">VIM Tasks</h1>
<h2 id="find--replace">Find &#x26; Replace</h2>
<pre><code class="language-bash"># replace findThis with ReplaceWithThis
:%s/findThis/ReplaceWithThis/g
</code></pre>
<h2 id="spell-check-commands">Spell-Check commands</h2>
<pre><code class="language-text"># enable/disable
:set spell spelllang=en_ca
:setlocal spell spelllang=en_ca
:set nospell

# next/last
]s
[s

# suggest change
z=

# add/remove word from dictionary
zg
zw`
</code></pre>
<hr>
<h1 id="use-an-insecure-docker-registry">Use an Insecure Docker Registry</h1>
<p>To use a registry with no HTTP cert:</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
    "insecure-registries" : ["myregistrydomain.com:5000"]
}
</code></pre>
<hr>
<h1 id="ansible-ignore-errors">Ansible ignore errors</h1>
<p>The command is <code>ignore_errors: yes</code></p>
<p>Just throw this into the task that's failing. It will still fail, but the play
won't stop.</p>
<p>Example:</p>
<pre><code class="language-yml">- name: Ensure the docker service is running
  script: ../tools/validate-docker-execute.sh
    ignore_errors: yes
</code></pre>
<hr>
<h1 id="how-to-tail--f-on-dmesg">How to tail -f on dmesg</h1>
<pre><code class="language-bash">dmesg -wH
</code></pre>
<hr>
<h1 id="ubuntu-downgrade-a-package">Ubuntu Downgrade a package</h1>
<pre><code class="language-bash"># Find the available versions under Versions:
apt-cache showpkg &#x3C;package>

# Install one of those
apt-get install &#x3C;package>=&#x3C;version>
</code></pre>
<hr>
<h1 id="make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</h1>
<p>Useful for OS that ship with python3 but not python2.
Edit the inventory, find a group of the groups and apply this var.</p>
<pre><code class="language-ini">[baremetal:vars]
ansible_python_interpreter=/usr/bin/python3
</code></pre>
<hr>
<h1 id="bash-print-all-but-first-line">Bash: Print all but first line</h1>
<p>Use <code>tail -n +2</code>.
This was unintuitive to me because +1 seems like it should do it.</p>
<pre><code class="language-bash"># Example
docker images | awk '{print $3}' | tail -n +2
</code></pre>
<hr>
<h1 id="bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</h1>
<p>Use trap.</p>
<ul>
<li><a href="http://redsymbol.net/articles/bash-exit-traps/">This is a nice post covering lots of examples</a>.</li>
<li><a href="https://github.com/GoogleCloudPlatform/endpoints-quickstart/blob/master/scripts/deploy_api.sh">Here's where I first saw it used</a></li>
</ul>
<pre><code class="language-bash">temp_file=$(mktemp)

cleanup() {
  rm "$temp_file"
}

trap cleanup EXIT
</code></pre>


<p>How to run a docker container on AWS ECS (Elastic Compute Cloud)  using the AWS
web console. A container from the Elastic Container Registry will be launched.
It will run on Amazon's <a href="https://aws.amazon.com/fargate/">Fargate Container-as-a-Service</a>.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#launch-a-container-on-fargate">Launch a container on Fargate</a></li>
<li><a href="#publish-containers-network-to-the-internet">Publish container's network to the internet</a></li>
<li><a href="#using-a-custom-domain-name">Using a custom domain name</a></li>
</ul>
<h1 id="launch-a-container-on-fargate">Launch a container on Fargate</h1>
<ol>
<li>Upload an image to AWS ECR. <a href="/aws-ecr.html">See my guide on using AWS ECR</a>.</li>
<li>Open <a href="https://ca-central-1.console.aws.amazon.com/ecs/">AWS ECS</a></li>
<li>Under Container Definition, select Configure</li>
<li>Fill in the form
<ol>
<li>For the image URL, use the URL of the image uploaded to
ECR. I was testing a reverse proxy image, so it looked like this:
<code>000000000000.dkr.ecr.ca-central-1.amazonaws.com/test/revproxy:latest</code></li>
<li>I have my nginx reverse proxy a 256MB soft limit for RAM</li>
<li>For my reverse proxy I made port mappings for port 80 and 443</li>
<li>No need to fill out any of the advanced fields</li>
</ol>
</li>
<li>Fill out the Task Definition section, click Next</li>
<li>Define your service
<ol>
<li>Without a load-balancer, the only way to access your container will be by
using <a href="https://aws.amazon.com/blogs/compute/access-private-applications-on-aws-fargate-using-amazon-api-gateway-privatelink/">PrivateLink</a>.</li>
<li>During the initial cluster create wizard you can choose an Application
Load Balancer. That seems to work well.</li>
</ol>
</li>
<li>Name the Fargate cluster, next</li>
<li>Click create</li>
</ol>
<p>It will take a little while. Once its finished you can click on View Service.</p>
<hr>
<h1 id="publish-containers-network-to-the-internet">Publish container's network to the internet</h1>
<p>Navigate to the newly created service and go to the Details tab. From there you
can see a Load Balancing section. It should have a link to the new load
balancer.</p>
<p>You can also access this page from the left nav-bar by going to Load Balancing,
then choosing the new load balancer.</p>
<p>In the Description tab here, you can find the DNS name that AWS has assigned
to this load balancer. Unfortunately you can't depend on the IP address behind
this FQDN to stay the same.</p>
<hr>
<h1 id="using-a-custom-domain-name">Using a custom domain name</h1>
<p>I'm working on that. Its not intuitive. If you know how, I'd like to hear about
it..</p>


<p>Docker supports shared memory. The default is only 64MB though which is TINY.</p>
<p>The small size became a problem during a containerized OpenStack upgrade.</p>
<p>This will be a really short post because I haven't done much playing with the
shared memory yet, I just think its really neat.</p>
<hr>
<h1 id="change-the-devshm-size-on-the-system">Change the /dev/shm size on the system</h1>
<p>This lets you change the size of shared memory on a host.</p>
<pre><code class="language-bash">mount -o remount,size=1G /dev/shm
</code></pre>
<hr>
<h1 id="update-docker-daemons-default-shared-memory-size">Update Docker Daemon's Default Shared Memory Size</h1>
<p>This will update the default across the whole Docker service.</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
  "default-shm-size": "512M"
}
</code></pre>
<hr>
<h1 id="set-a-containers-shared-memory-size">Set a Containers Shared Memory Size</h1>
<p>Here's how to change it for a specific container at runtime.</p>
<pre><code class="language-bash">docker run -it --shm-size=512M
</code></pre>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to make a Docker registry on AWS Elastic Container
Registry (ECR).
To secure the registry, least-privilege roles are created and assigned to
service account users in the AWS AIM tool</p>
<hr>
<h3 id="table-of-contents">Table of contents</h3>
<ul>
<li><a href="#create-an-aws-ecr-repository">Create an AWS ECR Repository</a></li>
<li><a href="#create-an-aws-iam-user-for-the-registry">Create an AWS IAM User For the Registry</a>
<ul>
<li><a href="#create-groups-for-ecr-access">Create Groups for ECR Access</a>
<ul>
<li><a href="#make-a-group-with-write-access">Make a group with write access</a></li>
<li><a href="#make-a-read-only-group">Make a read-only group</a></li>
</ul>
</li>
<li><a href="#create-users-for-ecr-access">Create Users for ECR Access</a></li>
</ul>
</li>
<li><a href="#connect-docker-to-the-new--registry">Connect Docker to the new  Registry</a>
<ul>
<li><a href="#install-the-aws-cli">Install the AWS CLI</a></li>
<li><a href="#configure-the-aws-cli-user">Configure the AWS CLI USER</a></li>
<li><a href="#get-the-docker-login-command">Get the docker login command</a></li>
</ul>
</li>
<li><a href="#push-images-to-the-ecr-registry">Push Images to the ECR Registry</a></li>
</ul>
<h1 id="create-an-aws-ecr-repository">Create an AWS ECR Repository</h1>
<p>Navigate to the
<a href="https://ca-central-1.console.aws.amazon.com/ecr/repositories">AWS ECR Repositories page</a>
and click Create a repository.</p>
<p>Fill in the form to name your repository.</p>
<h1 id="create-an-aws-iam-user-for-the-registry">Create an AWS IAM User For the Registry</h1>
<p>Its a good idea to have user accounts with write access, and others with
read-only access. This way you can push to the registry with your privileged
user but you never need to expose those credentials outside your development
or CI/CD environment.</p>
<h2 id="create-groups-for-ecr-access">Create Groups for ECR Access</h2>
<p>First, review the AWS Group access levels <a href="https://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr_managed_policies.html">here</a>.</p>
<h3 id="make-a-group-with-write-access">Make a group with write access</h3>
<p>Users in this group can push to existing registries but they can't use the CLI
to make new ones.</p>
<ol>
<li>Go to the <a href="https://console.aws.amazon.com/iam">AWS IAM page</a></li>
<li>Click Groups on the sidebar</li>
<li>Name the group, ex: <code>ecr-write</code></li>
<li>Attach Policy: <code>AmazonEC2ContainerRegistryPowerUser</code>, next</li>
<li>Create Group</li>
</ol>
<h3 id="make-a-read-only-group">Make a read-only group</h3>
<p>Its the same procedure as above, but use the read-only policy instead:
<code>AmazonEC2ContainerRegistryReadOnly</code>.</p>
<p>Consider also making an admin group with the
<code>AmazonEC2ContainerRegistryFullAccess</code> permission.</p>
<h2 id="create-users-for-ecr-access">Create Users for ECR Access</h2>
<p>Make two users, one for pushing and one for pulling. You can share the user
will read access to anyone who needs to consume the images.</p>
<p>To make a user:</p>
<ol>
<li>Go to the <a href="https://console.aws.amazon.com/iam">AWS IAM page</a></li>
<li>Click Users on the sidebar</li>
<li>Add User</li>
<li>Enter user name, check Programmatic Access, next</li>
<li>Choose from the groups you just created to grant read/write access, next</li>
<li>Skip the tags, next</li>
<li>Create User. Be sure to save the secret access key in your password manager.</li>
</ol>
<h1 id="connect-docker-to-the-new--registry">Connect Docker to the new  Registry</h1>
<h2 id="install-the-aws-cli">Install the AWS CLI</h2>
<p>The one from apt is really old and doesn't work. Use pip.</p>
<pre><code class="language-bash">apt-get install -y python python-pip
pip install aws awscli
</code></pre>
<p>Test it out</p>
<pre><code class="language-bash">python -m awscli help
</code></pre>
<h2 id="configure-the-aws-cli-user">Configure the AWS CLI USER</h2>
<p>When prompted, enter your key, secret key, region (ca-central-1), and output
format (none):</p>
<pre><code class="language-bash">aws configure
</code></pre>
<h2 id="get-the-docker-login-command">Get the docker login command</h2>
<p>Use the AWS CLI to fetch and execute the docker login command. You can also
export this command to a file to run it, but there's no point since the token
gets invalidated after 12 hours.</p>
<pre><code class="language-bash">$(aws ecr get-login --no-include-email --region ca-central-1)
</code></pre>
<hr>
<h1 id="push-images-to-the-ecr-registry">Push Images to the ECR Registry</h1>
<p>If you click on the Push Commands button in the ECR registry page you'll get
examples programmatically generated for your registry.</p>
<p>Pushing is kind of unintuitive. Unlike a normal <code>registry:2</code> docker registry,
ECR won't automatically create your repositories for you. You have to manually
create each one. You can use the web UI, or log in as an admin user and run the
following:</p>
<pre><code class="language-bash">aws ecr create-repository --repository-name kolla/ubuntu-source-base
</code></pre>
<p>For scripting, this is a handy way of creating the repo if it doesn't exist:</p>
<pre><code class="language-bash">aws ecr describe-repositories --repository-names $repo_name \
  || aws ecr create-repository --repository-name $repo_name
</code></pre>
<p>Then you can push.</p>
<pre><code class="language-bash">docker push $myId.dkr.ecr.ca-central-1.amazonaws.com/$repo:$tag
</code></pre>
<p>If you first create the repository, you get an error like this: <code>name unknown: The repository with name 'kolla/ubuntu-source-base' does not exist in the registry with id '...'</code></p>


<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#pricing">Pricing</a></li>
<li><a href="#write-the-cloud-build-file">Write the Cloud Build file</a></li>
<li><a href="#iam-roles">IAM Roles</a>
<ul>
<li><a href="#docker-image-to-cloud-registry">Docker Image to Cloud Registry</a></li>
<li><a href="#static-website-to-google-cloud-storage">Static Website to Google Cloud Storage</a></li>
</ul>
</li>
<li><a href="#define-cloud-build-triggers">Define Cloud Build Triggers</a></li>
</ul>
<h1 id="pricing">Pricing</h1>
<p>Current pricing data can be found <a href="https://cloud.google.com/cloud-build/pricing">here</a>.</p>
<ul>
<li>For the first 120 builds in a day, cloud build is <strong>free</strong>.</li>
<li>each build after the free quota costs $0.003 US / build-minute when using the
single core build server with no additional SSD storage</li>
</ul>
<h1 id="write-the-cloud-build-file">Write the Cloud Build file</h1>
<p>Google Cloud Build uses a file, <code>cloudbuild.yml</code>, to define the tasks that will
be executed when a build is triggered.</p>
<p>The online references for the syntax are pretty good:</p>
<ul>
<li><a href="https://cloud.google.com/cloud-build/docs/build-config">Google's Cloud-Build Documentation</a></li>
<li><a href="https://cloud.google.com/cloud-build/docs/configuring-builds/substitute-variable-values">Variables Available in cloudbuild.yaml</a></li>
</ul>
<p>Below are some example configuration files used for this site.</p>
<h1 id="iam-roles">IAM Roles</h1>
<p><a href="https://cloud.google.com/cloud-build/docs/securing-builds/configure-access-control">Official Documentation</a></p>
<ul>
<li>Cloud Build Editor</li>
<li>Cloud Build Viewer</li>
</ul>
<h2 id="docker-image-to-cloud-registry">Docker Image to Cloud Registry</h2>
<p>Cloud Build is great for keeping your Docker Registry images aligned with your
version control code.</p>
<p>In this example I use Cloud Build to build and push a <a href="/docker-pelican-image.html">pelican image</a>
to the <a href="/google-container-registry.html">Google Container Registry</a>.</p>
<p>Be sure to keep the Dockerfile code in version control. Aside from being a good
idea in general, Cloud Build can also use Git commits as a build trigger.
<a href="https://github.com/kylep/pelican">Here's my Pelican Docker image repository</a>.</p>
<pre><code class="language-yaml">steps:
- name: 'gcr.io/cloud-builders/docker'
  args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/pelican', '.' ]
images:
- 'gcr.io/$PROJECT_ID/pelican'
</code></pre>
<h2 id="static-website-to-google-cloud-storage">Static Website to Google Cloud Storage</h2>
<p>Another use for Cloud Builder is to generate static website files and then sync
them up with Cloud Storage every time the source control files or template are
changed.</p>
<p>Here's an example that triggers when I commit code to my
<a href="https://github.com/kylep/kyle.pericak.com">blog content git repository</a>. It
copies the content into <code>/workspace</code> and uses the above pelican image to render
the content and output it to the <code>output/</code> directory.</p>
<p>Once the site content is generated, the next step uses the <code>gcloud</code> image's
<code>gsutil -m rsync</code> command to upload the files to the Google Storage Bucket.</p>
<hr>
<h1 id="define-cloud-build-triggers">Define Cloud Build Triggers</h1>
<p>Here's the procedure I used to link a Cloud Build trigger to my Pelican image.</p>
<ol>
<li>Go to <a href="https://console.cloud.google.com/cloud-build">Google Cloud-Build</a>.</li>
<li>On the left side-bar, navigate to Triggers</li>
<li>Add Trigger</li>
<li>Select source: GitHub</li>
<li>Authenticate: Log in if needed</li>
<li>Select Repository: This is the repo that will be watched for changes</li>
<li>Name: This is the name of the trigger</li>
<li>Description: Something like <code>Push to master branch</code></li>
<li>Trigger Type: <code>Branch</code></li>
<li>Branch: <code>master</code></li>
<li>Build Configuration: <code>cloudbuild.yaml</code></li>
<li><strong>Create Trigger</strong></li>
</ol>
<p>With this trigger in place, all pushes to the pelican project will trigger the
rebuilding of its image, followed by updating the image in GCR with the latest
changes.</p>


<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#why-use-google-container-registry">Why use Google Container Registry</a></li>
<li><a href="#pricing">Pricing</a></li>
<li><a href="#command-line">Command Line</a></li>
<li><a href="#authentication">Authentication</a>
<ul>
<li><a href="#create-a-service-account">Create a service account</a></li>
<li><a href="#authenticate-your-local-docker-client">Authenticate your local Docker client</a></li>
</ul>
</li>
<li><a href="#using-the-google-container-registry">Using the Google Container Registry</a>
<ul>
<li><a href="#pull-image">Pull Image</a></li>
<li><a href="#push-image">Push Image</a>
<ul>
<li><a href="#access-denied">Access Denied</a></li>
</ul>
</li>
<li><a href="#delete-an-image">Delete an Image</a></li>
</ul>
</li>
</ul>
<h1 id="why-use-google-container-registry">Why use Google Container Registry</h1>
<p>The Google Container Registry is a SaaS version of the public <a href="https://hub.docker.com/_/registry">Registry</a>
docker image. It solves a number of problems that the public Docker image has.
Namely:</p>
<ul>
<li><strong>Authentication</strong>: GCR has built-in access controls, where the Registry image
does not. With the Docker Registry you need to put something like NGINX in
front of it to provide access control.</li>
<li><strong>HTTPS</strong>: Like with the authentication problem, Registry needs a reverse proxy
in front of it to serve a certificate. Also, you need to worry about
obtaining and maintaining the valid cert files. Without a valid cert, each
Docker client needs to list the registry as an insecure registry, which is
both insecure and inconvenient.</li>
<li><strong>Serverless</strong>: To host a traditional Docker registry, you need a public
server running Docker. With Google's Container Registry, there's no server to
maintain.</li>
<li><strong>Vulnerability Scanning</strong>: GCR can <a href="https://cloud.google.com/container-registry/docs/get-image-vulnerabilities">scan your images for security issues</a>.</li>
</ul>
<hr>
<h1 id="pricing">Pricing</h1>
<p>The pricing is well documented <a href="https://cloud.google.com/container-registry/pricing">here</a>,
and it has a lot of variables.</p>
<p>Basically though, it boils down to the following approximate costs:</p>
<ul>
<li><strong>Storage</strong>: $0.026 USD / GB / month</li>
<li><strong>Network</strong>: ~ $0.12 USD / GB</li>
<li><strong>Vulnerability Scanning</strong>: $0.26 per scanned image</li>
</ul>
<hr>
<h1 id="command-line">Command Line</h1>
<p>GCR can be managed from the web UI or the command line. I try and keep my steps
limited to the web UI, but the command line is really easy to use. You can find
its documentation <a href="https://cloud.google.com/sdk/gcloud/reference/container/images">here</a>.</p>
<hr>
<h1 id="authentication">Authentication</h1>
<p>Before you start, be sure to install:</p>
<ul>
<li><a href="https://cloud.google.com/sdk/">Google Cloud SDK</a></li>
<li><a href="https://docs.docker.com/install/">Docker</a></li>
</ul>
<p>To interact with Google Container Registry you need to first authenticate your
local Docker client. Usually this is done with the <code>docker login &#x3C;registry></code>
command, but that's not how it's done with Google container registry. Instead
we'll use the <code>gcloud</code> command line.</p>
<h2 id="create-a-service-account">Create a service account</h2>
<p>While this isn't strictly necessary, it's good practice.</p>
<p>The container registry uses GCS buckets to store the images. It doesn't have
dedicated roles, and instead uses the GCS roles. You can find a detailed list
of the roles <a href="https://cloud.google.com/container-registry/docs/access-control">here</a>.
The two important roles are <code>roles/storage.admin</code> for write and
<code>roles/storage.objectViewer</code> for read.</p>
<p>To create the SA from the Google Cloud Web UI:</p>
<ol>
<li>Open the <a href="https://console.cloud.google.com/iam-admin/serviceaccounts">IAM Service Accounts page</a></li>
<li>If needed, select or create your project</li>
<li>Click <code>Create Service Account</code></li>
<li>Create a member, such as <code>gcr-readonly</code> or <code>gcr-admin</code>.
When possible, its nice to have the account name and account ID match.</li>
<li>Assign a useful description to the service-account.</li>
<li>Assign a role. <code>Storage Object Viewer</code> grants read access and
<code>Storage Admin</code>.</li>
<li>Create and download a key. Name it something like <code>gcr-readonly.json</code>.</li>
</ol>
<h2 id="authenticate-your-local-docker-client">Authenticate your local Docker client</h2>
<p>Use the <code>gcloud</code> tool to deploy some <code>credHelpers</code> to <code>~/.docker/config.json</code>.</p>
<pre><code class="language-bash">gcloud auth configure-docker --project &#x3C;project name>
</code></pre>
<p>Next up, it's time to authenticate.</p>
<p>You could run <code>gcloud auth login</code>, but that doesn't use the service account we
just created and also requires a browser to open a generated link.</p>
<p>Instead, use the service account's JSON file generated in the above step with
the <code>--key-file</code> argument as follows:</p>
<pre><code class="language-bash">gcloud auth activate-service-account --key-file /vagrant/auth/gcr-readonly.json
</code></pre>
<p>If successful, you'll get a confirmation message:</p>
<pre><code class="language-bash">Activated service account credentials for: [gcr-readonly@*.gserviceaccount.com]
</code></pre>
<hr>
<h1 id="using-the-google-container-registry">Using the Google Container Registry</h1>
<p>The GCR works just like any other registry once you've ran the <code>gcloud auth</code>
command. You don't need to run any <code>docker login</code> command.</p>
<h2 id="pull-image">Pull Image</h2>
<p>Here's how to pull a Pelican image from GCR:</p>
<pre><code class="language-bash">docker pull gcr.io/myProject/pelican
</code></pre>
<h2 id="push-image">Push Image</h2>
<p>It's pretty straightforward. Replace <code>myProject</code> with your project name.</p>
<p>In this example I add a tag to my pelican image, calling it <code>pelican2</code>, and
push it up to the registry as a new image. You'd normally make your own Docker
image.</p>
<pre><code class="language-bash">sudo docker tag gcr.io/myProject/pelican gcr.io/myProject/pelican2
sudo docker push gcr.io/myProject/pelican2
</code></pre>
<p>If you didn't get an error, then the image is now pushed.</p>
<h3 id="access-denied">Access Denied</h3>
<p>If you receive the following message, your user account lacks write access:</p>
<pre><code class="language-text">denied: Token exchange failed for project 'myProject'. Caller does not have permission 'storage.buckets.get'.
</code></pre>
<p>To resolve this, create a service account  with the role <code>roles/storage.admin</code>.
If there's a finer grained access level to permit this, I'd be interested to
hear about it.</p>
<pre><code class="language-bash">gcloud auth activate-service-account --key-file /vagrant/auth/gcr-admin.json
sudo docker push gcr.io/kylepericak/pelican2
</code></pre>
<h2 id="delete-an-image">Delete an Image</h2>
<p>At the time of writing this, there's a warning in the cloud UI saying that you
can't delete images unless you use the CLI.</p>
<p>Here's how to do that. Be sure to replace <code>myProject</code> with your project.</p>
<pre><code class="language-bash"># List current images
gcloud config set project myProject

# Ignore this warning if you're using a service account:
# WARNING: You do not appear to have access to project [myProject] or it does not exist.

# Delete the image
cloud container images delete gcr.io/myProject/pelican2
</code></pre>


<p><strong>This post is linked to from the <a href="/blog-website.html">Blog Website Project</a></strong></p>
<hr>
<p>Nobody wants to test in production.</p>
<p>This guide covers how to preview your Pelican content locally as you write it.
Every time you save a markdown file or update the Pelican theme, the entire
site will re-generate and be hosted by a local web-server for you to review.</p>
<h1 id="install-dependencies">Install Dependencies</h1>
<ul>
<li><a href="https://docs.docker.com/install/">Docker</a></li>
<li><a href="https://git-scm.com/book/en/v2/Getting-Started-Installing-Git">Git</a></li>
<li>Text Editor: Your choice, I use <a href="https://github.com/vim/vim">Vim</a></li>
<li>Browser: Chrome has nice developer tools but anything works</li>
</ul>
<hr>
<h1 id="get-the-pelican-image">Get the Pelican Image</h1>
<p><strong>Option 1: Build your own</strong></p>
<p>This is the approach I suggest. You can see the guide I wrote explaining how to
build a Pelican image <a href="/docker-pelican-image.html">here</a>.</p>
<p><strong>Option 2: Use a pre-made image</strong>
I've shared my image for public use <a href="https://hub.docker.com/r/kpericak/pelican">here</a>.
If you plan on using it, you're best bet is to check how I use it on GitHub.</p>
<hr>
<h1 id="launch-a-persistent-pelican-container">Launch a Persistent Pelican Container</h1>
<p>Once you've either pulled or built the image, it will be listed in your Docker
image list.</p>
<pre><code class="language-bash">docker image list
</code></pre>
<p>Set the following variables. Note the paths should be absolute (vs relative).</p>
<ul>
<li><code>$image_name</code>: docker image name</li>
<li><code>$conf_file</code>: path to <code>pelicanconf.py</code> configuration file</li>
<li><code>$content_dir</code>: path to <code>content/</code> directory, where markdown files reside</li>
<li><code>$output_dir</code>: path to <code>output/</code> directory, where generated files will land.</li>
<li><code>$listen_port</code>: Which port to listen on for the dev web-server</li>
</ul>
<p>Optionally, you can set a theme path too. If you omit this and its associated
volume mount then the theme built into the Pelican image will be used instead.
This enables theme development.</p>
<ul>
<li><code>$theme_path</code>: Path to the Pelican theme</li>
</ul>
<p>Start the Docker container in daemon mode (<code>-d</code>)  as persistent container.
By setting the run command to <code>tail -f /dev/null</code>, the container will never
stop itself and instead sit idle until told to do otherwise.</p>
<p>Mount the directories specified by those variables into the container.</p>
<p>Forward <code>$listen_port</code> to the workstation so the site can be opened in a local
browser.</p>
<pre><code class="language-bash">image_name=""
conf_file=""
content_dir=""
output_dir=""
theme_path=""
listen_port=8000

docker run -d \
  --name pelican \
  -v $conf_file:/pelicanconf.py \
  -v $conftent_dir:/content \
  -v $output_dir:/output \
  -v $theme_path:/theme" \
  -p 0.0.0.0:$listen_port:8000 \
  image_name \
  tail -f /dev/null
</code></pre>
<hr>
<h1 id="launch-development-web-server">Launch Development Web-Server</h1>
<p>At this point you have a pelican container running, but it isn't doing
anything. The files are all mounted in, and it's prepared to forward its local
port 8000 to your specified workstation port, so all that's left is to start
Pelican's built-in development service.</p>
<p>This service will watch both the markdown files and the theme files for
changes, then reload the site files. It will also host the website on a
non-production web-server.</p>
<pre><code class="language-bash">docker exec -it pelican \
  pelican \
    -s /pelicanconf.py \
    --debug \
    --autoreload \
    --listen \
    --port 8000 \
    --bind 0.0.0.0 \
    /content \
    -o /output
</code></pre>


<p><strong>This post is linked to from the <a href="/blog-website.html">Blog Website Project</a></strong></p>
<hr>
<p>Pelican is a free and open source static website generator written in Python.
It accepts a theme, a config file, and a directory of markdown documents as
input and uses them to generate the HTML and CSS files that make up a static
website.</p>
<p>This post covers how to build a Docker image that contains the Pelican software
and the theme which I chose to use with it. This image is reusable and
<a href="https://hub.docker.com/r/kpericak/pelican">freely available on Docker Hub</a>.</p>
<h1 id="dockerfile">Dockerfile</h1>
<p>As with any Docker image, the most important piece is the Dockerfile.</p>
<p>You can see the current file I'm using <a href="https://github.com/kylep/pelican/blob/master/Dockerfile">here</a>.</p>
<p>In this Dockerfile, Pelican, its theme, and plugins are installed to the image.</p>
<pre><code class="language-text">FROM ubuntu:bionic
ADD requirements.txt ./
RUN apt-get update \
    &#x26;&#x26; apt-get install -y --no-install-recommends \
         git \
         python \
         python-pip \
         python-setuptools \
    &#x26;&#x26; rm -rf /var/lib/apt/lists/* \
    &#x26;&#x26; pip install -r requirements.txt \
    &#x26;&#x26; git clone https://github.com/kylep/voidy-bootstrap.git /theme \
    &#x26;&#x26; mkdir -p /tmp-plugins/ /plugins \
    &#x26;&#x26; git clone --recurse \
           https://github.com/getpelican/pelican-plugins.git /tmp-plugins \
    &#x26;&#x26; cp -r /tmp-plugins/series /plugins/ \
    &#x26;&#x26; rm -r /tmp-plugins
</code></pre>
<h3 id="from-ubuntubionic">FROM ubuntu:bionic</h3>
<p>This section specifies the Docker base image. The above docker file uses
<a href="https://hub.docker.com/_/ubuntu/">the official Docker Hub Ubuntu image</a>.</p>
<h3 id="add-requirementstxt">ADD requirements.txt</h3>
<p>The requirements.txt file lists the python packages which Pelican depends upon.
Docker's <code>ADD</code> will copy that file into the image at the specified path.</p>
<p>The minimum packages required are:</p>
<pre><code class="language-text">pelican
markdown
</code></pre>
<h3 id="run-">RUN ...</h3>
<p>The run task is basically a bash script all squished into one line. While its
harder to read with all the line continuation and <code>&#x26;&#x26;</code> symbols, using them
leads to Docker only building a single layer.</p>
<p><a href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/">See the best practices for writing docker images</a>.</p>
<p>Since this is an Ubuntu image, <code>apt</code> is the package manager. It installs Git
and some python packages. Git is required to pull the Pelican theme from
Github.</p>
<p>Only the minimum Python packages are installed by apt, since pip is the
preferred python package manager.</p>
<p><code>rm -rf /var/lib/apt/lists/*</code> removes the leftover apt artifacts, helping to
shrink the Docker image.</p>
<p>After the theme is pulled from GitHub, Pelican's open source plugins are also
pulled. The ones desired are copied to <code>/plugins</code>, then the (huge) checkout is
deleted to reduce image size.</p>
<hr>
<h1 id="running-the-docker-image">Running the Docker Image</h1>
<p>Locally, you need to mount the content into the container. Something like this
will work:</p>
<pre><code class="language-bash">docker run \
  --name pelican \
  -v $(pwd):/workspace \
  gcr.io/kylepericak/pelican:latest \
  pelican -o /workspace/output -s /workspace/pelicanconf.py
</code></pre>
<p>The above command will load the <code>content/</code> and <code>pelicanconf.py</code> files into
Pelican, and write the static website to <code>output/</code>.</p>


Kyle Pericak

"It works in my environment"

Categories

Tags