<p>The <a href="/bot-wiki.html">bot-wiki post</a> introduced a structured wiki
designed for machine consumption. The whole point was to build
something a RAG pipeline could actually use. This is that pipeline.</p>
<h2 id="the-stack">The stack</h2>
<p>The wiki has 21 markdown pages with structured frontmatter:
title, summary, keywords, scope. The RAG pipeline is a single
Python script that:</p>
<ol>
<li>Embeds all 21 pages into vectors using OpenRouter's API</li>
<li>Stores them in a FAISS flat index (129 KB, committed to git)</li>
<li>At query time, embeds the question, finds the nearest pages,
and feeds them to an LLM for a grounded answer</li>
</ol>
<p>Dependencies: <code>faiss-cpu</code>, <code>openai</code> (for the OpenAI-compatible
API), and <code>pyyaml</code>. That's it.</p>
<h2 id="why-faiss">Why FAISS</h2>
<p><a href="https://github.com/facebookresearch/faiss">FAISS</a> is Meta's
library for similarity search over dense vectors. It runs
in-process, no server needed. You give it vectors, it gives
you nearest neighbors. The index is a flat file you can
commit to git.</p>
<p>I picked it because this is 21 vectors. There's no need for
a database, a server, or a managed service. <code>pip install faiss-cpu</code>, build an index, save it, done. The index is
129 KB. It loads in milliseconds.</p>
<p>If you're working at larger scale or want more features,
the main alternatives are:</p>
<ul>
<li><a href="https://github.com/chroma-core/chroma">Chroma</a>: embedded
vector database with a nice Python API. Good if you want
metadata filtering or persistence without managing files
yourself. Still runs locally.</li>
<li><a href="https://github.com/qdrant/qdrant">Qdrant</a>: vector search
engine written in Rust. Runs as a server, supports
filtering and payloads. Better fit when you have millions
of vectors or need a proper API.</li>
<li><a href="https://www.pinecone.io/">Pinecone</a>: fully managed cloud
vector database. Zero ops, but you're paying per query and
your data lives on someone else's servers.</li>
</ul>
<p>If you're already running PostgreSQL,
<a href="https://github.com/pgvector/pgvector">pgvector</a> is worth
a look. It adds vector columns and similarity search to
your existing database. No new service to run, and your
vectors live next to your relational data.</p>
<p>For 21 pages committed to a git repo, FAISS is the obvious
choice. No dependencies beyond the pip package, no running
services, no accounts.</p>
<h2 id="why-whole-page-chunks">Why whole-page chunks</h2>
<p>The wiki pages are short, 18 to 65 lines each. Splitting them
by section would lose context without gaining anything. The
frontmatter was designed to enrich embeddings, so each chunk
looks like this:</p>
<pre><code>Title: Claude Code
Summary: Anthropic's CLI-based AI coding assistant. Runs
  in terminal, edits files directly, supports MCP tool
  integrations and custom hooks.
Keywords: claude-code, anthropic, cli, ai-coding-assistant,
  mcp, hooks
Scope: Covers Claude Code setup, configuration, MCP
  integration, hooks, and usage patterns.

Claude Code is Anthropic's CLI tool for AI-assisted software
engineering. It operates directly in the terminal...
</code></pre>
<p>Prepending the summary, keywords, and scope to the body gives
the embedding model more signal about what the page covers.
21 chunks, 5,434 tokens total.</p>
<h3 id="chunking-in-general">Chunking in general</h3>
<p>Most RAG guides recommend chunks of 256 to 512 tokens. That's
a reasonable default for long documents where you want to
retrieve a specific paragraph, not the whole thing. Smaller
chunks give more precise retrieval. Larger chunks give the
LLM more context per result.</p>
<p>The tradeoffs:</p>
<ul>
<li><strong>Too small</strong> (under 100 tokens): chunks lose meaning on
their own. "See above" or "as mentioned" with no referent.
Retrieval finds fragments, not answers.</li>
<li><strong>Too large</strong> (over 1000 tokens): embeddings become diluted.
A chunk about five topics matches all five weakly instead
of one strongly. You also burn more of the LLM's context
window per retrieved result.</li>
<li><strong>Overlap</strong> helps when chunks split mid-thought. A common
pattern is 512-token chunks with 50-100 tokens of overlap.
This adds redundancy but reduces the chance of cutting a
relevant passage in half.</li>
</ul>
<p>The wiki pages average about 260 tokens each, which lands
right in the sweet spot. They're also self-contained by
design, each page covers one topic with clear boundaries.
No reason to split them further.</p>
<p>If I were indexing longer documents (blog posts, docs, book
chapters), I'd split by section headers first, then by token
count within sections. Semantic boundaries beat arbitrary
ones.</p>
<h2 id="building-the-index">Building the index</h2>
<pre><code class="language-bash">python3 -m venv .venv
.venv/bin/pip install -r bin/requirements-rag.txt
.venv/bin/python bin/wiki-rag.py build
</code></pre>
<pre><code>Embedding 21 wiki pages...
Indexed 21 pages, 1536 dimensions, 5434 tokens
Index: 129,069 bytes, metadata: 29,663 bytes
Saved to blog/markdown/wiki/.index
</code></pre>
<p>The embedding model is <code>text-embedding-3-small</code> via OpenRouter.
1536 dimensions per vector. The FAISS index uses <code>IndexFlatIP</code>
(inner product on normalized vectors = cosine similarity). No
approximate search needed at this scale.</p>
<p>The index and a <code>metadata.json</code> file (slug, title, summary,
full chunk text for each page) get saved to
<code>blog/markdown/wiki/.index/</code>. Both files are committed to git.
Any tool that wants to query the wiki can just read the index
directly without needing the embedding API.</p>
<p>The build is a manual step, not part of <code>npm run build</code>. Wiki
content doesn't change often, and there's no reason to pay for
embeddings on every deploy.</p>
<h2 id="querying">Querying</h2>
<p>A query hits two models. First, the question gets embedded
into a vector using the same embedding model that indexed
the wiki pages. FAISS compares that vector against the 21
stored vectors and returns the closest matches. Those
matches are wiki pages that are semantically similar to the
question, not keyword matches, but meaning matches.</p>
<p>Then the retrieved pages get passed to an LLM (Claude
Sonnet) as context, along with the original question. The
LLM reads the pages and writes an answer grounded in what
they actually say. Without the retrieval step, the LLM
would just guess from its training data. With it, the LLM
answers from your specific documents.</p>
<p>That's the "retrieval-augmented" part of RAG: find relevant
documents first, then generate an answer from them.</p>
<pre class="mermaid">
graph LR
    Q[Question] --> E[Embed question]
    E --> F[FAISS search]
    F --> R[Top 3 wiki pages]
    R --> P[Build prompt]
    P --> LLM[Claude Sonnet]
    LLM --> A[Answer + sources]
</pre>
<pre><code class="language-bash">.venv/bin/python bin/wiki-rag.py query \
  "What MCP servers does this project use?"
</code></pre>
<pre><code>Question: What MCP servers does this project use?

Top 3 matches:
  1. [0.644] MCP Integrations (wiki/mcp)
  2. [0.563] Linear MCP (wiki/mcp/linear)
  3. [0.524] GA4 MCP (wiki/mcp/ga4)

Answer:
Based on the provided context, this project uses three MCP
servers:

1. Playwright, for browser automation
2. Linear, for project management
3. GA4, for accessing website traffic data and reports

All MCP servers run as subprocesses communicating via JSON-RPC
over stdio, and are configured in ~/.claude.json or a
project-level .claude.json file.
</code></pre>
<p>The retrieval nailed the right pages. Here's a different query:</p>
<pre><code class="language-bash">.venv/bin/python bin/wiki-rag.py query "How do I run security scans?"
</code></pre>
<pre><code>Question: How do I run security scans?

Top 3 matches:
  1. [0.484] Security Toolkit (wiki/devops/security-toolkit)
  2. [0.439] DevOps &#x26; Security (wiki/devops)
  3. [0.343] OpenClaw Kubernetes Security (wiki/openclaw/kubernetes)
</code></pre>
<p>The answer included the actual docker commands from the security
toolkit wiki page, copy-paste ready. The third result
(Kubernetes security) is a reasonable stretch, the word
"security" overlaps, but the top two are exactly right.</p>
<h2 id="the-script">The script</h2>
<p>The interesting parts are small. Embedding is a single API call
since all 21 pages fit in one batch:</p>
<pre><code class="language-python">client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)
response = client.embeddings.create(
    model="openai/text-embedding-3-small",
    input=texts,
)
</code></pre>
<p>OpenRouter's embedding endpoint is OpenAI-compatible, so the
<code>openai</code> Python package works with just a base URL swap.</p>
<p>FAISS search returns similarity scores and indices:</p>
<pre><code class="language-python">scores, indices = index.search(query_vec, k)
for score, idx in zip(scores[0], indices[0]):
    page = metadata[idx]
    # score is cosine similarity (0 to 1)
</code></pre>
<p>The retrieved pages get stuffed into a prompt with instructions
to only use the provided context and cite sources. The LLM
(Claude Sonnet via OpenRouter) generates the answer.</p>
<h2 id="cost">Cost</h2>
<p>Indexing 21 pages used 5,434 tokens at
<code>text-embedding-3-small</code> rates ($0.02/1M tokens). That's
$0.0001. Each query embeds a short question (maybe 20 tokens)
plus one chat completion. A query costs roughly $0.002 to
$0.01 depending on the response length.</p>
<p>Building the index and running test queries cost $0.03
total on OpenRouter. That covers the embedding API calls
and the chat completions for query-mode answers. The
development itself (writing the script, iterating on the
blog post) ran through Claude Code on an Anthropic Max
subscription, so that cost isn't on the OpenRouter bill.</p>
<p>The wiki is also a write target, not just a read one. I have
a journalist agent that runs on a K8s CronJob every morning,
searches for AI news, and commits a digest to the wiki automatically.
<a href="/cron-event-triggered-ai-agents-k8s.html">Cron/event-triggered AI agents on K8s</a>
covers how that side works.</p>


<h1 id="problem-description">Problem Description</h1>
<p>You need to transfer files to a locked down Windows server from a Linux server.
Sure, you could carve out a file-share. There's an easier way for a one-off
transfer.</p>
<p>This is a bit of an edge-case since newer Windows systems come with SCP
installed, but the image I needed to transfer to had it disabled. Older
versions like Server 2012 also don't support SCP without extra software.</p>
<h1 id="hosting-files-with-pythons-web-server">Hosting files with Python's web server</h1>
<p>Make a directory and move your file into it. In my case, I'm transferring the
CloudBase-Init setup package.</p>
<p><strong>Note</strong>: If you use the package itself inside either a python script or
an interactive Python session you can change things like the listen IP and
port. Using <code>python -m</code> is just easier.</p>
<pre><code class="language-bash"># Move the file to a directory where it's the only thing there
mv CloudbaseInitSetup_1_1_0_x64.msi http/

# Change to that directory
cd http

# Start the web server. It will listen on 0.0.0.0:8000 by default.
python -m SimpleHTTPServer
</code></pre>
<h1 id="downloading-the-file">Downloading the file</h1>
<p>Now the file can be downloaded from your Linux server. In your Windows system,
open Internet Explorer and navigate to the Linux IP address. Don't forget to
add the <code>:8000</code> to specify the port. From here you can easily download the
file. Be sure to close the server (<code>CTRL+c</code>) when you're done.</p>


<p>This post covers some of the ways you can use Ceph from Python</p>
<p>The two libraries used here:</p>
<ul>
<li><a href="https://docs.ceph.com/docs/master/rados/api/python/">rados</a></li>
<li><a href="https://docs.ceph.com/docs/giant/rbd/librbdpy/">rbd</a></li>
</ul>
<hr>
<h1 id="setup">Setup</h1>
<h2 id="deploy-cephconf">Deploy ceph.conf</h2>
<p>Grab the ceph.conf file from your cluster or make a new valid one and place
it on the filesystem of the server which will execute the Python code.</p>
<p>Also copy the keyring over. Check <a href="https://docs.ceph.com/docs/master/rados/api/python/">here</a>
if you want to use anything other than the admin keyring.</p>
<h2 id="install-dependencies">Install Dependencies</h2>
<p>Ceph doesn't distribute a pure python library. Instead it seems to rely on some
C stuff that needs to be installed, so you need to use apt instead of pip.
This is a pain, since so far as I can tell you can't isolate these dependencies
inside a virtual environment.</p>
<pre><code class="language-bash">apt-get install python-rados
</code></pre>
<hr>
<h1 id="examples">Examples</h1>
<p>Be sure that the rados and rbd libraries, and <code>ceph.conf</code> are deployed first.</p>
<h2 id="instantiate-rados-and-rbd-instances">Instantiate rados and rbd instances</h2>
<p>The <code>cluster</code> and <code>rbd_inst</code> variables will be used through these examples.
The <code>rbd_inst</code> methods will expect <code>ioctx</code> as their first argument.</p>
<pre><code class="language-python">import rbd
import rados

cluster = rados.Rados(conffile='/etc/ceph/ceph.conf')
cluster.connect()

pool_name = 'volumes'
ioctx = cluster.open_ioctx(pool_name)
rbd_inst = rbd.RBD()
</code></pre>
<h1 id="usage-examples">Usage examples</h1>
<pre><code class="language-python"># Get cluster stats
# Returns dict with kb, num_objects, kb_avail, and kb_used
cluster.get_cluster_stats()

# Get cluster health by issuing a command to ceph's monitor
cmd = {"prefix":"status", "format":"json"}
## mon_command returns a tuple of (return code, data/buffer, errors)
command_result = cluster.mon_command(json.dumps(cmd), b'', timeout=5)
status = json.loads(command_result[1])

# Get Pool usage stats
# Returns num_objects_degraded, num_kb, num_wr, num_rd, num_objects_unfound,
#   num_objects_missing_on_primary, num_object_clones, num_objects,
#   num_object_copies, num_bytes, num_rd_kb, num_wr_kb
ioctx.get_stats()

# List pools
cluster.list_pools()

# Create pool
pool_name = 'test_pool'
cluster.create_pool(pool_name)

# Delete pool
cluster.delete_pool(pool_name)


# List rados block devices
rbd_inst.list(ioctx)

# Create block device
rbd_name = 'sample-device'
size_gb = 10
rbd_inst.create(ioctx, rbd_name, size_gb)

# Query a block device (image)
image = rbd.Image(ioctx, rbd_name)

# Resize image
new_size=20
image.resize(new_size)

# Create a snapshot
snap_name = 'sample-snap'
image.create_snap(snap_name)

# List snaps of image
image.list_snaps()

# Revert to snapshot
snap_name = 'sample-snap'
image.rollback_to_snap(snap_name)

# Delete a snapshot
image.remove_snap(snap_name)

# Close an open image (when you're done working with it)
image.close()

# Delete an image/volume. Must not have snapshots. Must be closed.
rbd_inst.remove(ioctx, rbd_name)
</code></pre>


<p>This post is a follow-up to my <a href="/aws-api-gateway-iam.html">authenticated API Gateway post</a>.</p>
<p>I've written a Lambda function and presented it using API Gateway. The API
uses Amazon's AIM keys for authentication. The next step is to write the code
to source control (AWS CodeBuild Git) and have it automatically deploy.</p>
<p><strong>Note</strong>: <a href="https://docs.aws.amazon.com/lambda/latest/dg/build-pipeline.html">This official guide on AWS</a>
is really good, and I copied big parts it. I use Python instead of
JavaScript in this post, and include my own experience while following the
guide.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#create-an-s3-bucket">Create an S3 Bucket</a></li>
<li><a href="#commit-your-function-to-source-control">Commit your function to source control</a></li>
<li><a href="#define-an-aws-sam-template">Define an AWS SAM template</a>
<ul>
<li><a href="#write-the-template-file">Write the template file</a></li>
<li><a href="#test-the-template">Test the template</a></li>
</ul>
</li>
<li><a href="#create-a-codebuild-project">Create a CodeBuild Project</a>
<ul>
<li><a href="#write-the-buildspecyml">Write the buildspec.yml</a></li>
<li><a href="#create-a-role-for-codebuild">Create a role for CodeBuild</a></li>
<li><a href="#define-the-build-project">Define the build project</a></li>
</ul>
</li>
<li><a href="#create-a-pipeline">Create a Pipeline</a>
<ul>
<li><a href="#create-iam-policy-for-codedeploy">Create IAM Policy for CodeDeploy</a></li>
<li><a href="#create-iam-role-for-codedeploy">Create IAM Role for CodeDeploy</a></li>
<li><a href="#create-the-pipeline">Create the Pipeline</a></li>
</ul>
</li>
</ul>
<h1 id="create-an-s3-bucket">Create an S3 Bucket</h1>
<p>Open the <a href="https://s3.console.aws.amazon.com/s3/">S3 Console</a> and create a
bucket to store your build artifacts.</p>
<hr>
<hr>
<h1 id="commit-your-function-to-source-control">Commit your function to source control</h1>
<p>I'm using a dedicated repository on AWS CodeCommit to store the code, and
Python3 for the language. Here's a very simple Lambda function I use for
testing:</p>
<pre><code class="language-python">def lambda_handler(event, context):
    """ run a hello function """
    body = 'NO HTTP METHOD'
    if 'httpMethod' in event and event['httpMethod'] == 'POST':
        request_body = event['body']
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a POST with body: {request_body}'
    elif 'httpMethod' in event and event['httpMethod'] == 'GET':
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a GET'
    return {
        'statusCode': 200,
        'body': json.dumps(body)
    }
</code></pre>
<hr>
<h1 id="define-an-aws-sam-template">Define an AWS SAM template</h1>
<h2 id="write-the-template-file">Write the template file</h2>
<p>This was my first time using AWS SAM, so consider these my observations and not
necessarily fact.</p>
<p>SAM stands for Serverless Application Model.</p>
<ul>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html">SAM Reference Page</a></li>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-api.html">API Event type</a></li>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html">What is SAM</a></li>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-specification.html">SAM Specification</a></li>
</ul>
<p>The fields I've changed from Amazon's example for this post:</p>
<ul>
<li><strong>Description</strong> - string describing the serverless application. This
description will be attached to the CloudFormation stack.</li>
<li><strong>Resources.[resourceName]</strong> - self-defined resource, my function.
The resource is a Lambda function because it's type is
<code>AWS::Serverless::Function</code></li>
<li><strong>Handler</strong> - Which method will run in the function</li>
<li><strong>Runtime</strong> - The language to be used</li>
<li><strong>CodeUri</strong> - Path to my file defining the function</li>
<li><strong>Resource.[resourceName].Events[eventName]</strong> - Name of the event which
triggers the function.</li>
<li><strong>Resource.[resourceName].Events[eventName].path</strong> - path for the URL</li>
</ul>
<pre><code class="language-yaml"># template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Sample Continuously deploy function
Resources:
  TestCicdFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: cicd-test.lambda_handler
      Runtime: python3.8
      CodeUri: ./cicd-test.py
      Events:
        TestCicdAPI:
          Type: Api
          Properties:
            Path: /Test
            Method: GET
</code></pre>
<h2 id="test-the-template">Test the template</h2>
<p>Install the AWS SAM CLI application.</p>
<pre><code class="language-bash">pip install --user aws-sam-cli
</code></pre>
<p>Build the package and upload it to your S3 bucket. From your project
root, set your bucket name then run the following.</p>
<pre><code class="language-bash"># Set the bucket name first
BUCKET=''

aws cloudformation package \
  --template-file template.yml \
  --s3-bucket $BUCKET \
  --output-template-file outputtemplate.yml
</code></pre>
<p>The package command will upload your python file to S3 and output a new
template file with the S3 path replacing the <code>CodeUri</code>.</p>
<pre><code class="language-text">Uploading to &#x3C;ID>  381 / 381.0  (100.00%)
Successfully packaged artifacts and wrote output template to file outputtemplate.yml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /home/vagrant/arcus-lambda/outputtemplate.yml --stack-name &#x3C;YOUR STACK NAME>
</code></pre>
<p>Now deploy it. Since there's no CloudFormation stack created yet, I make a new
one named <code>test-cicd-lambda</code>. Note that in the above command output, the
example they give for deploy is wrong. You need to include the
<code>--capabilities CAPABILITY_IAM</code> argument for this template too.</p>
<pre><code class="language-bash">aws cloudformation deploy \
  --capabilities CAPABILITY_IAM \
  --template-file /home/vagrant/arcus-lambda/outputtemplate.yml \
  --stack-name test-cicd-lambda
</code></pre>
<p>The application is now deployed.</p>
<ul>
<li>Stack is visible in the  <a href="https://ca-central-1.console.aws.amazon.com/cloudformation">CloudFormation console</a></li>
<li>API is created in the <a href="https://ca-central-1.console.aws.amazon.com/apigateway/">API Gateway console</a></li>
<li>Function is created in the <a href="https://ca-central-1.console.aws.amazon.com/lambda/">Lambda console</a>.
The function also has an Application page in the web console now.</li>
</ul>
<hr>
<h1 id="create-a-codebuild-project">Create a CodeBuild Project</h1>
<p>I recently did a whole post on this <a href="/aws-codepipeline-ecr.html">here</a>, so I'm
going to be light on the details. Head over to that post to fill in any blanks.
In that post I was building Docker images and pushing them to ECR. This is
similar, except we push python files to Lambda instead.</p>
<h2 id="write-the-buildspecyml">Write the buildspec.yml</h2>
<p>Put this in the project root, it defines what CodeBuild will do. We'll use an
environment variable for the <code>BUCKET</code> definition.</p>
<p>If you're not sure what to pick for a runtime, check <a href="https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-available.html">this list</a>.</p>
<pre><code class="language-yaml"># buildspec.yml
version: 0.2
phases:
  install:
    runtime-versions:
        python: 3.8
  build:
    commands:
      - aws cloudformation package --template-file template.yml --s3-bucket $BUCKET --output-template-file outputtemplate.yml
artifacts:
  type: zip
  files:
    - template.yml
    - outputtemplate.yml
</code></pre>
<p>Commit the file to source control.</p>
<h2 id="create-a-role-for-codebuild">Create a role for CodeBuild</h2>
<p>I use the role I defined for CodeBuild in my previous post. It's a bit overly
permissive but it works. Other than the policies which get created
automatically, mine has these:</p>
<ul>
<li><code>AWSCodeCommitFullAccess</code></li>
<li><code>AmazonEC2ContainerRegistryFullAccess</code></li>
<li><code>AmazonS3FullAccess</code></li>
<li><code>CloudWatchLogsFullAccess</code></li>
<li><code>AWSCodeBuildAdminAccess</code></li>
</ul>
<h2 id="define-the-build-project">Define the build project</h2>
<ol>
<li>Open the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codebuild/">CodeBuild console</a></li>
<li>Create build project</li>
<li>Name the build and give it a description</li>
<li>Source: Add your repository</li>
<li>Environment
<ol>
<li>Managed Image</li>
<li>Operating system: Ubuntu</li>
<li>Runtimes: Standard</li>
<li>Image: the newest one</li>
<li>Environment type: Linux</li>
<li>Privileged: False
<ol>
<li>Service Role: Existing role - choose the one you made above</li>
</ol>
</li>
<li>Additional Configuration - environment variables
<ul>
<li>name: BUCKET</li>
<li>value: Your bucket name</li>
</ul>
</li>
</ol>
</li>
<li>Buildspec: use buildspec file</li>
<li>Artifacts: no artifacts</li>
<li>Logs: set a group name for CloudWatch</li>
<li>Create build project</li>
</ol>
<p>Test the build before moving on and make sure it works.</p>
<hr>
<h1 id="create-a-pipeline">Create a Pipeline</h1>
<h2 id="create-iam-policy-for-codedeploy">Create IAM Policy for CodeDeploy</h2>
<p>Open the IAM console and create a new policy.
Define the policy using the following JSON.
These wildcards will cover everything you need, though it might be a bit
permissive.</p>
<pre><code class="language-json">{
    "Statement": [
        {
            "Action": [
                "apigateway:*",
                "codedeploy:*",
                "lambda:*",
                "cloudformation:CreateChangeSet",
                "iam:GetRole",
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:PutRolePolicy",
                "iam:AttachRolePolicy",
                "iam:DeleteRolePolicy",
                "iam:DetachRolePolicy",
                "iam:PassRole",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetBucketVersioning"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ],
    "Version": "2012-10-17"
}
</code></pre>
<h2 id="create-iam-role-for-codedeploy">Create IAM Role for CodeDeploy</h2>
<p>In the role wizard of the IAM page:</p>
<ol>
<li>Create role</li>
<li>Trusted Entity: <code>CloudFormation</code></li>
<li>Permissions:
<ol>
<li><code>AWSLambdaExecute</code></li>
<li>Your new policy created above</li>
</ol>
</li>
<li>Role Name: assign a name</li>
</ol>
<h2 id="create-the-pipeline">Create the Pipeline</h2>
<p>When code is committed, you want CodeBuild to run and re-deploy your serverless
application.</p>
<ol>
<li>Open the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codepipeline/pipelines">CodePipeline console</a></li>
<li>Click Create Pipeline</li>
<li>Name the pipeline</li>
<li>Service Role: Create a new service role for the pipeline. AWS will set sane
permissions for it, no special role is needed. I like to have a consistent
naming convention as you end up with a ton of IAM entities after a while.</li>
<li>Role Name: Pick a name</li>
<li>Allow AWS CodePipeline to create a role</li>
<li>Next</li>
<li>Source Provider: AWS CodeCommit</li>
<li>Repository Name: Select the repository name</li>
<li>Branch name: master</li>
<li>Change detection: Amazon CloudWatch Events</li>
<li>Next</li>
<li>Build Provider: AWS CodeBuild</li>
<li>Project Name: Choose the project you just made</li>
<li>Next</li>
<li>Deploy Provider: AWS CloudFormation</li>
<li>Action Mode: Create or replace a change set</li>
<li>Stack Name: choose a name. I used the one from the test above.</li>
<li>Change Set Name: choose a name</li>
<li>Template:
<ul>
<li>Artifact name: <code>BuildArtifact</code></li>
<li>File name: <code>outputtemplate.yml</code></li>
</ul>
</li>
<li>Capabilities: <code>CAPABILITY_IAM</code></li>
<li>Role Name: The role you defined for CodeDeploy above</li>
<li>Next</li>
<li>Create Pipeline</li>
</ol>
<p>That should do it. Now whenever you push code to your master branch, the
serverless application will re-deploy.</p>


<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#build-an-api-gateway-resource">Build an API Gateway resource</a>
<ul>
<li><a href="#build-a-lambda-function-for-it-to-run">Build a Lambda function for it to run</a></li>
<li><a href="#define-the-api">Define the API</a></li>
</ul>
</li>
<li><a href="#iam-setup">IAM Setup</a>
<ul>
<li><a href="#create-policy">Create Policy</a></li>
<li><a href="#bind-policy-to-a-user">Bind policy to a user</a></li>
</ul>
</li>
<li><a href="#test-the-api-in-postman">Test the API in Postman</a></li>
<li><a href="#execute-the-api-from-python">Execute the API from Python</a></li>
</ul>
<h1 id="build-an-api-gateway-resource">Build an API Gateway resource</h1>
<h2 id="build-a-lambda-function-for-it-to-run">Build a Lambda function for it to run</h2>
<p>Open the lambda console and create a function. Here's an example python
function that says hello to whoever sent the authenticated request.</p>
<pre><code class="language-python">def lambda_handler(event, context):
    """ run a hello function """
    body = 'NO HTTP METHOD'
    if 'httpMethod' in event and event['httpMethod'] == 'POST':
        request_body = event['body']
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a POST with body: {request_body}'
    elif 'httpMethod' in event and event['httpMethod'] == 'GET':
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a GET'
    return {
        'statusCode': 200,
        'body': json.dumps(body)
    }
</code></pre>
<h2 id="define-the-api">Define the API</h2>
<ol>
<li>Go to the <a href="https://ca-central-1.console.aws.amazon.com/apigateway/">API Gateway Console</a></li>
<li>Create</li>
<li>Protocol: REST</li>
<li>New API</li>
<li>API Name: <code>&#x3C;name></code></li>
<li>Endpoint type: Regional</li>
<li>Create API</li>
</ol>
<p>Create a method:</p>
<ol>
<li>Actions</li>
<li>Create Method</li>
<li>POST</li>
<li>Check mark</li>
<li>Integration Type: Lambda Function</li>
<li>Use Lambda Proxy Integration: Checked</li>
<li>Lambda Region: ca-central-1</li>
<li>Lambda Function: HelloWorld</li>
</ol>
<p>Enable IAM Authorization</p>
<ol>
<li>Click Method Request</li>
<li>Set authorization to <code>AWS_IAM</code></li>
</ol>
<h1 id="iam-setup">IAM Setup</h1>
<h2 id="create-policy">Create Policy</h2>
<p>Collect the ARN of the API Gateway. Click your endpoint, then the method.
The ARN will display under Method Request in the Method Execution diagram.</p>
<p>Navigate to the IAM console and create a new policy.</p>
<pre><code class="language-json">{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "execute-api:Invoke"
            ],
            "Resource": [
								"arn:aws:execute-api:ca-central-1:850047500507:42mydti3lg/*/POST/*"
            ]
        }
    ]
}
</code></pre>
<h2 id="bind-policy-to-a-user">Bind policy to a user</h2>
<p>Here's how I do it:</p>
<ol>
<li>Open the IAM console</li>
<li>If you don't have a group yet, make one</li>
<li>Attach the policy to the group</li>
<li>Attach the group to a user</li>
</ol>
<p>Collect the key and key ID of that user.</p>
<h1 id="test-the-api-in-postman">Test the API in Postman</h1>
<p>Postman is nice since you know it works, and it natively handles building the
AWS request headers. It's the easiest way I know to run a one-off test of your
API.</p>
<p>Install <a href="https://www.getpostman.com/">Postman</a>.</p>
<ol>
<li>New > Request - fill in some names.</li>
<li>Set the request type to POST</li>
<li>Enter the URL</li>
<li>Enter any request data</li>
<li>Go to the Authorization tab and choose AWS signature</li>
<li>Enter the key and key ID from your IAM user</li>
<li>Set the region, in my case ca-central-1</li>
<li>Send</li>
</ol>
<p>You should get a valid response from your API. If you skip any of the
authorization steps you'll get an error instead.</p>
<h1 id="execute-the-api-from-python">Execute the API from Python</h1>
<p>End users don't use postman. Usually they'd be using JavaScript, and in my case
they use Python to access my API. This was <strong>really complicated</strong>.</p>
<p><a href="https://github.com/DavidMuller/aws-requests-auth">Here's an open source library that can do it</a>.</p>
<p>I've written my own version of this for the Breqwatr deployment tool. I didn't
want to depend on the above link. Also I tried to rewrite the client as
functions with the hope of making it easier to read.</p>
<p>You can find it in <a href="https://github.com/breqwatr/breqwatr-deployment-tool">Breqwatr's GitHub</a>
under <code>lib/aws</code>.</p>
<p>Here's the high level flow of how it gets the auth header:</p>
<pre><code class="language-python">def get_authorization_header(time, key_id, secret_key, region, method, host,
                             body, uri, query):
    """ Return an aws authorization header """
    # canonical request is a multi-line string with a particular format
    canonical_headers = get_canonical_headers(host, time)
    payload_hash = get_payload_hash(body)
    canonical_request = get_canonical_request(
         method=method,
         uri=uri,
         query=query,
         canonical_headers=canonical_headers,
         payload_hash=payload_hash)
    # create a string-to-sign from the canonical requests' digest
    cr_digest = hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()
    credential_scope = get_credential_scope(time, region)
    string_to_sign = get_string_to_sign(time, credential_scope, cr_digest)
    # sign the String-To-Sign with a signing key derived from secret iam key
    signing_key = get_signing_key(secret_key, time, region)
    signature = get_signature(signing_key, string_to_sign)
    # return the headers all in one string
    return (
        f'AWS4-HMAC-SHA256 Credential={key_id}/{credential_scope}, '
        f'SignedHeaders=host;x-amz-date, '
        f'Signature={signature}')
</code></pre>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<h1 id="create--terminate-an-ec2-instance-from-python">Create &#x26; Terminate an EC2 Instance from Python</h1>
<p>The IAM user tested for this task has the <code>AmazonEC2FullAccess</code> policy.</p>
<h2 id="create">Create</h2>
<p>Before you begin, determine the following:</p>
<ul>
<li>Image AMI: The example I provide here works, it's for Ubuntu 18.04 in Canada</li>
<li>Security group name</li>
<li>Instance name</li>
<li>Instance type</li>
</ul>
<pre><code class="language-python">import boto3

key_id = 'XXXXXXXXXXXXXX'
key_secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXX'

session = boto3.Session(
    aws_access_key_id=key_id,
    aws_secret_access_key=key_secret)

# Define the VM properties
image_id='ami-0d0eaed20348a3389' # bionic ca-central LTS
instance_name='Test9'
security_group_name = 'SupportService'
instance_type = 't2.micro'

# Put the properties into the expected format
tag_spec = [{
    'ResourceType': 'instance',
    'Tags': [{
        'Key': 'Name',
        'Value': instance_name }]}]
security_groups=[security_group_name]

# Reserve the instance
ec2_client = session.client('ec2')
reservation = ec2_client.run_instances(
        ImageId=image_id,
        MinCount=1,
        MaxCount=1,
        InstanceType=instance_type,
        KeyName='bwsupport',
        TagSpecifications=tag_spec,
        SecurityGroups=security_groups )

# Now the VM is creating. Optionally you can wait and inspect the instace
instance_data = reservation['Instances'][0]
iid = instance_data['InstanceId']
print(f'Reserved instance "{instance_name}" as {iid}. Waiting until running.')
ec2_resource = session.resource('ec2')
instance = ec2_resource.Instance(iid)
instance.wait_until_running()
state = instance.state['Name']
print(f'Instancei {iid} is {state} - Public IP: {instance.public_ip_address}')
</code></pre>
<h2 id="terminate">Terminate</h2>
<p>You can use the same code as above to get the <code>instance</code> variable.
From there, it's this easy:</p>
<pre><code class="language-python">instance.terminate()
instance.wait_until_terminated()
</code></pre>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to restrict access to a Lambda function using AWS IAM
roles. It builds on the function written <a href="/aws-lambda.html">in my last post</a></p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#create-a-policy-for-your-lambda-function">Create a policy for your lambda function</a>
<ul>
<li><a href="#get-the-arn-of-your-function">Get the ARN of your function</a></li>
<li><a href="#create-the-policy">Create the policy</a></li>
</ul>
</li>
<li><a href="#invoke-lambda-from-python">Invoke Lambda from Python</a></li>
</ul>
<h1 id="create-a-policy-for-your-lambda-function">Create a policy for your lambda function</h1>
<h2 id="get-the-arn-of-your-function">Get the ARN of your function</h2>
<p>Navigate to your function in the AWS Lambda Console. On the top right, the ARN
is displayed. Copy it, you'll use it in the policy.</p>
<h2 id="create-the-policy">Create the policy</h2>
<p>Open the IAM Console and create a new policy.
The JSON looks like this. Note the ARN copied from the Function.</p>
<pre><code class="language-JSON">{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LambdaInvokeHelloWorld",
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction",
                "lambda:InvokeAsync"
            ],
            "Resource": "arn:aws:lambda:ca-central-1:850047500507:function:HelloWorld"
        }
    ]
}
</code></pre>
<p>Name the policy and assign it to your user. This user should have a Key ID and
Secret Key.</p>
<hr>
<h1 id="invoke-lambda-from-python">Invoke Lambda from Python</h1>
<p>Note that the payloads are in byte format.</p>
<pre><code class="language-python">import json
import boto3

key_id = 'XXXXXXXXXXXXXXXXX'
key_secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

session = boto3.Session(
    aws_access_key_id=key_id,
    aws_secret_access_key=key_secret)

client = session.client('lambda')

body = {'name': 'Kyle'}
payload = str.encode(json.dumps(body))
response = client.invoke(FunctionName='HelloWorld', Payload=payload)
print(response['Payload'].read())
</code></pre>
<p>If the user identified by that key has the above permission, they'll be able
to execute this function. Otherwise they'll get a nice permissions error.</p>


<p><strong>This post is linked to from the <a href="/gcp.html">GCP: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to interact with Google's Cloud Firestore, using examples
written in Python. The concepts should apply to any language.</p>
<p>Firestore is the newer version of Datastore. You can find the differences
<a href="https://cloud.google.com/firestore/docs/firestore-or-datastore">here</a>.
In this post, the  database is configured in Native mode.
Other than some extra language support, Native mode's coolest new feature seems
to be the ability to listen to a post. From what I can tell, this could replace
the need for pub/sub in some scenarios.</p>
<p>Throughout the post I'll use a project named "example". Be sure to replace this
with your own project name.</p>
<hr>
<h1 id="pricing">Pricing</h1>
<p>Official Firestore pricing data is <a href="https://cloud.google.com/firestore/pricing">here</a>.</p>
<p><strong>Note that Firestore mode is almost twice the price of Datastore mode.</strong>
Datastore pricing can be found <a href="https://cloud.google.com/datastore/pricing">here</a>.</p>
<p>Every day a number of free operations are allowed, which for a small
application makes Datastore very affordable. Even after the free tier though,
the pricing is pretty good. Firestore and Datastore have the same free tiers.</p>
<h2 id="summary-of-pricing-datastore-vs-firestore">Summary of pricing: Datastore vs Firestore</h2>
<ul>
<li>Storage: $/GB/Month
<ul>
<li>Datastore: 0.108</li>
<li>Firestore: 0.180</li>
</ul>
</li>
<li>Reads: $/100,000
<ul>
<li>Datastore: 0.036</li>
<li>Firestore: 0.060</li>
</ul>
</li>
<li>Writes: $/100,000
<ul>
<li>Datastore: $0.108</li>
<li>Firestore: $0.180</li>
</ul>
</li>
<li>Deletes: $/100,000
<ul>
<li>Datastore: 0.012</li>
<li>Firestore: 0.020</li>
</ul>
</li>
</ul>
<p>Max entity size and document size are both 1MB, so you could realistically just
use JSON strings in entities instead to save money if you don't need any of
Firestore's other features.</p>
<hr>
<h1 id="datastorefirestore-concepts">Datastore/Firestore Concepts</h1>
<p>Both options use a non-relational/NoSQL database. NoSQL databases in general
allow for the direct serialization of a programs objects/dictionaries.</p>
<h2 id="firestore">Firestore</h2>
<p><a href="https://cloud.google.com/firestore/docs/data-model">Official Firestore data model docs</a></p>
<p>Firestore uses a document database organized into documents and collections.
This is a change from the older Datastore model of Entities organized by Kinds.</p>
<p>Documents support nested data structures ("Maps"). Think of them like JSON
objects, but with more types supported. Documents are limited to 1MB each.</p>
<p>Documents are stored in Collections.
Collections are containers for documents, like a filesystem's directories.
Documents can also contain their own nested collections, which store other
documents. Nesting has a depth limit of 100.</p>
<p>Documents are identified by References. In python, a reference looks like
this:</p>
<pre><code class="language-python">collection('collection1').document('document1')
</code></pre>
<h2 id="datastore">Datastore</h2>
<p>In Datastore, objects are called Entities. The key-value data pairs stored by
entities are called properties.</p>
<p>Entities are uniquely identified by keys.
Entity Keys consists of the following components:</p>
<ul>
<li><em>Namespace</em>: Basically a tenant identifier, like a user ID. By default, the
namespace is an empty string. Any entity created as a "child" will inherit
its parent's namespace.</li>
<li><em>Kind</em>: A string describing the category or purpose of the entity.</li>
<li><em>Identifier</em>: Unlike namespace and kind which are shared between like
entities, the identifier is unique to each entity. This is either a string
called a "key name", or an integer numeric ID.</li>
<li><em>Ancestor Path</em>: An optional component of the key, identifying its parents to
form a database hierarchy.</li>
</ul>
<hr>
<h1 id="create-iam-service-account">Create IAM Service Account</h1>
<p>Before your code can access Datastore/Firestore, access needs to be configured
in the <a href="https://console.cloud.google.com/iam-admin">Identity Access Management</a>
tool.</p>
<p>First, create a <a href="https://console.cloud.google.com/iam-admin/serviceaccounts">Service Account</a>:</p>
<ol>
<li>Open the Service Account page</li>
<li>Select your project</li>
<li>Click "Create Service Account"</li>
<li>Service Account Details:
<ol>
<li>Service account name: <code>PythonDatastoreAdmin</code></li>
<li>Service account ID: <code>pythondatastoreadmin</code></li>
<li>Service account description: SA with Full Datastore Access for Python</li>
</ol>
</li>
<li>Grant this service account access to project (optional)
<ol>
<li>Select a role: Datastore > Cloud Datastore Owner</li>
<li>Continue</li>
</ol>
</li>
<li>Grant users access to this service account (optional)
<ol>
<li>Download a key file for Python to use. Use JSON as the key type.</li>
<li>Done</li>
</ol>
</li>
</ol>
<p>I downloaded my key to <code>$HOME/gcp_keys/datastore_admin.json</code></p>
<hr>
<h1 id="python-environment-setup">Python Environment Setup</h1>
<p>Build a virtualenv. If you're in a git repo, don't forget to ignore this
directory.</p>
<pre><code class="language-bash">python3 -m venv env3
source env3/bin/activate
</code></pre>
<p>Write the requirement file</p>
<p><code>vi requirements.txt</code></p>
<pre><code class="language-text">google-cloud-firestore
</code></pre>
<p>Install the requirements</p>
<pre><code class="language-bash">pip install -r rqeuirements.txt
</code></pre>
<hr>
<h1 id="python-code-examples">Python Code Examples</h1>
<p>A fake project name of "example" will be used. Be sure to use a valid one.</p>
<p>Some notes about the Firestore python API:</p>
<ul>
<li><code>firestore.client().collection</code> returns a <a href="https://googleapis.dev/python/firestore/latest/collection.html">CollectionReference</a>,
which allows you to operate on the named collection.</li>
<li><code>firestore.client().collection().list_documents</code> returns a <a href="google.api_core.page_iterator.GRPCIterator">Page Iterator</a>.
Casting a page iterator to a list will trigger a read operation, returning a
list of <code>DocumentReference</code> objects.</li>
<li><code>firestore.client().collection().document</code> returns
a <a href="https://googleapis.dev/python/firestore/latest/document.html">DocumentReferece</a>,
which interacts with the actual Firestore documents. The <code>DocumentReference</code>
object can operate (get/set/create) on the actual Document.</li>
<li><code>firestore.client().collection().document().get()</code> will return
a <a href="https://developers.google.com/android/reference/com/google/firebase/firestore/DocumentSnapshot">DocumentSnapshot</a>
of the document. The document snapshot is how you read data from a document.
Document snapshots are a representation of the document state taken at the
instant of their creation.</li>
<li><code>firestore.client().collection().document().create()</code> makes a new document.
If the object already  existed, a <code>google.api_core.exceptions.AlreadyExists</code>
exception is thrown.</li>
<li>Write operations (create &#x26; set) return a <a href="https://cloud.google.com/firestore/docs/reference/rest/v1/WriteResult">WriteResult</a>.</li>
</ul>
<pre><code class="language-python">from os import environ
from google.cloud import firestore

# The key file is an exported JSON file from the IAM tool
key_file_path = '{}/gcp_keys/datastore_admin.json'.format(environ['HOME'])

# When running this on a local system, the cloud.google package from GCP
# expects the GOOGLE_APPLICATION_CREDENTIALS environment variable to be set
# to the exported IAM key file's path
environ['GOOGLE_APPLICATION_CREDENTIALS'] = key_file_path

# google.cloud.datastore.Client expects the project name as an argument
project = 'example'

# Instantiate the datastore client. This step authenticates to GCP.
# See the Credential Error note below if authentication fails
client = firestore.Client(project)

# Get a reference to the test collection. This returns a
# google.cloud.firestore_v1.collection.CollectionReference object
ref_col_test = client.collection('test')

# Get a reference to a document which may or may not exist. This returns a
# google.cloud.firestore_v1.document.DocumentReference object
ref_doc_document1 = ref_col_test.document('document1')

# Perform the actual database query to get the document. get() returns a
# google.cloud.firestore_v1.document.DocumentSnapshot object.
snapshot_document1 = ref_doc_document1.get()

# Note that there is no data
assert document1.to_dict() is None

# You can use document.exists to check if the document already exists.
# Here's an example of doing a safe insert of some sample data:
sample_data = {
    'nums': {'num1': 100, 'num2': 200},
    'strs': {'animal1': 'dog', 'animal2': 'cat'}}
if snapshot_document1.exists is False:
    createe_write_result = ref_doc_document1.create(sample_data)

document1_data = ref_doc_document1.get().to_dict()

# Finally, update the data
sample_data['strs']['animal3'] = 'bird'
set_write_result = ref_doc_document1.set(sample_data)

# List the documents in the collection. The iterator returns a paginated list
# of document references. You can then operate on those references.
doc_list_iterator = ref_col_test.list_documents()
doc_ref_list = list(doc_list_iterator)
sample_document_snapshot = doc_ref_list[0].get()

# Show that the changes were applied to the data. This will return 'bird'
animal3 = sample_document_snapshot.to_dict()['strs']['animal3']
</code></pre>
<h2 id="troubleshooting">Troubleshooting</h2>
<h3 id="credentials-error">Credentials Error</h3>
<p>If your local google cloud authentication isn't set up correctly, you'll get an
error saying:</p>
<pre><code class="language-text">google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
</code></pre>
<p>If that happens, review the IAM Service account steps.</p>
<h3 id="iterator-has-already-started">Iterator has already started</h3>
<p>If you try and cast a <code>GRPCIterator</code> to list twice, you get a <code>ValueError</code>.</p>
<pre><code class="language-text">ValueError: ('Iterator has already started', &#x3C;google.api_core.page_iterator.GRPCIterator object)
</code></pre>
<p>The <code>ValueError</code> is thrown because <code>._started</code> is <code>True</code> in the iterator. It has
already done its thing and won't work again. I didn't see any good way to
restart the iterator without discarding it.
You can see the details in the <a href="https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/page_iterator.py">page iterator source code</a>.</p>
<p>Best course is just to get a new iterator from the reference object.
See in the following example how the ID changes. The <code>doc_list_iterator</code> var is
being pointed at a new iterator. The old iterator gets gc'd by Python as usual.</p>
<pre><code>>>> id(doc_list_iterator)
4422123408
>>> doc_list_iterator = ref_col_test.list_documents()
>>> id(doc_list_iterator)
4422123472
</code></pre>
<p>It will work as expected.</p>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to upload and download files from Amazon's S3 using
Python. Setup is done from the web UI.</p>
<h1 id="create-a-bucket">Create a Bucket</h1>
<p>Go to the <a href="https://s3.console.aws.amazon.com">AWS S3 page</a>.</p>
<p>Click on "+ Create Bucket" and fill out the form. For access, choose
Block all public access. For the rest of the options, the defaults are fine.</p>
<hr>
<h1 id="create-iam-credentials">Create IAM Credentials</h1>
<p>IAM stands for Identity and Access Management.</p>
<h2 id="create-a-policy">Create a Policy</h2>
<p>From the left nav-bar, go to policies. The default policies don't grant
programmatic access, so you need to make a new one.</p>
<p>For admin rights, I just used the visual policy builder and granted it
access to everything. I'll come back and update this guide with how to lock
that down later.</p>
<hr>
<h1 id="add-the-policy-to-a-group">Add the policy to a group</h1>
<p>Go to the groups on the left. Either make a new one or edit an existing one.
Under Permissions, click Attach Policy. If you want, you can also add
<code>AmazonS3FullAccess</code> to see the difference between the two.</p>
<h2 id="createedit-a-user">Create/Edit a User</h2>
<p>On the left nav-bar, go to Users > Add User. Or edit a user that exists.
Fill in the wizard and assign the group.</p>
<hr>
<h1 id="use-python-to-interact-with-s3">Use Python to interact with S3</h1>
<h2 id="install-boto3">Install Boto3</h2>
<p>Python needs boto3 installed to interact with AWS.</p>
<pre><code class="language-bash">pip install boto3
</code></pre>
<h2 id="uploading-and-downloading-files">Uploading and Downloading Files</h2>
<pre><code class="language-python">import boto3

# Fill in the key and key id, or read them from stdin/file/whatever
key_id = ''
key = ''

# Create the authenticated s3 client
session = boto3.Session(aws_access_key_id=key_id, aws_secret_access_key=key)
client = session.client('s3')

# Define the file/object
filename = '/example/file/path'
bucket_name = 'my_bucket'
key = 'my_key'

# Upload Example
client.upload_file(filename, bucket_name, key)

# Download Example
client.download_file(bucket_name, object_name, path)
</code></pre>


<p><strong>This post is linked to from the <a href="/open-source-cli-project.html">FOSS Python App Project</a></strong></p>
<hr>
<p>At a high level, the steps to upload a package to PyPi are as follows:</p>
<ol>
<li>Write the project code</li>
<li>Create a release on GitHub and put it in the <code>download_url</code> of setup.py</li>
<li>Write a setup.py file to build and describe the project</li>
<li>Write a setup.cfg file pointing to the readme</li>
<li>create a source distribution</li>
</ol>
<h1 id="write-the-project-code">Write the project code</h1>
<p>I'm not going to cover how to build a python project in this post, but here's
the very small and simple project I used when learning this procedure:
<a href="https://github.com/kylep/jsc2f">GitHub: kylep/jsc2f</a>.</p>
<hr>
<h1 id="create-a-release-on-github">Create a Release on GitHub</h1>
<p>Log into your GitHub project and click on releases. If you can't find the link,
you can also just append <code>/releases</code> to the end of the URL to get there.</p>
<p>Click "Create a new release". Enter your tag and branch, along with a release
title and description.</p>
<p>Under the Assets section of the new release there's a (tar.gz) link to download
the source code. If you right click it and copy the URL, that's used next in
the setup.py file.</p>
<hr>
<h1 id="write-a-setuppy-file">Write a setup.py file</h1>
<p>Here's mine. Check GitHub to see if I've changed it.
The <code>download_url</code> value comes from the GitHub release.</p>
<pre><code class="language-python">from setuptools import setup

# Without this you'll get no description on the pypi site
with open("README.md", "r") as fh:
    long_description = fh.read()

setup(
    name='jsc2f',
    packages=['jsc2f'], # Alternatively you can use find_packages()
    version='0.1', # match the github release you make later
    license='MIT', # match the LICENSE file in the project
    description='Saves a JSON fields SQLs cell to a file, or UPDATE it back',
    author='Kyle Pericak',
    author_email='kyle@pericak.com',
    url='kyle.pericak.com/jsc2f', # either your own page or github
    download_url='https://github.com/kylep/jsc2f/archive/v0.1.tar.gz',
    keywords=['SQL', 'JSON', 'file'],
    install_requires=[
        'click',
        'mysql-connector'
    ],
    entry_points='''
        [console_scripts]
        jsc2f=jsc2f.cli:cli
    ''',
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Information Technology',
        'Topic :: Database',
        'Programming Language :: Python :: 3',
        'License :: OSI Approved :: MIT License',
        'Natural Language :: English'

</code></pre>
<h2 id="picking-a-topic">Picking a topic:</h2>
<p>I used <a href="https://pypi.org/pypi?%3Aaction=list_classifiers">this classifier list</a></p>
<hr>
<h1 id="write-setupcfg">Write setup.cfg</h1>
<p>This goes in the project root next to setup.py.</p>
<p><code>vi setup.cfg</code></p>
<pre><code class="language-ini">[metadata]
description-file = README.md
</code></pre>
<h1 id="creating-a-source-distribution">Creating a source distribution</h1>
<p>For this you use the python <a href="https://docs.python.org/2/distutils/sourcedist.html">sdist</a>
tool and <a href="https://pypi.org/project/twine/">twine</a>.</p>
<p>From the git project root:</p>
<pre><code class="language-bash">pip install twine

# This will create dist/
python setup.py sdist

# Twine uploads the dist/ files to Pypi
twine upload --skip-existing dist/*
# Enter your username and password for PyPI
</code></pre>


<p>Often I need to operate on a huge JSON string that's saved in a MaraiDB
database. Here's how I do that.</p>
<p>It needs to be said that raw SQL queries like this are generally a bad idea
compared to something like <a href="https://docs.sqlalchemy.org/en/13/orm/">SQLAlchemy's ORM</a>.
Having said that, this example is way smaller and more portable.</p>
<p>The way this works is you open up an interactive python session, ideally in
a virtualenv. Set the variable values at the top. Run the two functions as
needed. In another window/pane, edit the JSON files.</p>
<p>Another option is to use <code>jq</code> with interactive MySQL commands. I'm not covering
that in this post, though.</p>
<h1 id="environment-setup">Environment setup</h1>
<p>This is best done in a venv if <code>mysql-connector</code> doesn't need to be installed
on the base system.</p>
<pre><code class="language-bash">pip install mysql-connector
</code></pre>
<h1 id="the-code">The code</h1>
<p>It's not a bad idea to write this into a parameterized CLI or whatever.
I just run this in an interactive python shell in a dedicated window/pane.</p>
<pre><code class="language-python">import json
import mysql.connector as mariadb

# Fill in the blanks
db_host= ''  # IP address
db_port = 3306
db_username = ''
db_password = ''
db_name = ''
table_name = ''
column_name = ''
where = ''  # (optional) where = 'WHERE `x` == 1'


def save_to_file(filename):
    """Read the json data from the database, save to filename"""
    conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
    cursor = conn.cursor()
    read_query = 'SELECT {} from {}{}'.format(column_name, table_name, where)
    cursor.execute(read_query)
    data = cursor.fetchone()[0]
    conn.close()
    jdata = json.dumps(json.loads(data), indent=4, sort_keys=True)
    with open(filename, 'w+') as out_file:
        out_file.write(jdata)
    print('done')


def update_from_file(filename):
    """Read data from filename, write to database"""
    with open(filename) as json_file:
        jdata = json.load(json_file)
    data = json.dumps(jdata)
    conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
    cursor = conn.cursor()
    update_query = "UPDATE {} SET {} = '{}'{}".format(table_name, column_name,
                                                      data, where)
    cursor.execute(update_query)
    conn.commit()
    conn.close()
    print('done')
</code></pre>
<h1 id="usage">Usage</h1>
<p>From an interactive python terminal, paste the above script in.</p>
<p>Then to save the file, run something like:</p>
<pre><code class="language-python">save_to_file(filename='/home/kyle/sqldata.json')
</code></pre>
<p>Then you can update the file using an editor <code>vi /home/kyle/sqldata.json</code>.</p>
<p>Back in the Python terminal, you can then push the data back up.</p>
<pre><code class="language-python">update_from_file(filename='/home/kyle/sqldata.json')
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page contains python-related mini-guides and minor posts.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#pylint-ignore-comments">Pylint Ignore Comments</a>
<ul>
<li><a href="#turn-off-linting-for-now">Turn off Linting for now</a></li>
</ul>
</li>
<li><a href="#get-an-object-from-list-of-objects-where-the-object-property-has-value-x">Get an object from list of objects where the object property has value x</a>
<ul>
<li><a href="#a-subset-list-is-expected-list-comprehension">A subset list is expected: list comprehension</a></li>
<li><a href="#get-one-match-next-w-generator-expression">Get one match: next() w/ generator expression</a></li>
</ul>
</li>
<li><a href="#iterate-through-dictionary">Iterate through dictionary</a></li>
<li><a href="#print-json-nicely-in-ascii-no-unicode-u-prefix">Print JSON nicely in ASCII (no unicode 'u' prefix)</a></li>
<li><a href="#replace-a-string-in-a-file">Replace a string in a file</a></li>
<li><a href="#get-environment-variable">Get environment variable</a></li>
<li><a href="#use-jinja2-for-templating">Use Jinja2 for templating</a>
<ul>
<li><a href="#jinja2-templating-with-files">Jinja2 templating with files</a></li>
</ul>
</li>
<li><a href="#execute-sql-query-against-mysql-or-mariadb">Execute SQL query against MySQL or MariaDB</a></li>
<li><a href="#read-from-std-in-bash-pipe">Read from std in (bash pipe)</a></li>
<li><a href="#custom-exceptions">Custom Exceptions</a></li>
<li><a href="#cant-run-pip-after-updating">Can't run pip after updating</a></li>
</ul>
<h1 id="pylint-ignore-comments">Pylint Ignore Comments</h1>
<p>I can never remember the syntax to ignore certain Pylint warnings. Yeah you can
turn them off everywhere but usually I just want to ignore a certain occurrence.</p>
<pre><code class="language-python"># - This file is a mess and I don't want pylint here:
# pylint: disable-all

# - troublesome import not being found:
# pylint: disable=import-error

# - redefined-builtin (W0622):
# pylint: disable=W0622

# - too-many-public-methods (R0904):
# pylint: disable=R0903

# - too-many-arguments (R0913):
# pylint: disable-msg=R0913

# - too-many-locals (R0914)
# pylint: disable=R0914

# - Invalid constant name:
# pylint: disable=C0103

# - Catching too general exception:
# pylint: disable=W0703

# - Using python3 compatible print statement makes python2 linter mad
# pylint: disable=superfluous-parens

# - Ignore too-long line (this needs both the pylint command and the trailing no-qa)
# pylint: disable=line-too-long
binaryValue = b'QVdTOmV5SndZWGxzYjJGa0lqb2lVSFkwWXpKeFNEaHZXVkkxVmtwRmVXdFhXamhKUVRGcGNFWk1kelJKZEV0V1UwSk9'  # noqa: E50

# - Wrong Import Position (Required when putting python version check up top)
# flake8: noqa=402
# pylint: disable=wrong-import-position
</code></pre>
<h2 id="turn-off-linting-for-now">Turn off Linting for now</h2>
<p>In VIM, run:</p>
<pre><code>:ALEToggle
</code></pre>
<h1 id="get-an-object-from-list-of-objects-where-the-object-property-has-value-x">Get an object from list of objects where the object property has value x</h1>
<h2 id="a-subset-list-is-expected-list-comprehension">A subset list is expected: list comprehension</h2>
<p>See <a href="https://docs.python.org/3/tutorial/datastructures.html">Section 5.1.3 - List comprehension</a></p>
<pre><code class="language-python"># This will return a list of objects whith a value
[obj for obj in haystack_list if obj.value == search]

# Example of list comprehension to get the even numbers in a list
nums = [1, 2, 3, 4, 5, 6, 7, 8]
even_nums = [num for num in nums if num % 2 == 0]
</code></pre>
<h2 id="get-one-match-next-w-generator-expression">Get one match: next() w/ generator expression</h2>
<p>The <code>next()</code> method will return the first element found in an iterable.</p>
<p>See <a href="https://www.python.org/dev/peps/pep-0289/">PEP 289 - Generator expressions</a></p>
<pre><code class="language-python"># Raises a StopIteration exception if no match is found
needle = next((obj for obj in haystack_list if obj.value == value))

# To return a value instead of raising an exception, pass a second argument
needle = next((obj for obj in haystack_list if obj.value == search), None)

# Example of what Next is really doing. This returns 1.
next(iter([1,2,3]))
</code></pre>
<hr>
<h1 id="iterate-through-dictionary">Iterate through dictionary</h1>
<pre><code class="language-python">ex_dict = {'a': 1, 'b': 2, 'c': 3}
for key in ex_dict:
    print(key)
    print(ext_dict[key]) # value
</code></pre>
<p>This works because <code>ex_dict.keys() == list(ex_dict)</code></p>
<hr>
<h1 id="print-json-nicely-in-ascii-no-unicode-u-prefix">Print JSON nicely in ASCII (no unicode 'u' prefix)</h1>
<p>This is like using <a href="https://docs.python.org/3/library/pprint.html">pprint</a> on
the JSON object but without the unicode prefixes.</p>
<pre><code class="language-python">data = ... # the json string
jdata = json.dumps(json.loads(data),  indent=4, sort_keys=True)
print(jdata)
</code></pre>
<hr>
<h1 id="replace-a-string-in-a-file">Replace a string in a file</h1>
<p>Though honestly, <a href="https://www.geeksforgeeks.org/sed-command-in-linux-unix-with-examples/">sed</a>
is usually a better option. Or if there are a lot of values, I like to use a
jinja template.</p>
<pre><code class="language-python">file_path = '/home/user/example_file'
find_str = 'REPLACE_THIS'
replace_str = 'WITH_THIS'

# Open the file to read the value
with open(file_path, 'r') as read_file:
    file_content = read_file.read()

# Replace the string
file_content = file_content.replace(find_str, replace_str)

# Write the file
with open(file_path, 'w') as write_file:
    write_file.write(replaced_content)
</code></pre>
<hr>
<h1 id="get-environment-variable">Get environment variable</h1>
<p>I mostly use this with <a href="https://docs.docker.com/engine/reference/builder/">Dockerfile ENV values</a>.</p>
<pre><code class="language-python">import os

environment_var_name = 'MY_ENV_VAR'
environment_var_vlue os.environ[environment_var_name]
</code></pre>
<hr>
<h1 id="use-jinja2-for-templating">Use Jinja2 for templating</h1>
<p>I use this for config files, like <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_templating.html">how Ansible does it</a>.</p>
<pre><code class="language-python">from jinja2 import Template

replacements = {'USERNAME': 'kyle', 'PASSWORD': 'example'}
template_text = 'connection = mysql://{{USERNAME}}:{{PASSWORD}}@127.0.0.1'
template = Template(template_text)
replaced_text = template.render(**replacements)
</code></pre>
<h2 id="jinja2-templating-with-files">Jinja2 templating with files</h2>
<p>Same idea as above but it reads a .j2 file as input and writes it to a config
file. Can combine this nicely with environment variables to write config files
as part of a Docker start script.</p>
<pre><code class="language-python">from jinja2 import Template

def apply_template(jinja2_file, output_file, replacements):
    """Replace the replacements values in jinja2_file, write to output_file"""
    with open(jinja2_file, 'r') as j2_file:
        j2_text = j2_file.read()
    template = Template(j2_text)
    replaced_text = template.render(**replacements)
    with open(output_file, 'w+') as write_file:
        write_file.write(replaced_text)

</code></pre>
<hr>
<h1 id="execute-sql-query-against-mysql-or-mariadb">Execute SQL query against MySQL or MariaDB</h1>
<pre><code class="language-python">import mysql.connector as mariadb

conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
cursor = conn.cursor()
cursor.execute(sql_query)
conn.commit() # if it was a write action
conn.close()
</code></pre>
<hr>
<h1 id="read-from-std-in-bash-pipe">Read from std in (bash pipe)</h1>
<p>For instance I have a file named x that has a big string where I want to
replace commas with spaces. Sed works but I can never remember the syntax.</p>
<pre><code class="language-python">cat x | python -c "import sys; print(sys.stdin.read())"

# Example of using python instead of sed to replace , with spaces
cat x | python -c "import sys; print(sys.stdin.read().replace(',',' '))"

# You can use this in fun ways like combining it with bash loops
cat x | python -c \
  "import sys; print(sys.stdin.read().replace(',','\n'))" | while read line; do
  echo $line
done
</code></pre>
<hr>
<h1 id="custom-exceptions">Custom Exceptions</h1>
<pre><code class="language-python">class MyCustomException(Error):
    pass
</code></pre>
<hr>
<h1 id="cant-run-pip-after-updating">Can't run pip after updating</h1>
<p>So you ran into a problem because ubuntu ships an ancient version of pip, and
some dependency package like <code>subprocess32</code> is breaking your install. A forum
post says you should update pip.</p>
<pre><code class="language-bash">pip install -U pip
</code></pre>
<p>Except now pip won't run at all.</p>
<pre><code class="language-bash">pip install python-openstackclient
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in &#x3C;module>
    from pip import main
ImportError: cannot import name main
</code></pre>
<p>I think maybe the pip executable file moved after the update.
To fix it:</p>
<pre><code class="language-bash">hash -r pip
</code></pre>
<p>Now pip works again.</p>
<p>As an aside, if <code>subprocess32</code> really was your problem, updating pip didn't
help. For some reason you have to use apt to install <code>python-subprocess32</code>.</p>


Kyle Pericak

RAG on the Bot-Wiki

Transferring files to Windows through Python's SimpleHTTPServer

Python Ceph Examples

AWS CodeBuild CICD - Deploy to Lambda

Authenticating with AWS IAM in AWS API Gateway

Create & Terminate EC2 Instances from Python

IAM Auth for Lambda

Google Cloud Firestore Basics - Python

Using AWS S3 from Python

Upload a Python Package to PyPi

Working with JSON SQL values in Python

Python Reference Page