<h1 id="aside-leveraging-ai-for-blog-content">(Aside) Leveraging AI for Blog Content</h1>
<p>Most of this change was written by Codex, and finished up by Cursor. Cursor did a much better job actually running and fixing the errors in the build. When I joined Super.com, my current employer, I rewrote this entire SSG blog engine from Pelican into React + Next as a learning exercise. I'd been writing production code in Angular for years so it wasn't a big jump. I've picked at this blog over the years, but purely as an engine to keep my skills sharp since I'm a senior leader now and spend most of my days in meetings or planning technical and organizational strategy. This is the first blog post in a long time.</p>
<p>AI wrote the first draft. It was bad. For the code, that was ok. It took some iterating but AI codes well enough now if you hold its hand.</p>
<p>The mermaid chart syntax example in this post is all AI. The actual writing style of AI makes my eyes glaze over and I don't want to subject other people to it, so I rewrote basically all of the actual text. No sense contributing too much towards a dead internet, right?</p>
<p>Have feelings about the idea of AI being used for this sort of thing? I'd love to hear your thoughts. You can find my socials on the about page.</p>
<hr>
<h1 id="how-the-rendering-works">How the rendering works</h1>
<p>Take a look at the <a href="https://github.com/mermaid-js/mermaid">official GitHUb page for Mermaid-JS</a> for more details about mermaid in general. It's really neat and I'm convinced it'll be considered the standard for flow charts for the next while.</p>
<p>The markdown loader still starts by parsing front matter with <a href="https://github.com/jonschlinkert/gray-matter"><code>gray-matter</code></a>, but the Remark pipeline now detects any fenced code block labeled <code>mermaid</code> and rewrites it into HTML with the <code>mermaid</code> class.</p>
<p>It's all pretty simple from there. You can see <code>remarkMermaidToHtml</code> (from <a href="https://github.com/kylep/multi/blob/main/apps/blog/blog/utils/remarkMermaid.js">remarkMermaidToHtml in remarkMermaid.js</a>) is new here:</p>
<pre><code class="language-javascript">	static async getRenderedMarkdownAndProps(rawMarkdown){
		// contentHtml is the rendered markdown, metaData is the front matter as a flat object
		const { data: metaData, content: markdown } = matter(rawMarkdown);
		MarkdownService.#serializeDates(metaData); 
                const result = await remark()
                        .use(remarkGfm)
                        .use(remarkToc, {heading: 'Table of contents'})
                        .use(remarkMermaidToHtml)
                        .use(remarkHtml, { sanitize: false })
                        .process(markdown);
		const contentHtml = result.toString();
		const props = {props: {contentHtml, metaData},};
		return props;
	 };
</code></pre>
<hr>
<h1 id="mermaid-syntax-reference">Mermaid Syntax Reference</h1>
<p>Here's a trivial reference of how to use Mermaid syntax.</p>
<p>I plan to use this reference myself for future content, and may update it with more useful examples in time.</p>
<h2 id="flowchart">Flowchart</h2>
<pre><code class="language-text">graph TD;
  Source[Markdown] --> Remark;
  Remark --> |HTML| Browser;
  Browser --> MermaidRuntime;
  MermaidRuntime --> |SVG| PublishedPost;
</code></pre>
<pre class="mermaid">
graph TD;
  Source[Markdown] --> Remark;
  Remark --> |HTML| Browser;
  Browser --> MermaidRuntime;
  MermaidRuntime --> |SVG| PublishedPost;
</pre>
<h2 id="sequence-diagram">Sequence diagram</h2>
<pre><code class="language-text">sequenceDiagram
  participant Author
  participant Blog
  participant Mermaid
  Author->>Blog: Write markdown with mermaid fences
  Blog->>Mermaid: Request runtime initialization
  Mermaid-->>Blog: Render SVG
  Blog-->>Author: Display diagram
</code></pre>
<pre class="mermaid">
sequenceDiagram
  participant Author
  participant Blog
  participant Mermaid
  Author->>Blog: Write markdown with mermaid fences
  Blog->>Mermaid: Request runtime initialization
  Mermaid-->>Blog: Render SVG
  Blog-->>Author: Display diagram
</pre>
<h2 id="state-diagram">State diagram</h2>
<pre><code class="language-text">stateDiagram-v2
  [*] --> Draft
  Draft --> Review: lint &#x26; build
  Review --> Published: merge
  Published --> Archived
</code></pre>
<pre class="mermaid">
stateDiagram-v2
  [*] --> Draft
  Draft --> Review: lint & build
  Review --> Published: merge
  Published --> Archived
</pre>
<h2 id="class-diagram">Class diagram</h2>
<pre><code class="language-text">classDiagram
  class MarkdownService {
    +getRenderedMarkdownAndProps()
  }
  class BlogPostContentPage {
    +useEffect mermaid bootstrap
  }
  MarkdownService --> BlogPostContentPage : HTML
</code></pre>
<pre class="mermaid">
classDiagram
  class MarkdownService {
    +getRenderedMarkdownAndProps()
  }
  class BlogPostContentPage {
    +useEffect mermaid bootstrap
  }
  MarkdownService --> BlogPostContentPage : HTML
</pre>


<p>When you import a RedHat VM, say from VMWare, to a KVM environment like OpenStack it probably won't boot and instead goes straight to the dracut prompt.</p>
<p>This happens because the ramdisk doesn't have virtio drivers. You need to inject them. Import the volume and attach it to a running Linux server.
Personally I use Ubuntu, but it doesn't matter since you'll <code>chroot</code> in.</p>
<h1 id="steps">Steps</h1>
<h2 id="mount-the-volume">Mount the volume</h2>
<ol>
<li>In your cloud environment, mount the RedHat boot volume to your working Ubuntu (or whatever) machine.</li>
<li>Find the boot and data volumes, and mount them too.</li>
<li>bind-mount your system paths into the root volume's mount point</li>
</ol>
<pre><code class="language-bash">mkdir /mnt/boot /mnt/root
mount &#x3C;boot device> /mnt/boot
mount &#x3C;root device> /mnt/root

for x in sys proc run dev tmp; do mount --bind /$x /mnt/root/$x; done
</code></pre>
<h2 id="use-dracut-to-update-the-initrd-file">Use Dracut to update the initrd file</h2>
<ol>
<li>chroot into the volume</li>
<li>Check <code>/etc/grub/grub.conf</code> to see which initrd file is being used. Something like <code>3.10.0-1062.1.2.el7.x86_64</code>.</li>
<li>check if the drivers are present</li>
<li>install the drivers</li>
<li>confirm they're present</li>
</ol>
<pre><code class="language-bash"># chroot in
cd /mnt
chroot root bash

# set the version
cat /etc/grub/grub.cnf
version=
# example: version=3.10.0-1062.1.2.el7.x86_64

# check for virtio drivers
lsinitrd /boot/initramfs-$version.img | grep virtio

# if not found, add them
cd /boot
dracut -f \
  /boot/initramfs-$version.img \
  $version \
  --add-drivers "virtio virtio_pci virtio_blk virtio_net"

# prove it worked
lsinitrd /boot/initramfs-$version.img | grep virtio

# leave the chroot
exit
</code></pre>
<p>This is a good time to do any other modifications you like, such as installing cloud-init or enabling DHCP.</p>
<h2 id="clean-up">Clean up</h2>
<p>Remove the mounts</p>
<pre><code class="language-bash">for x in sys proc run dev tmp; do umount /mnt/root/$x; done
umount /mnt/root
umount /mnt/boot
</code></pre>
<h2 id="validate">Validate</h2>
<p>Now you should be able to boot your RedHat VM without dracut coming up.</p>


<p>Normally when I need to make a CSR, I would use <code>openssl</code> to generate one like you'll see all over the internet:</p>
<pre><code class="language-bash">openssl req -new -key example.com.key -out example.com.csr
</code></pre>
<p>This generates a CSR and asks you to provide the Common Name (URL) of your site.</p>
<p>Recently I've noticed that certs from the above CSR don't work.
From some research, Chrome's version 58 (2017) is when it changed, so I guess it took me a while to catch on.</p>
<p>Using the CommonName (CN) instead of SubjectAltName (SAN) in your cert was deprecated in RFC 2818 (forever ago).
For a long time for browsers didn't enforce that requirement and could fall back to CN, but it seems like that allows homograph attacks
(fake "a" in apple.com, for instance), where the SAN field somehow does not.</p>
<p>Even if you add and trust the issuing and root CA certs, Chrome will still throw <code>ERR_CERT_COMMON_NAME_INVALID</code>.</p>
<p>Generating a CSR with a SAN is not intuitive. Unless you want to use heredoc, you need to create a file first.</p>
<ul>
<li>The <code>req_distinguished_name</code> values set the prompts during the CSR questionairre.</li>
<li>The DNS.x values are the SAN entries. If you have one URL, then commonName and DNS.1 will match, and you'll have no others (no DNS.2, ect).</li>
</ul>
<p><code>vi san.ini</code></p>
<pre><code>[ req ]
default_bits       = 2048
distinguished_name = req_distinguished_name
req_extensions     = req_ext

[ req_distinguished_name ]
C = Country Name (2 letter code)
ST = State or Province Name (full name)
L = Locality Name (eg, city)
O = Organization Name (eg, company)
OU = Organization Unit (eg, IT)
CN = Common Name (server FQDN)

[ req_ext ]
subjectAltName = @alt_names

[alt_names]
DNS.1 =
DNS.2 =
DNS.3 =
</code></pre>
<p>Now with this file and your private key in hand, you can create your CSR:</p>
<pre><code class="language-bash"># Set the name of your domain
domain=
# Generate the private key and CSR
openssl req -new -newkey rsa:2048 -nodes -keyout $domain.key -out $domain.csr -config san.ini
</code></pre>
<p>Check your work:</p>
<pre><code class="language-bash">openssl req -in $domain.csr -noout -text
</code></pre>
<p>That's it. Send the CSR off to the CA admin or whoever carves out your certificates. If you're in a corporate environment and need to install that CA's cert on Ubuntu afterward, see
<a href="/ubuntu-trust-corporate-ca.html">Ubuntu: Blindly Trusting the Corporate CA</a>.</p>


<p>Sometimes you want to test a switch's trunk port without having to carve our an access port to plug your MacBook into.</p>
<p>Here's how:</p>
<ol>
<li>Open System Preferences</li>
<li>Click Network</li>
<li>Select your Thunderbolt Ethernet adapter</li>
<li>Click the gear below the sidebar listing the interfaces</li>
<li>Click Manage Virtual Interfaces</li>
<li>Click the + button</li>
<li>New VLAN</li>
<li>Enter a name, such as "VLAN-200"</li>
<li>Enter a VLAN ID in the Tag field, such as 200</li>
<li>Choose Tunderbolt Ethernet for your Interface</li>
<li>Click Create</li>
<li>On the sidebar, your new vlan ("VLAN-200") is now an option. Click it.</li>
<li>Configure as needed.</li>
</ol>


<h1 id="tldr">TL;DR</h1>
<pre><code># replace "import queue" with:
from multiprocessing import Queue as queue
</code></pre>
<hr>
<h1 id="steps-to-fix">Steps to fix</h1>
<p>In Ubuntu 18.04 you'd typically install the openstack CLI client like this:</p>
<pre><code class="language-bash">pip install python-openstackclient
</code></pre>
<p>Sadly, this installs a broken version of the client. If you try and run any command you get a stack trace:</p>
<pre><code class="language-bash">Traceback (most recent call last):
  File "/usr/local/bin/openstack", line 6, in &#x3C;module>
    from openstackclient.shell import main
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", line 24, in &#x3C;module>
    from osc_lib import shell
  File "/usr/local/lib/python2.7/dist-packages/osc_lib/shell.py", line 33, in &#x3C;module>
    from osc_lib.cli import client_config as cloud_config
  File "/usr/local/lib/python2.7/dist-packages/osc_lib/cli/client_config.py", line 18, in &#x3C;module>
    from openstack.config import exceptions as sdk_exceptions
  File "/usr/local/lib/python2.7/dist-packages/openstack/__init__.py", line 16, in &#x3C;module>
    import openstack.config
  File "/usr/local/lib/python2.7/dist-packages/openstack/config/__init__.py", line 17, in &#x3C;module>
    from openstack.config.loader import OpenStackConfig  # noqa
  File "/usr/local/lib/python2.7/dist-packages/openstack/config/loader.py", line 33, in &#x3C;module>
    from openstack.config import cloud_region
  File "/usr/local/lib/python2.7/dist-packages/openstack/config/cloud_region.py", line 44, in &#x3C;module>
    from openstack import proxy
  File "/usr/local/lib/python2.7/dist-packages/openstack/proxy.py", line 24, in &#x3C;module>
    from openstack import resource
  File "/usr/local/lib/python2.7/dist-packages/openstack/resource.py", line 49, in &#x3C;module>
    from openstack import utils
  File "/usr/local/lib/python2.7/dist-packages/openstack/utils.py", line 13, in &#x3C;module>
    import queue
ImportError: No module named queue
</code></pre>
<p>To fix it, replace <code>import queue</code> with <code>from multiprocessing import Queue as queue</code> everywhere that it's called.</p>
<p>In my case:</p>
<ul>
<li><code>usr/local/lib/python2.7/dist-packages/openstack/utils.py</code> - line 13</li>
<li><code>/usr/local/lib/python2.7/dist-packages/openstack/cloud/openstackcloud.py</code> - line 14</li>
</ul>


<h1 id="install-kvm">Install KVM</h1>
<p>Install the packages</p>
<pre><code class="language-bash">apt-get install -y qemu-kvm libvirt-bin bridge-utils
</code></pre>
<p>Package notes from <a href="https://help.ubuntu.com/community/KVM/Installation">ubuntu.com</a>:</p>
<ul>
<li>libvirt-bin provides libvirtd which you need to administer qemu and kvm instances using libvirt</li>
<li>qemu-kvm (kvm in Karmic and earlier) is the backend</li>
<li>ubuntu-vm-builder powerful command line tool for building virtual machines</li>
<li>bridge-utils provides a bridge from your network to the virtual machines</li>
</ul>
<p>Add users to groups</p>
<pre><code class="language-bash">adduser `id -un` libvirt
adduser `id -un` kvm
</code></pre>
<p>Verify install</p>
<pre><code class="language-bash">virsh list --all
</code></pre>
<hr>
<h1 id="networking">Networking</h1>
<h2 id="configure-network-bridge">Configure network bridge</h2>
<p>Your network interfaces should be configured with NetPlan right now.
Since bridge-utils is installed, you can use NetPlan to do the rest.</p>
<p>Before doing this, it's a good idea to double-check your IPMI/IDRAC/ILO is working.</p>
<pre><code class="language-yaml"># Example netplan config defining bridge
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: no
    eno2:
      dhcp4: no
  bridges:
    br0:
      interfaces: [eno2]
      addresses: ["10.1.0.14/24"]
      gateway4: 10.1.0.1
      mtu: 1500
      nameservers:
        addresses: ["10.1.0.154"]
      parameters:
        stp: true
        forward-delay: 4
      dhcp4: no
      dhcp6: no
</code></pre>
<hr>
<h1 id="defining-the-vm-using-virt-install">Defining the VM using virt-install</h1>
<p>Install virt-install - it's kind of big, about 160MB</p>
<pre><code class="language-bash">apt-get install -y virtinst
</code></pre>
<h2 id="optional---list-os-variant-options">Optional - list os-variant options</h2>
<pre><code class="language-bash">apt-get install -y libosinfo-bin

osinfo-query os
</code></pre>
<h2 id="create-the-vm">Create the VM</h2>
<ul>
<li><code>--disk bus=</code>: 'ide', 'scsi', 'usb', 'virtio' or 'xen'.</li>
</ul>
<h3 id="booting-from-an-existing-qcow2-image">Booting from an existing qcow2 image</h3>
<pre><code class="language-bash">virt-install \
  --name MyImportedVM \
  --description "Imported virtual machine" \
  --graphics vnc,listen=0.0.0.0 \
  --noautoconsole \  
  --os-type=linux \
  --memory 8192 \
  --vcpus=4 \
  --disk path=/var/lib/libvirt/images/importedVM.qcow2,bus=virtio\
  --boot hd \
  --network bridge:br0,model=virtio
</code></pre>
<h3 id="building-a-new-windows-vm">Building a new Windows VM</h3>
<p>Windows can be difficult on KVM because it doesn't ship with Virtio drivers. Using the non-virtio
disk and network emulation is comparatively slow. Things are made harder because <code>virt-install</code>
doesn't let you use --cdrom twice. To mount two iso files at once (so virtio drivers can be installed),
use the --disk argument with device=cdrom. Also be sure to use bus=ide so Windows can read it before
installing the virtio drivers.</p>
<p>The virtio drivers can be downloaded <a href="https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso">here</a></p>
<pre><code class="language-bash">virt-install \
  --name Server2019 \
  --description "Windows Server 2019 Template" \
  --os-type=windows \
  --memory 8192 \
  --vcpus=4 \
  --disk path=/var/lib/libvirt/images/windows2019.qcow2,bus=virtio,size=30 \
  --disk /var/lib/libvirt/isos/WindowsServer2019StandardCore_1909.iso,device=cdrom,bus=ide \
  --disk /var/lib/libvirt/isos/virtio-win.iso,device=cdrom,bus=ide \
  --graphics vnc,listen=0.0.0.0 \
  --noautoconsole \
  --network bridge=br0,model=virtio
</code></pre>
<ul>
<li>When installing the OS through your VNC viewer, no drives will show up.</li>
<li>Click the "Load Driver" button, and browse to the virtio iso that was mounted.</li>
<li>Expand the <code>viostor</code> folder and choose your OS. Click the <code>amd64</code> subdirectory. Click OK.</li>
<li>RedHat VirtIO SCSI Controller should show up, click Next</li>
<li>Finish the install</li>
</ul>
<p>Once the install finishes the server will turn off.</p>
<p>Edit the server using <code>virsh edit</code> to remove the disk:</p>
<pre><code class="language-bash">virsh edit Server2019
</code></pre>
<p>Choose your editor of choice (vim, obviously), and remove the boot <code>cdrom</code> disk.
Leave the virtio one, you stil need it. It looks like this:</p>
<pre><code class="language-xml">    &#x3C;disk type='file' device='cdrom'>
      &#x3C;driver name='qemu' type='raw'/>
      &#x3C;source file='/var/lib/libvirt/isos/WindowsServer2019StandardCore_1909.iso'/>
      &#x3C;target dev='hda' bus='ide'/>
      &#x3C;readonly/>
      &#x3C;address type='drive' controller='0' bus='0' target='0' unit='0'/>
    &#x3C;/disk>
</code></pre>
<p>Turn it back on using <code>virsh start</code>:</p>
<pre><code class="language-bash">virsh start Server2019
</code></pre>
<h4 id="fix-the-other-drivers">Fix the other drivers</h4>
<p>The storage driver is now installed, but the network driver isn't there.
Since the VM was launched with <code>--network bridge=br0,model=virtio</code>, the VirtIO
driver is required. Not using the virtio NIC leads to some pretty terrible performance.</p>
<p>Log into the server and open up the device manager.
Navigate to "other devices". Right click the question marked network driver
and update the driver. Select your virtio disk and allow searching subdirectories,
it should find the driver and install it for you.</p>
<p>Do the same for the unidentified PCI device, that will install the VirtIO Baloon driver.</p>
<p>The server is now ready to use, or be made into a template. If you intend to use this server
with OpenStack later, consider installing cloud-init from <a href="https://cloudbase.it/cloudbase-init/">cloudbase-init</a>.</p>
<hr>
<h1 id="accessing-the-vm-through-the-console">Accessing the VM through the console</h1>
<p>When defining the VM, <code>--graphics vnc,listen=0.0.0.0</code> was used. This enabled a VNC server underneath the
VM which can be used to connect in. To find the connection details, run <code>virsh vncdisplay</code>, which will show
the port offset of that VM's VNC console.</p>
<pre><code class="language-bash">virsh vncdisplay Server2019
</code></pre>
<p>The output will be something like <code>:0</code> or <code>:1</code>. Add 5900 to that, and that's your listen port.
If you needed to use a different port, <code>--graphics</code> supports a <code>,port=</code> argument.</p>
<hr>
<h1 id="exporting-to-glance">Exporting to Glance</h1>
<p>If you want to use this file in OpenStack Glance next, you need to convert it. It's not intuitive, but the file looks like it's basically a thick-provisioned qcow2 (somehow). Use qemu-img:</p>
<pre><code class="language-bash"># thin provision the volume
qemu-img convert -f qcow2 -O qcow2 &#x3C;boot-vol> &#x3C;new-file.qcow2>
# upload it as an image
openstack image create --container-format bare --disk-format qcow2 --public --file &#x3C;new-file.qcow2> "&#x3C;image name>"
</code></pre>


<h1 id="regular-tcpdump">Regular TCPDUMP</h1>
<p>Since tcpdump is installed by default, it's the first thing to use</p>
<pre><code class="language-bash"># Only listen on one interface
tcpdump -i &#x3C;interface/any>

# Filter to a host
tcpdump 'host &#x3C;ip address/fqdn>'

# filter to a port
tcpdump 'tcp port 5000'

# combine filters
tcpdump '(tcp port 5000 or tcp port 35357) and host 192.168.0.100'

# print packets in ascii (basically always want this)
tcpdump -A

</code></pre>
<h1 id="using-tshark">Using tshark</h1>
<p>Another option is wireshark's command line, tshark:</p>
<pre><code class="language-bash">apt-get install tshark
</code></pre>
<p>Note that tshark is really slow to get started compared to tcpdump, you need to wait a good while
until it says something like "Capturing on 'eno1'"</p>
<pre><code class="language-bash"># show interfaces
tshark -D

# Capture everything from a given source and interface
tshark -i eno1 -Y "ip.src == 10.1.0.76"

# Capture HTTP to and from a given source
tshark -i eno1 -Y "ip.addr == 10.1.0.76 and http"
</code></pre>
<hr>
<p><strong>See also:</strong> <a href="/ops-cheatsheet.html">Operations Reference Page</a> for more Linux ops
recipes: vim find-and-replace, passwordless sudo, loopback volumes,
journalctl, and bash exit traps.</p>


<h1 id="install--configure-nfs-server">Install &#x26; Configure NFS Server</h1>
<p>These steps were tested on Ubuntu 18.04.</p>
<pre><code class="language-bash"># Install the NFS server package
apt-get install nfs-kernel-server

# Make a directory to publish
mkdir /nfs

# Set permissions
chown nobody:nogroup /nfs
chmod 777 /nfs
</code></pre>
<p>Configure NFS by writing <code>/etc/exports</code>. In this example I'm allowing access
from anyone on the <code>10.10.0.0/16</code> subnet. This is only secure if you trust that
entire subnet and any systems which might be routed to it.</p>
<pre><code class="language-text">/nfs 10.10.0.0/16(rw,sync,no_subtree_check)
</code></pre>
<p>Apply the changes</p>
<pre><code class="language-bash">exportfs -a
systemctl restart nfs-kernel-server
</code></pre>
<h1 id="nfs-client-setup">NFS client setup</h1>
<pre><code class="language-bash"># Install client software
apt-get install nfs-common

# Create mount point
mkdir /nfs

# Mount the NFS share manually
mount 10.10.10.9:/nfs /nfs
</code></pre>


<h1 id="problem-description">Problem Description</h1>
<p>You need to transfer files to a locked down Windows server from a Linux server.
Sure, you could carve out a file-share. There's an easier way for a one-off
transfer.</p>
<p>This is a bit of an edge-case since newer Windows systems come with SCP
installed, but the image I needed to transfer to had it disabled. Older
versions like Server 2012 also don't support SCP without extra software.</p>
<h1 id="hosting-files-with-pythons-web-server">Hosting files with Python's web server</h1>
<p>Make a directory and move your file into it. In my case, I'm transferring the
CloudBase-Init setup package.</p>
<p><strong>Note</strong>: If you use the package itself inside either a python script or
an interactive Python session you can change things like the listen IP and
port. Using <code>python -m</code> is just easier.</p>
<pre><code class="language-bash"># Move the file to a directory where it's the only thing there
mv CloudbaseInitSetup_1_1_0_x64.msi http/

# Change to that directory
cd http

# Start the web server. It will listen on 0.0.0.0:8000 by default.
python -m SimpleHTTPServer
</code></pre>
<h1 id="downloading-the-file">Downloading the file</h1>
<p>Now the file can be downloaded from your Linux server. In your Windows system,
open Internet Explorer and navigate to the Linux IP address. Don't forget to
add the <code>:8000</code> to specify the port. From here you can easily download the
file. Be sure to close the server (<code>CTRL+c</code>) when you're done.</p>


<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#commands">Commands</a>
<ul>
<li><a href="#modify-pool-replica-count">Modify Pool Replica Count</a></li>
<li><a href="#show-current-space-used">Show current space used</a></li>
<li><a href="#show-status">Show status</a></li>
<li><a href="#list-create-and-delete-volumes">List, create and delete volumes</a></li>
<li><a href="#presenting-a-volume-to-this-host">Presenting a volume to this host</a></li>
<li><a href="#snapshots">Snapshots</a></li>
<li><a href="#save-rbd-as-file">Save RBD as file</a></li>
<li><a href="#python-automation">Python Automation</a></li>
</ul>
</li>
</ul>
<h1 id="commands">Commands</h1>
<h2 id="modify-pool-replica-count">Modify Pool Replica Count</h2>
<p>Show and modify the replica count (size) in a given pool.</p>
<p>You can do the same with <code>min_size</code> if needed.</p>
<pre><code class="language-bash"># Show the current replica count
#   ceph osd pool get &#x3C;pool name> size
ceph osd pool get images size

# Set the replica count for a given pool
#   ceph osd pool set &#x3C;pool name> size &#x3C;number of replicas>
ceph osd pool set images size 2
</code></pre>
<h2 id="show-current-space-used">Show current space used</h2>
<pre><code class="language-bash">ceph df
</code></pre>
<h2 id="show-status">Show status</h2>
<pre><code class="language-bash"># Show the status right now
ceph -s

# Show the status and watch it
ceph -w
</code></pre>
<h2 id="list-create-and-delete-volumes">List, create and delete volumes</h2>
<pre><code class="language-bash"># LIST
# rbd ls -p &#x3C;pool name>
rbd ls -p volumes

# CREATE
#   rbd create --size &#x3C;size in gb> &#x3C;pool>/&#x3C;volume id>
rbd create --size 10 volumes/myVolume

# DELETE
#   rbd rm &#x3C;pool>/&#x3C;volume id>
rbd rm volumes/myVolume
</code></pre>
<h2 id="presenting-a-volume-to-this-host">Presenting a volume to this host</h2>
<pre><code class="language-bash"># PRESENT TO HOST
#   rbd map &#x3C;pool>/&#x3C;volume id>
rbd map volumes/myVolume

# LIST MAPPED
rbd showmapped

# REMOVE FROM HOST
#   rbd unmap &#x3C;pool>/&#x3C;volume id>
rbd unmap volumes/myVolume
</code></pre>
<p>Once you've presented the volume you can use it like this:</p>
<pre><code class="language-bash">mkfs.ext4 /dev/rbd0
mkdir /data
mount /dev/rbd0 /data
</code></pre>
<h2 id="snapshots">Snapshots</h2>
<pre><code class="language-bash"># LIST SNAPSHOTS
#   rbd snap ls &#x3C;pool>/&#x3C;volume id>
rbd snap ls volumes/myVolume

# REVERT TO SNAPSHOT
# rbd snap rollback &#x3C;pool>/&#x3C;volume id>@&#x3C;snapshot id>
rbd snap rollback volumes/myVolume@sampleSnapshot

# DELETE SNAPSHOT
#   rbd snap rm &#x3C;pool>/&#x3C;volume id>@&#x3C;snap id>
rbd snap rm volumes/myVolume@sampleSnapshot
</code></pre>
<h2 id="save-rbd-as-file">Save RBD as file</h2>
<pre><code class="language-bash">#   rbd export &#x3C;pool>/&#x3C;volume> &#x3C;filename>
rbd export volumes/myVolume myVolume.raw
</code></pre>
<h2 id="python-automation">Python Automation</h2>
<p>To automate any of the above from Python, the <code>rados</code> and <code>rbd</code> libraries
wrap the same operations. See <a href="/python-ceph.html">Python Ceph Examples</a> for
pool management, volume creation, and snapshot handling via the Python API.</p>


<hr>
<p><strong>Note</strong>: There are a lot of ways to do this, I'm only documenting the approach
I chose. For instance, I'm using the master branch and <code>docs/</code>, but you can use
another branch too.</p>
<hr>
<h1 id="initial-setup">Initial Setup</h1>
<h2 id="cname-file">CNAME file</h2>
<p>Pick a domain name. When I was writing this post, I was setting up the GitHub
Pages site for <a href="http://bwdt.breqwatr.com">Breqwatr Deployment Tool</a>, so I used
<code>bwdt.breqwatr.com</code>.</p>
<p>Create a file in your project's master branch named <code>docs/CNAME</code>, and put your
desired FQDN in that file. Push it to master.</p>
<h2 id="cname-dns-entry">CNAME DNS Entry</h2>
<p>I'm using CloudFlare, so I logged in there and made a new CNAME entry pointing
<code>bwdt.breqwatr.com</code> to <code>breqwatr.github.io</code>.</p>
<h2 id="repository-setup">Repository Setup</h2>
<p>As the repository owner, open the repo up in GitHub and go to the Settings
page. Look for the GitHub Pages option.</p>
<ul>
<li>Set your source to <code>master branch /docs folder</code>.</li>
<li>Choose a theme</li>
<li>Confirm custom domain in the "custom domain" field</li>
</ul>
<hr>
<h1 id="write-the-config-file">Write the config file</h1>
<p>Pull the changes GitHub made to your registry, then edit <code>docs/_config.yml</code>.</p>
<p>Give your page a title and description.</p>
<pre><code class="language-yaml">theme: jekyll-theme-slate
title: Breqwatr Deployment Tool
description: A private cloud deployment and management toolkit
</code></pre>
<hr>
<h1 id="write-content">Write Content</h1>
<p>Create a file named <code>docs/index.md</code> and write your page content here.</p>
<p>GitHub Pages will link this <code>index.md</code> file to the <code>/</code> path of your chosen
domain.</p>
<p>The content is written in Markdown, which is really intuitive and carries over
nicely from any <code>README.md</code> files you'll have written before.</p>
<h2 id="sub-pages">Sub-pages</h2>
<p>GitHub Pages supports breaking your content up into more than one static page.
Create a new content file such as <code>installation.md</code>, and GitHub Pages will
host it with a slug of <code>/installation</code>.</p>
<p>From the index file, link to your new secondary file using Markdown. Here's an
example of linking to an installation guide from a table of contents list.</p>
<pre><code class="language-text">Table of contents

- [Installation](/installation.html)
</code></pre>


<p>This post covers some of the ways you can use Ceph from Python</p>
<p>The two libraries used here:</p>
<ul>
<li><a href="https://docs.ceph.com/docs/master/rados/api/python/">rados</a></li>
<li><a href="https://docs.ceph.com/docs/giant/rbd/librbdpy/">rbd</a></li>
</ul>
<hr>
<h1 id="setup">Setup</h1>
<h2 id="deploy-cephconf">Deploy ceph.conf</h2>
<p>Grab the ceph.conf file from your cluster or make a new valid one and place
it on the filesystem of the server which will execute the Python code.</p>
<p>Also copy the keyring over. Check <a href="https://docs.ceph.com/docs/master/rados/api/python/">here</a>
if you want to use anything other than the admin keyring.</p>
<h2 id="install-dependencies">Install Dependencies</h2>
<p>Ceph doesn't distribute a pure python library. Instead it seems to rely on some
C stuff that needs to be installed, so you need to use apt instead of pip.
This is a pain, since so far as I can tell you can't isolate these dependencies
inside a virtual environment.</p>
<pre><code class="language-bash">apt-get install python-rados
</code></pre>
<hr>
<h1 id="examples">Examples</h1>
<p>Be sure that the rados and rbd libraries, and <code>ceph.conf</code> are deployed first.</p>
<h2 id="instantiate-rados-and-rbd-instances">Instantiate rados and rbd instances</h2>
<p>The <code>cluster</code> and <code>rbd_inst</code> variables will be used through these examples.
The <code>rbd_inst</code> methods will expect <code>ioctx</code> as their first argument.</p>
<pre><code class="language-python">import rbd
import rados

cluster = rados.Rados(conffile='/etc/ceph/ceph.conf')
cluster.connect()

pool_name = 'volumes'
ioctx = cluster.open_ioctx(pool_name)
rbd_inst = rbd.RBD()
</code></pre>
<h1 id="usage-examples">Usage examples</h1>
<pre><code class="language-python"># Get cluster stats
# Returns dict with kb, num_objects, kb_avail, and kb_used
cluster.get_cluster_stats()

# Get cluster health by issuing a command to ceph's monitor
cmd = {"prefix":"status", "format":"json"}
## mon_command returns a tuple of (return code, data/buffer, errors)
command_result = cluster.mon_command(json.dumps(cmd), b'', timeout=5)
status = json.loads(command_result[1])

# Get Pool usage stats
# Returns num_objects_degraded, num_kb, num_wr, num_rd, num_objects_unfound,
#   num_objects_missing_on_primary, num_object_clones, num_objects,
#   num_object_copies, num_bytes, num_rd_kb, num_wr_kb
ioctx.get_stats()

# List pools
cluster.list_pools()

# Create pool
pool_name = 'test_pool'
cluster.create_pool(pool_name)

# Delete pool
cluster.delete_pool(pool_name)


# List rados block devices
rbd_inst.list(ioctx)

# Create block device
rbd_name = 'sample-device'
size_gb = 10
rbd_inst.create(ioctx, rbd_name, size_gb)

# Query a block device (image)
image = rbd.Image(ioctx, rbd_name)

# Resize image
new_size=20
image.resize(new_size)

# Create a snapshot
snap_name = 'sample-snap'
image.create_snap(snap_name)

# List snaps of image
image.list_snaps()

# Revert to snapshot
snap_name = 'sample-snap'
image.rollback_to_snap(snap_name)

# Delete a snapshot
image.remove_snap(snap_name)

# Close an open image (when you're done working with it)
image.close()

# Delete an image/volume. Must not have snapshots. Must be closed.
rbd_inst.remove(ioctx, rbd_name)
</code></pre>
<p>For the CLI equivalents of these operations, see the
<a href="/ceph-reference.html">Ceph Reference Page</a>.</p>


<p>This post is a follow-up to my <a href="/aws-api-gateway-iam.html">authenticated API Gateway post</a>.</p>
<p>I've written a Lambda function and presented it using API Gateway. The API
uses Amazon's AIM keys for authentication. The next step is to write the code
to source control (AWS CodeBuild Git) and have it automatically deploy.</p>
<p><strong>Note</strong>: <a href="https://docs.aws.amazon.com/lambda/latest/dg/build-pipeline.html">This official guide on AWS</a>
is really good, and I copied big parts it. I use Python instead of
JavaScript in this post, and include my own experience while following the
guide.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#create-an-s3-bucket">Create an S3 Bucket</a></li>
<li><a href="#commit-your-function-to-source-control">Commit your function to source control</a></li>
<li><a href="#define-an-aws-sam-template">Define an AWS SAM template</a>
<ul>
<li><a href="#write-the-template-file">Write the template file</a></li>
<li><a href="#test-the-template">Test the template</a></li>
</ul>
</li>
<li><a href="#create-a-codebuild-project">Create a CodeBuild Project</a>
<ul>
<li><a href="#write-the-buildspecyml">Write the buildspec.yml</a></li>
<li><a href="#create-a-role-for-codebuild">Create a role for CodeBuild</a></li>
<li><a href="#define-the-build-project">Define the build project</a></li>
</ul>
</li>
<li><a href="#create-a-pipeline">Create a Pipeline</a>
<ul>
<li><a href="#create-iam-policy-for-codedeploy">Create IAM Policy for CodeDeploy</a></li>
<li><a href="#create-iam-role-for-codedeploy">Create IAM Role for CodeDeploy</a></li>
<li><a href="#create-the-pipeline">Create the Pipeline</a></li>
</ul>
</li>
</ul>
<h1 id="create-an-s3-bucket">Create an S3 Bucket</h1>
<p>Open the <a href="https://s3.console.aws.amazon.com/s3/">S3 Console</a> and create a
bucket to store your build artifacts.</p>
<hr>
<hr>
<h1 id="commit-your-function-to-source-control">Commit your function to source control</h1>
<p>I'm using a dedicated repository on AWS CodeCommit to store the code, and
Python3 for the language. Here's a very simple Lambda function I use for
testing:</p>
<pre><code class="language-python">def lambda_handler(event, context):
    """ run a hello function """
    body = 'NO HTTP METHOD'
    if 'httpMethod' in event and event['httpMethod'] == 'POST':
        request_body = event['body']
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a POST with body: {request_body}'
    elif 'httpMethod' in event and event['httpMethod'] == 'GET':
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a GET'
    return {
        'statusCode': 200,
        'body': json.dumps(body)
    }
</code></pre>
<hr>
<h1 id="define-an-aws-sam-template">Define an AWS SAM template</h1>
<h2 id="write-the-template-file">Write the template file</h2>
<p>This was my first time using AWS SAM, so consider these my observations and not
necessarily fact.</p>
<p>SAM stands for Serverless Application Model.</p>
<ul>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html">SAM Reference Page</a></li>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-api.html">API Event type</a></li>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html">What is SAM</a></li>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-specification.html">SAM Specification</a></li>
</ul>
<p>The fields I've changed from Amazon's example for this post:</p>
<ul>
<li><strong>Description</strong> - string describing the serverless application. This
description will be attached to the CloudFormation stack.</li>
<li><strong>Resources.[resourceName]</strong> - self-defined resource, my function.
The resource is a Lambda function because it's type is
<code>AWS::Serverless::Function</code></li>
<li><strong>Handler</strong> - Which method will run in the function</li>
<li><strong>Runtime</strong> - The language to be used</li>
<li><strong>CodeUri</strong> - Path to my file defining the function</li>
<li><strong>Resource.[resourceName].Events[eventName]</strong> - Name of the event which
triggers the function.</li>
<li><strong>Resource.[resourceName].Events[eventName].path</strong> - path for the URL</li>
</ul>
<pre><code class="language-yaml"># template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Sample Continuously deploy function
Resources:
  TestCicdFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: cicd-test.lambda_handler
      Runtime: python3.8
      CodeUri: ./cicd-test.py
      Events:
        TestCicdAPI:
          Type: Api
          Properties:
            Path: /Test
            Method: GET
</code></pre>
<h2 id="test-the-template">Test the template</h2>
<p>Install the AWS SAM CLI application.</p>
<pre><code class="language-bash">pip install --user aws-sam-cli
</code></pre>
<p>Build the package and upload it to your S3 bucket. From your project
root, set your bucket name then run the following.</p>
<pre><code class="language-bash"># Set the bucket name first
BUCKET=''

aws cloudformation package \
  --template-file template.yml \
  --s3-bucket $BUCKET \
  --output-template-file outputtemplate.yml
</code></pre>
<p>The package command will upload your python file to S3 and output a new
template file with the S3 path replacing the <code>CodeUri</code>.</p>
<pre><code class="language-text">Uploading to &#x3C;ID>  381 / 381.0  (100.00%)
Successfully packaged artifacts and wrote output template to file outputtemplate.yml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /home/vagrant/arcus-lambda/outputtemplate.yml --stack-name &#x3C;YOUR STACK NAME>
</code></pre>
<p>Now deploy it. Since there's no CloudFormation stack created yet, I make a new
one named <code>test-cicd-lambda</code>. Note that in the above command output, the
example they give for deploy is wrong. You need to include the
<code>--capabilities CAPABILITY_IAM</code> argument for this template too.</p>
<pre><code class="language-bash">aws cloudformation deploy \
  --capabilities CAPABILITY_IAM \
  --template-file /home/vagrant/arcus-lambda/outputtemplate.yml \
  --stack-name test-cicd-lambda
</code></pre>
<p>The application is now deployed.</p>
<ul>
<li>Stack is visible in the  <a href="https://ca-central-1.console.aws.amazon.com/cloudformation">CloudFormation console</a></li>
<li>API is created in the <a href="https://ca-central-1.console.aws.amazon.com/apigateway/">API Gateway console</a></li>
<li>Function is created in the <a href="https://ca-central-1.console.aws.amazon.com/lambda/">Lambda console</a>.
The function also has an Application page in the web console now.</li>
</ul>
<hr>
<h1 id="create-a-codebuild-project">Create a CodeBuild Project</h1>
<p>I recently did a whole post on this <a href="/aws-codepipeline-ecr.html">here</a>, so I'm
going to be light on the details. Head over to that post to fill in any blanks.
In that post I was building Docker images and pushing them to ECR. This is
similar, except we push python files to Lambda instead.</p>
<h2 id="write-the-buildspecyml">Write the buildspec.yml</h2>
<p>Put this in the project root, it defines what CodeBuild will do. We'll use an
environment variable for the <code>BUCKET</code> definition.</p>
<p>If you're not sure what to pick for a runtime, check <a href="https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-available.html">this list</a>.</p>
<pre><code class="language-yaml"># buildspec.yml
version: 0.2
phases:
  install:
    runtime-versions:
        python: 3.8
  build:
    commands:
      - aws cloudformation package --template-file template.yml --s3-bucket $BUCKET --output-template-file outputtemplate.yml
artifacts:
  type: zip
  files:
    - template.yml
    - outputtemplate.yml
</code></pre>
<p>Commit the file to source control.</p>
<h2 id="create-a-role-for-codebuild">Create a role for CodeBuild</h2>
<p>I use the role I defined for CodeBuild in my previous post. It's a bit overly
permissive but it works. Other than the policies which get created
automatically, mine has these:</p>
<ul>
<li><code>AWSCodeCommitFullAccess</code></li>
<li><code>AmazonEC2ContainerRegistryFullAccess</code></li>
<li><code>AmazonS3FullAccess</code></li>
<li><code>CloudWatchLogsFullAccess</code></li>
<li><code>AWSCodeBuildAdminAccess</code></li>
</ul>
<h2 id="define-the-build-project">Define the build project</h2>
<ol>
<li>Open the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codebuild/">CodeBuild console</a></li>
<li>Create build project</li>
<li>Name the build and give it a description</li>
<li>Source: Add your repository</li>
<li>Environment
<ol>
<li>Managed Image</li>
<li>Operating system: Ubuntu</li>
<li>Runtimes: Standard</li>
<li>Image: the newest one</li>
<li>Environment type: Linux</li>
<li>Privileged: False
<ol>
<li>Service Role: Existing role - choose the one you made above</li>
</ol>
</li>
<li>Additional Configuration - environment variables
<ul>
<li>name: BUCKET</li>
<li>value: Your bucket name</li>
</ul>
</li>
</ol>
</li>
<li>Buildspec: use buildspec file</li>
<li>Artifacts: no artifacts</li>
<li>Logs: set a group name for CloudWatch</li>
<li>Create build project</li>
</ol>
<p>Test the build before moving on and make sure it works.</p>
<hr>
<h1 id="create-a-pipeline">Create a Pipeline</h1>
<h2 id="create-iam-policy-for-codedeploy">Create IAM Policy for CodeDeploy</h2>
<p>Open the IAM console and create a new policy.
Define the policy using the following JSON.
These wildcards will cover everything you need, though it might be a bit
permissive.</p>
<pre><code class="language-json">{
    "Statement": [
        {
            "Action": [
                "apigateway:*",
                "codedeploy:*",
                "lambda:*",
                "cloudformation:CreateChangeSet",
                "iam:GetRole",
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:PutRolePolicy",
                "iam:AttachRolePolicy",
                "iam:DeleteRolePolicy",
                "iam:DetachRolePolicy",
                "iam:PassRole",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetBucketVersioning"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ],
    "Version": "2012-10-17"
}
</code></pre>
<h2 id="create-iam-role-for-codedeploy">Create IAM Role for CodeDeploy</h2>
<p>In the role wizard of the IAM page:</p>
<ol>
<li>Create role</li>
<li>Trusted Entity: <code>CloudFormation</code></li>
<li>Permissions:
<ol>
<li><code>AWSLambdaExecute</code></li>
<li>Your new policy created above</li>
</ol>
</li>
<li>Role Name: assign a name</li>
</ol>
<h2 id="create-the-pipeline">Create the Pipeline</h2>
<p>When code is committed, you want CodeBuild to run and re-deploy your serverless
application.</p>
<ol>
<li>Open the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codepipeline/pipelines">CodePipeline console</a></li>
<li>Click Create Pipeline</li>
<li>Name the pipeline</li>
<li>Service Role: Create a new service role for the pipeline. AWS will set sane
permissions for it, no special role is needed. I like to have a consistent
naming convention as you end up with a ton of IAM entities after a while.</li>
<li>Role Name: Pick a name</li>
<li>Allow AWS CodePipeline to create a role</li>
<li>Next</li>
<li>Source Provider: AWS CodeCommit</li>
<li>Repository Name: Select the repository name</li>
<li>Branch name: master</li>
<li>Change detection: Amazon CloudWatch Events</li>
<li>Next</li>
<li>Build Provider: AWS CodeBuild</li>
<li>Project Name: Choose the project you just made</li>
<li>Next</li>
<li>Deploy Provider: AWS CloudFormation</li>
<li>Action Mode: Create or replace a change set</li>
<li>Stack Name: choose a name. I used the one from the test above.</li>
<li>Change Set Name: choose a name</li>
<li>Template:
<ul>
<li>Artifact name: <code>BuildArtifact</code></li>
<li>File name: <code>outputtemplate.yml</code></li>
</ul>
</li>
<li>Capabilities: <code>CAPABILITY_IAM</code></li>
<li>Role Name: The role you defined for CodeDeploy above</li>
<li>Next</li>
<li>Create Pipeline</li>
</ol>
<p>That should do it. Now whenever you push code to your master branch, the
serverless application will re-deploy.</p>


<p><strong>This post is linked to from the <a href="/openstack.html">OpenStack Deep Dive Project</a></strong></p>
<hr>
<p>When you launch an instance in OpenStack, you can provide a script to
cloud-init that will be executed at startup time. The glance template needs to
have <a href="https://cloudinit.readthedocs.io/en/latest/">cloud-init</a> installed
(linux) or <a href="https://cloudbase.it/cloudbase-init/">Cloudbase-init</a> (windows).</p>
<h1 id="write-your-powershell-script">Write your Powershell Script</h1>
<p>You need to define the script ahead of time. Here's a super simple script
to add a local administrator:</p>
<pre><code class="language-powershell">#ps1
$name = "MyUser"
$password = "MyPassword"
$password_secure_string = ConvertTo-SecureString -AsPlainText -Force $password
$new_user = New-LocalUser -Name $name -Password $password_secure_string -AccountNeverExpires
Add-LocalGroupMember -Group "Administrators" -Member $new_user
</code></pre>
<hr>
<h1 id="create-the-instance">Create the Instance</h1>
<p>There's more than one way to do just about everything, but here's how I do it.</p>
<pre><code class="language-bash"># source openrc file
source my-openrc.sh

# Collect environment details
openstack flavor list
openstack network list
openstack image list

# Set the VM specs
flavor=""
network=""
image=""
name="CloudInitDemo"
size=60
vol_name="$name-boot"

# Create a boot volume
openstack volume create --image $image --bootable --size $size $vol_name

# create server and specify cloud-init script
script_file="/home/kyle/localAdmin.ps1"
openstack server create \
  --volume $vol_name \
  --flavor $flavor \
  --network $network \
  --user-data $script_file \
  $name
</code></pre>


<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#build-an-api-gateway-resource">Build an API Gateway resource</a>
<ul>
<li><a href="#build-a-lambda-function-for-it-to-run">Build a Lambda function for it to run</a></li>
<li><a href="#define-the-api">Define the API</a></li>
</ul>
</li>
<li><a href="#iam-setup">IAM Setup</a>
<ul>
<li><a href="#create-policy">Create Policy</a></li>
<li><a href="#bind-policy-to-a-user">Bind policy to a user</a></li>
</ul>
</li>
<li><a href="#test-the-api-in-postman">Test the API in Postman</a></li>
<li><a href="#execute-the-api-from-python">Execute the API from Python</a></li>
</ul>
<h1 id="build-an-api-gateway-resource">Build an API Gateway resource</h1>
<h2 id="build-a-lambda-function-for-it-to-run">Build a Lambda function for it to run</h2>
<p>Open the lambda console and create a function. Here's an example python
function that says hello to whoever sent the authenticated request.</p>
<pre><code class="language-python">def lambda_handler(event, context):
    """ run a hello function """
    body = 'NO HTTP METHOD'
    if 'httpMethod' in event and event['httpMethod'] == 'POST':
        request_body = event['body']
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a POST with body: {request_body}'
    elif 'httpMethod' in event and event['httpMethod'] == 'GET':
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a GET'
    return {
        'statusCode': 200,
        'body': json.dumps(body)
    }
</code></pre>
<h2 id="define-the-api">Define the API</h2>
<ol>
<li>Go to the <a href="https://ca-central-1.console.aws.amazon.com/apigateway/">API Gateway Console</a></li>
<li>Create</li>
<li>Protocol: REST</li>
<li>New API</li>
<li>API Name: <code>&#x3C;name></code></li>
<li>Endpoint type: Regional</li>
<li>Create API</li>
</ol>
<p>Create a method:</p>
<ol>
<li>Actions</li>
<li>Create Method</li>
<li>POST</li>
<li>Check mark</li>
<li>Integration Type: Lambda Function</li>
<li>Use Lambda Proxy Integration: Checked</li>
<li>Lambda Region: ca-central-1</li>
<li>Lambda Function: HelloWorld</li>
</ol>
<p>Enable IAM Authorization</p>
<ol>
<li>Click Method Request</li>
<li>Set authorization to <code>AWS_IAM</code></li>
</ol>
<h1 id="iam-setup">IAM Setup</h1>
<h2 id="create-policy">Create Policy</h2>
<p>Collect the ARN of the API Gateway. Click your endpoint, then the method.
The ARN will display under Method Request in the Method Execution diagram.</p>
<p>Navigate to the IAM console and create a new policy.</p>
<pre><code class="language-json">{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "execute-api:Invoke"
            ],
            "Resource": [
								"arn:aws:execute-api:ca-central-1:850047500507:42mydti3lg/*/POST/*"
            ]
        }
    ]
}
</code></pre>
<h2 id="bind-policy-to-a-user">Bind policy to a user</h2>
<p>Here's how I do it:</p>
<ol>
<li>Open the IAM console</li>
<li>If you don't have a group yet, make one</li>
<li>Attach the policy to the group</li>
<li>Attach the group to a user</li>
</ol>
<p>Collect the key and key ID of that user.</p>
<h1 id="test-the-api-in-postman">Test the API in Postman</h1>
<p>Postman is nice since you know it works, and it natively handles building the
AWS request headers. It's the easiest way I know to run a one-off test of your
API.</p>
<p>Install <a href="https://www.getpostman.com/">Postman</a>.</p>
<ol>
<li>New > Request - fill in some names.</li>
<li>Set the request type to POST</li>
<li>Enter the URL</li>
<li>Enter any request data</li>
<li>Go to the Authorization tab and choose AWS signature</li>
<li>Enter the key and key ID from your IAM user</li>
<li>Set the region, in my case ca-central-1</li>
<li>Send</li>
</ol>
<p>You should get a valid response from your API. If you skip any of the
authorization steps you'll get an error instead.</p>
<h1 id="execute-the-api-from-python">Execute the API from Python</h1>
<p>End users don't use postman. Usually they'd be using JavaScript, and in my case
they use Python to access my API. This was <strong>really complicated</strong>.</p>
<p><a href="https://github.com/DavidMuller/aws-requests-auth">Here's an open source library that can do it</a>.</p>
<p>I've written my own version of this for the Breqwatr deployment tool. I didn't
want to depend on the above link. Also I tried to rewrite the client as
functions with the hope of making it easier to read.</p>
<p>You can find it in <a href="https://github.com/breqwatr/breqwatr-deployment-tool">Breqwatr's GitHub</a>
under <code>lib/aws</code>.</p>
<p>Here's the high level flow of how it gets the auth header:</p>
<pre><code class="language-python">def get_authorization_header(time, key_id, secret_key, region, method, host,
                             body, uri, query):
    """ Return an aws authorization header """
    # canonical request is a multi-line string with a particular format
    canonical_headers = get_canonical_headers(host, time)
    payload_hash = get_payload_hash(body)
    canonical_request = get_canonical_request(
         method=method,
         uri=uri,
         query=query,
         canonical_headers=canonical_headers,
         payload_hash=payload_hash)
    # create a string-to-sign from the canonical requests' digest
    cr_digest = hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()
    credential_scope = get_credential_scope(time, region)
    string_to_sign = get_string_to_sign(time, credential_scope, cr_digest)
    # sign the String-To-Sign with a signing key derived from secret iam key
    signing_key = get_signing_key(secret_key, time, region)
    signature = get_signature(signing_key, string_to_sign)
    # return the headers all in one string
    return (
        f'AWS4-HMAC-SHA256 Credential={key_id}/{credential_scope}, '
        f'SignedHeaders=host;x-amz-date, '
        f'Signature={signature}')
</code></pre>


Kyle Pericak

Mermaid Diagrams in Markdown

Adding Virtio drivers for KVM to RHEL's initrd

Creating a CSR with a SAN - openssl

VLAN Tagging Mac Thunderbolt NIC IP Traffic

OpenStack: Fixing the CLI (python2) error: No module named queue

Create Windows KVM VM from Command Line

Packet Tracing Reference

Ubuntu NFS Server Setup

Transferring files to Windows through Python's SimpleHTTPServer

Ceph Reference Page

Github Pages: Basics

Python Ceph Examples

AWS CodeBuild CICD - Deploy to Lambda

OpenStack Cloudbase-init: PowerShell user_data Scripts

Authenticating with AWS IAM in AWS API Gateway