<h1 id="install-kvm">Install KVM</h1>
<p>Install the packages</p>
<pre><code class="language-bash">apt-get install -y qemu-kvm libvirt-bin bridge-utils
</code></pre>
<p>Package notes from <a href="https://help.ubuntu.com/community/KVM/Installation">ubuntu.com</a>:</p>
<ul>
<li>libvirt-bin provides libvirtd which you need to administer qemu and kvm instances using libvirt</li>
<li>qemu-kvm (kvm in Karmic and earlier) is the backend</li>
<li>ubuntu-vm-builder powerful command line tool for building virtual machines</li>
<li>bridge-utils provides a bridge from your network to the virtual machines</li>
</ul>
<p>Add users to groups</p>
<pre><code class="language-bash">adduser `id -un` libvirt
adduser `id -un` kvm
</code></pre>
<p>Verify install</p>
<pre><code class="language-bash">virsh list --all
</code></pre>
<hr>
<h1 id="networking">Networking</h1>
<h2 id="configure-network-bridge">Configure network bridge</h2>
<p>Your network interfaces should be configured with NetPlan right now.
Since bridge-utils is installed, you can use NetPlan to do the rest.</p>
<p>Before doing this, it's a good idea to double-check your IPMI/IDRAC/ILO is working.</p>
<pre><code class="language-yaml"># Example netplan config defining bridge
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: no
    eno2:
      dhcp4: no
  bridges:
    br0:
      interfaces: [eno2]
      addresses: ["10.1.0.14/24"]
      gateway4: 10.1.0.1
      mtu: 1500
      nameservers:
        addresses: ["10.1.0.154"]
      parameters:
        stp: true
        forward-delay: 4
      dhcp4: no
      dhcp6: no
</code></pre>
<hr>
<h1 id="defining-the-vm-using-virt-install">Defining the VM using virt-install</h1>
<p>Install virt-install - it's kind of big, about 160MB</p>
<pre><code class="language-bash">apt-get install -y virtinst
</code></pre>
<h2 id="optional---list-os-variant-options">Optional - list os-variant options</h2>
<pre><code class="language-bash">apt-get install -y libosinfo-bin

osinfo-query os
</code></pre>
<h2 id="create-the-vm">Create the VM</h2>
<ul>
<li><code>--disk bus=</code>: 'ide', 'scsi', 'usb', 'virtio' or 'xen'.</li>
</ul>
<h3 id="booting-from-an-existing-qcow2-image">Booting from an existing qcow2 image</h3>
<pre><code class="language-bash">virt-install \
  --name MyImportedVM \
  --description "Imported virtual machine" \
  --graphics vnc,listen=0.0.0.0 \
  --noautoconsole \  
  --os-type=linux \
  --memory 8192 \
  --vcpus=4 \
  --disk path=/var/lib/libvirt/images/importedVM.qcow2,bus=virtio\
  --boot hd \
  --network bridge:br0,model=virtio
</code></pre>
<h3 id="building-a-new-windows-vm">Building a new Windows VM</h3>
<p>Windows can be difficult on KVM because it doesn't ship with Virtio drivers. Using the non-virtio
disk and network emulation is comparatively slow. Things are made harder because <code>virt-install</code>
doesn't let you use --cdrom twice. To mount two iso files at once (so virtio drivers can be installed),
use the --disk argument with device=cdrom. Also be sure to use bus=ide so Windows can read it before
installing the virtio drivers.</p>
<p>The virtio drivers can be downloaded <a href="https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso">here</a></p>
<pre><code class="language-bash">virt-install \
  --name Server2019 \
  --description "Windows Server 2019 Template" \
  --os-type=windows \
  --memory 8192 \
  --vcpus=4 \
  --disk path=/var/lib/libvirt/images/windows2019.qcow2,bus=virtio,size=30 \
  --disk /var/lib/libvirt/isos/WindowsServer2019StandardCore_1909.iso,device=cdrom,bus=ide \
  --disk /var/lib/libvirt/isos/virtio-win.iso,device=cdrom,bus=ide \
  --graphics vnc,listen=0.0.0.0 \
  --noautoconsole \
  --network bridge=br0,model=virtio
</code></pre>
<ul>
<li>When installing the OS through your VNC viewer, no drives will show up.</li>
<li>Click the "Load Driver" button, and browse to the virtio iso that was mounted.</li>
<li>Expand the <code>viostor</code> folder and choose your OS. Click the <code>amd64</code> subdirectory. Click OK.</li>
<li>RedHat VirtIO SCSI Controller should show up, click Next</li>
<li>Finish the install</li>
</ul>
<p>Once the install finishes the server will turn off.</p>
<p>Edit the server using <code>virsh edit</code> to remove the disk:</p>
<pre><code class="language-bash">virsh edit Server2019
</code></pre>
<p>Choose your editor of choice (vim, obviously), and remove the boot <code>cdrom</code> disk.
Leave the virtio one, you stil need it. It looks like this:</p>
<pre><code class="language-xml">    &#x3C;disk type='file' device='cdrom'>
      &#x3C;driver name='qemu' type='raw'/>
      &#x3C;source file='/var/lib/libvirt/isos/WindowsServer2019StandardCore_1909.iso'/>
      &#x3C;target dev='hda' bus='ide'/>
      &#x3C;readonly/>
      &#x3C;address type='drive' controller='0' bus='0' target='0' unit='0'/>
    &#x3C;/disk>
</code></pre>
<p>Turn it back on using <code>virsh start</code>:</p>
<pre><code class="language-bash">virsh start Server2019
</code></pre>
<h4 id="fix-the-other-drivers">Fix the other drivers</h4>
<p>The storage driver is now installed, but the network driver isn't there.
Since the VM was launched with <code>--network bridge=br0,model=virtio</code>, the VirtIO
driver is required. Not using the virtio NIC leads to some pretty terrible performance.</p>
<p>Log into the server and open up the device manager.
Navigate to "other devices". Right click the question marked network driver
and update the driver. Select your virtio disk and allow searching subdirectories,
it should find the driver and install it for you.</p>
<p>Do the same for the unidentified PCI device, that will install the VirtIO Baloon driver.</p>
<p>The server is now ready to use, or be made into a template. If you intend to use this server
with OpenStack later, consider installing cloud-init from <a href="https://cloudbase.it/cloudbase-init/">cloudbase-init</a>.</p>
<hr>
<h1 id="accessing-the-vm-through-the-console">Accessing the VM through the console</h1>
<p>When defining the VM, <code>--graphics vnc,listen=0.0.0.0</code> was used. This enabled a VNC server underneath the
VM which can be used to connect in. To find the connection details, run <code>virsh vncdisplay</code>, which will show
the port offset of that VM's VNC console.</p>
<pre><code class="language-bash">virsh vncdisplay Server2019
</code></pre>
<p>The output will be something like <code>:0</code> or <code>:1</code>. Add 5900 to that, and that's your listen port.
If you needed to use a different port, <code>--graphics</code> supports a <code>,port=</code> argument.</p>
<hr>
<h1 id="exporting-to-glance">Exporting to Glance</h1>
<p>If you want to use this file in OpenStack Glance next, you need to convert it. It's not intuitive, but the file looks like it's basically a thick-provisioned qcow2 (somehow). Use qemu-img:</p>
<pre><code class="language-bash"># thin provision the volume
qemu-img convert -f qcow2 -O qcow2 &#x3C;boot-vol> &#x3C;new-file.qcow2>
# upload it as an image
openstack image create --container-format bare --disk-format qcow2 --public --file &#x3C;new-file.qcow2> "&#x3C;image name>"
</code></pre>


<h1 id="regular-tcpdump">Regular TCPDUMP</h1>
<p>Since tcpdump is installed by default, it's the first thing to use</p>
<pre><code class="language-bash"># Only listen on one interface
tcpdump -i &#x3C;interface/any>

# Filter to a host
tcpdump 'host &#x3C;ip address/fqdn>'

# filter to a port
tcpdump 'tcp port 5000'

# combine filters
tcpdump '(tcp port 5000 or tcp port 35357) and host 192.168.0.100'

# print packets in ascii (basically always want this)
tcpdump -A

</code></pre>
<h1 id="using-tshark">Using tshark</h1>
<p>Another option is wireshark's command line, tshark:</p>
<pre><code class="language-bash">apt-get install tshark
</code></pre>
<p>Note that tshark is really slow to get started compared to tcpdump, you need to wait a good while
until it says something like "Capturing on 'eno1'"</p>
<pre><code class="language-bash"># show interfaces
tshark -D

# Capture everything from a given source and interface
tshark -i eno1 -Y "ip.src == 10.1.0.76"

# Capture HTTP to and from a given source
tshark -i eno1 -Y "ip.addr == 10.1.0.76 and http"
</code></pre>


<h1 id="install--configure-nfs-server">Install &#x26; Configure NFS Server</h1>
<p>These steps were tested on Ubuntu 18.04.</p>
<pre><code class="language-bash"># Install the NFS server package
apt-get install nfs-kernel-server

# Make a directory to publish
mkdir /nfs

# Set permissions
chown nobody:nogroup /nfs
chmod 777 /nfs
</code></pre>
<p>Configure NFS by writing <code>/etc/exports</code>. In this example I'm allowing access
from anyone on the <code>10.10.0.0/16</code> subnet. This is only secure if you trust that
entire subnet and any systems which might be routed to it.</p>
<pre><code class="language-text">/nfs 10.10.0.0/16(rw,sync,no_subtree_check)
</code></pre>
<p>Apply the changes</p>
<pre><code class="language-bash">exportfs -a
systemctl restart nfs-kernel-server
</code></pre>
<h1 id="nfs-client-setup">NFS client setup</h1>
<pre><code class="language-bash"># Install client software
apt-get install nfs-common

# Create mount point
mkdir /nfs

# Mount the NFS share manually
mount 10.10.10.9:/nfs /nfs
</code></pre>


<h1 id="problem-description">Problem Description</h1>
<p>You need to transfer files to a locked down Windows server from a Linux server.
Sure, you could carve out a file-share. There's an easier way for a one-off
transfer.</p>
<p>This is a bit of an edge-case since newer Windows systems come with SCP
installed, but the image I needed to transfer to had it disabled. Older
versions like Server 2012 also don't support SCP without extra software.</p>
<h1 id="hosting-files-with-pythons-web-server">Hosting files with Python's web server</h1>
<p>Make a directory and move your file into it. In my case, I'm transferring the
CloudBase-Init setup package.</p>
<p><strong>Note</strong>: If you use the package itself inside either a python script or
an interactive Python session you can change things like the listen IP and
port. Using <code>python -m</code> is just easier.</p>
<pre><code class="language-bash"># Move the file to a directory where it's the only thing there
mv CloudbaseInitSetup_1_1_0_x64.msi http/

# Change to that directory
cd http

# Start the web server. It will listen on 0.0.0.0:8000 by default.
python -m SimpleHTTPServer
</code></pre>
<h1 id="downloading-the-file">Downloading the file</h1>
<p>Now the file can be downloaded from your Linux server. In your Windows system,
open Internet Explorer and navigate to the Linux IP address. Don't forget to
add the <code>:8000</code> to specify the port. From here you can easily download the
file. Be sure to close the server (<code>CTRL+c</code>) when you're done.</p>


<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#commands">Commands</a>
<ul>
<li><a href="#modify-pool-replica-count">Modify Pool Replica Count</a></li>
<li><a href="#show-current-space-used">Show current space used</a></li>
<li><a href="#show-status">Show status</a></li>
<li><a href="#list-create-and-delete-volumes">List, create and delete volumes</a></li>
<li><a href="#presenting-a-volume-to-this-host">Presenting a volume to this host</a></li>
<li><a href="#snapshots">Snapshots</a></li>
<li><a href="#save-rbd-as-file">Save RBD as file</a></li>
</ul>
</li>
</ul>
<h1 id="commands">Commands</h1>
<h2 id="modify-pool-replica-count">Modify Pool Replica Count</h2>
<p>Show and modify the replica count (size) in a given pool.</p>
<p>You can do the same with <code>min_size</code> if needed.</p>
<pre><code class="language-bash"># Show the current replica count
#   ceph osd pool get &#x3C;pool name> size
ceph osd pool get images size

# Set the replica count for a given pool
#   ceph osd pool set &#x3C;pool name> size &#x3C;number of replicas>
ceph osd pool set images size 2
</code></pre>
<h2 id="show-current-space-used">Show current space used</h2>
<pre><code class="language-bash">ceph df
</code></pre>
<h2 id="show-status">Show status</h2>
<pre><code class="language-bash"># Show the status right now
ceph -s

# Show the status and watch it
ceph -w
</code></pre>
<h2 id="list-create-and-delete-volumes">List, create and delete volumes</h2>
<pre><code class="language-bash"># LIST
# rbd ls -p &#x3C;pool name>
rbd ls -p volumes

# CREATE
#   rbd create --size &#x3C;size in gb> &#x3C;pool>/&#x3C;volume id>
rbd create --size 10 volumes/myVolume

# DELETE
#   rbd rm &#x3C;pool>/&#x3C;volume id>
rbd rm volumes/myVolume
</code></pre>
<h2 id="presenting-a-volume-to-this-host">Presenting a volume to this host</h2>
<pre><code class="language-bash"># PRESENT TO HOST
#   rbd map &#x3C;pool>/&#x3C;volume id>
rbd map volumes/myVolume

# LIST MAPPED
rbd showmapped

# REMOVE FROM HOST
#   rbd unmap &#x3C;pool>/&#x3C;volume id>
rbd unmap volumes/myVolume
</code></pre>
<p>Once you've presented the volume you can use it like this:</p>
<pre><code class="language-bash">mkfs.ext4 /dev/rbd0
mkdir /data
mount /dev/rbd0 /data
</code></pre>
<h2 id="snapshots">Snapshots</h2>
<pre><code class="language-bash"># LIST SNAPSHOTS
#   rbd snap ls &#x3C;pool>/&#x3C;volume id>
rbd snap ls volumes/myVolume

# REVERT TO SNAPSHOT
# rbd snap rollback &#x3C;pool>/&#x3C;volume id>@&#x3C;snapshot id>
rbd snap rollback volumes/myVolume@sampleSnapshot

# DELETE SNAPSHOT
#   rbd snap rm &#x3C;pool>/&#x3C;volume id>@&#x3C;snap id>
rbd snap rm volumes/myVolume@sampleSnapshot
</code></pre>
<h2 id="save-rbd-as-file">Save RBD as file</h2>
<pre><code class="language-bash">#   rbd export &#x3C;pool>/&#x3C;volume> &#x3C;filename>
rbd export volumes/myVolume myVolume.raw
</code></pre>


<hr>
<p><strong>Note</strong>: There are a lot of ways to do this, I'm only documenting the approach
I chose. For instance, I'm using the master branch and <code>docs/</code>, but you can use
another branch too.</p>
<hr>
<h1 id="initial-setup">Initial Setup</h1>
<h2 id="cname-file">CNAME file</h2>
<p>Pick a domain name. When I was writing this post, I was setting up the GitHub
Pages site for <a href="http://bwdt.breqwatr.com">Breqwatr Deployment Tool</a>, so I used
<code>bwdt.breqwatr.com</code>.</p>
<p>Create a file in your project's master branch named <code>docs/CNAME</code>, and put your
desired FQDN in that file. Push it to master.</p>
<h2 id="cname-dns-entry">CNAME DNS Entry</h2>
<p>I'm using CloudFlare, so I logged in there and made a new CNAME entry pointing
<code>bwdt.breqwatr.com</code> to <code>breqwatr.github.io</code>.</p>
<h2 id="repository-setup">Repository Setup</h2>
<p>As the repository owner, open the repo up in GitHub and go to the Settings
page. Look for the GitHub Pages option.</p>
<ul>
<li>Set your source to <code>master branch /docs folder</code>.</li>
<li>Choose a theme</li>
<li>Confirm custom domain in the "custom domain" field</li>
</ul>
<hr>
<h1 id="write-the-config-file">Write the config file</h1>
<p>Pull the changes GitHub made to your registry, then edit <code>docs/_config.yml</code>.</p>
<p>Give your page a title and description.</p>
<pre><code class="language-yaml">theme: jekyll-theme-slate
title: Breqwatr Deployment Tool
description: A private cloud deployment and management toolkit
</code></pre>
<hr>
<h1 id="write-content">Write Content</h1>
<p>Create a file named <code>docs/index.md</code> and write your page content here.</p>
<p>GitHub Pages will link this <code>index.md</code> file to the <code>/</code> path of your chosen
domain.</p>
<p>The content is written in Markdown, which is really intuitive and carries over
nicely from any <code>README.md</code> files you'll have written before.</p>
<h2 id="sub-pages">Sub-pages</h2>
<p>GitHub Pages supports breaking your content up into more than one static page.
Create a new content file such as <code>installation.md</code>, and GitHub Pages will
host it with a slug of <code>/installation</code>.</p>
<p>From the index file, link to your new secondary file using Markdown. Here's an
example of linking to an installation guide from a table of contents list.</p>
<pre><code class="language-text">Table of contents

- [Installation](/installation.html)
</code></pre>


<p>This post covers some of the ways you can use Ceph from Python</p>
<p>The two libraries used here:</p>
<ul>
<li><a href="https://docs.ceph.com/docs/master/rados/api/python/">rados</a></li>
<li><a href="https://docs.ceph.com/docs/giant/rbd/librbdpy/">rbd</a></li>
</ul>
<hr>
<h1 id="setup">Setup</h1>
<h2 id="deploy-cephconf">Deploy ceph.conf</h2>
<p>Grab the ceph.conf file from your cluster or make a new valid one and place
it on the filesystem of the server which will execute the Python code.</p>
<p>Also copy the keyring over. Check <a href="https://docs.ceph.com/docs/master/rados/api/python/">here</a>
if you want to use anything other than the admin keyring.</p>
<h2 id="install-dependencies">Install Dependencies</h2>
<p>Ceph doesn't distribute a pure python library. Instead it seems to rely on some
C stuff that needs to be installed, so you need to use apt instead of pip.
This is a pain, since so far as I can tell you can't isolate these dependencies
inside a virtual environment.</p>
<pre><code class="language-bash">apt-get install python-rados
</code></pre>
<hr>
<h1 id="examples">Examples</h1>
<p>Be sure that the rados and rbd libraries, and <code>ceph.conf</code> are deployed first.</p>
<h2 id="instantiate-rados-and-rbd-instances">Instantiate rados and rbd instances</h2>
<p>The <code>cluster</code> and <code>rbd_inst</code> variables will be used through these examples.
The <code>rbd_inst</code> methods will expect <code>ioctx</code> as their first argument.</p>
<pre><code class="language-python">import rbd
import rados

cluster = rados.Rados(conffile='/etc/ceph/ceph.conf')
cluster.connect()

pool_name = 'volumes'
ioctx = cluster.open_ioctx(pool_name)
rbd_inst = rbd.RBD()
</code></pre>
<h1 id="usage-examples">Usage examples</h1>
<pre><code class="language-python"># Get cluster stats
# Returns dict with kb, num_objects, kb_avail, and kb_used
cluster.get_cluster_stats()

# Get cluster health by issuing a command to ceph's monitor
cmd = {"prefix":"status", "format":"json"}
## mon_command returns a tuple of (return code, data/buffer, errors)
command_result = cluster.mon_command(json.dumps(cmd), b'', timeout=5)
status = json.loads(command_result[1])

# Get Pool usage stats
# Returns num_objects_degraded, num_kb, num_wr, num_rd, num_objects_unfound,
#   num_objects_missing_on_primary, num_object_clones, num_objects,
#   num_object_copies, num_bytes, num_rd_kb, num_wr_kb
ioctx.get_stats()

# List pools
cluster.list_pools()

# Create pool
pool_name = 'test_pool'
cluster.create_pool(pool_name)

# Delete pool
cluster.delete_pool(pool_name)


# List rados block devices
rbd_inst.list(ioctx)

# Create block device
rbd_name = 'sample-device'
size_gb = 10
rbd_inst.create(ioctx, rbd_name, size_gb)

# Query a block device (image)
image = rbd.Image(ioctx, rbd_name)

# Resize image
new_size=20
image.resize(new_size)

# Create a snapshot
snap_name = 'sample-snap'
image.create_snap(snap_name)

# List snaps of image
image.list_snaps()

# Revert to snapshot
snap_name = 'sample-snap'
image.rollback_to_snap(snap_name)

# Delete a snapshot
image.remove_snap(snap_name)

# Close an open image (when you're done working with it)
image.close()

# Delete an image/volume. Must not have snapshots. Must be closed.
rbd_inst.remove(ioctx, rbd_name)
</code></pre>


<p>This post is a follow-up to my <a href="/aws-api-gateway-iam.html">authenticated API Gateway post</a>.</p>
<p>I've written a Lambda function and presented it using API Gateway. The API
uses Amazon's AIM keys for authentication. The next step is to write the code
to source control (AWS CodeBuild Git) and have it automatically deploy.</p>
<p><strong>Note</strong>: <a href="https://docs.aws.amazon.com/lambda/latest/dg/build-pipeline.html">This official guide on AWS</a>
is really good, and I copied big parts it. I use Python instead of
JavaScript in this post, and include my own experience while following the
guide.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#create-an-s3-bucket">Create an S3 Bucket</a></li>
<li><a href="#commit-your-function-to-source-control">Commit your function to source control</a></li>
<li><a href="#define-an-aws-sam-template">Define an AWS SAM template</a>
<ul>
<li><a href="#write-the-template-file">Write the template file</a></li>
<li><a href="#test-the-template">Test the template</a></li>
</ul>
</li>
<li><a href="#create-a-codebuild-project">Create a CodeBuild Project</a>
<ul>
<li><a href="#write-the-buildspecyml">Write the buildspec.yml</a></li>
<li><a href="#create-a-role-for-codebuild">Create a role for CodeBuild</a></li>
<li><a href="#define-the-build-project">Define the build project</a></li>
</ul>
</li>
<li><a href="#create-a-pipeline">Create a Pipeline</a>
<ul>
<li><a href="#create-iam-policy-for-codedeploy">Create IAM Policy for CodeDeploy</a></li>
<li><a href="#create-iam-role-for-codedeploy">Create IAM Role for CodeDeploy</a></li>
<li><a href="#create-the-pipeline">Create the Pipeline</a></li>
</ul>
</li>
</ul>
<h1 id="create-an-s3-bucket">Create an S3 Bucket</h1>
<p>Open the <a href="https://s3.console.aws.amazon.com/s3/">S3 Console</a> and create a
bucket to store your build artifacts.</p>
<hr>
<hr>
<h1 id="commit-your-function-to-source-control">Commit your function to source control</h1>
<p>I'm using a dedicated repository on AWS CodeCommit to store the code, and
Python3 for the language. Here's a very simple Lambda function I use for
testing:</p>
<pre><code class="language-python">def lambda_handler(event, context):
    """ run a hello function """
    body = 'NO HTTP METHOD'
    if 'httpMethod' in event and event['httpMethod'] == 'POST':
        request_body = event['body']
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a POST with body: {request_body}'
    elif 'httpMethod' in event and event['httpMethod'] == 'GET':
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a GET'
    return {
        'statusCode': 200,
        'body': json.dumps(body)
    }
</code></pre>
<hr>
<h1 id="define-an-aws-sam-template">Define an AWS SAM template</h1>
<h2 id="write-the-template-file">Write the template file</h2>
<p>This was my first time using AWS SAM, so consider these my observations and not
necessarily fact.</p>
<p>SAM stands for Serverless Application Model.</p>
<ul>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html">SAM Reference Page</a></li>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-api.html">API Event type</a></li>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html">What is SAM</a></li>
<li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-specification.html">SAM Specification</a></li>
</ul>
<p>The fields I've changed from Amazon's example for this post:</p>
<ul>
<li><strong>Description</strong> - string describing the serverless application. This
description will be attached to the CloudFormation stack.</li>
<li><strong>Resources.[resourceName]</strong> - self-defined resource, my function.
The resource is a Lambda function because it's type is
<code>AWS::Serverless::Function</code></li>
<li><strong>Handler</strong> - Which method will run in the function</li>
<li><strong>Runtime</strong> - The language to be used</li>
<li><strong>CodeUri</strong> - Path to my file defining the function</li>
<li><strong>Resource.[resourceName].Events[eventName]</strong> - Name of the event which
triggers the function.</li>
<li><strong>Resource.[resourceName].Events[eventName].path</strong> - path for the URL</li>
</ul>
<pre><code class="language-yaml"># template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Sample Continuously deploy function
Resources:
  TestCicdFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: cicd-test.lambda_handler
      Runtime: python3.8
      CodeUri: ./cicd-test.py
      Events:
        TestCicdAPI:
          Type: Api
          Properties:
            Path: /Test
            Method: GET
</code></pre>
<h2 id="test-the-template">Test the template</h2>
<p>Install the AWS SAM CLI application.</p>
<pre><code class="language-bash">pip install --user aws-sam-cli
</code></pre>
<p>Build the package and upload it to your S3 bucket. From your project
root, set your bucket name then run the following.</p>
<pre><code class="language-bash"># Set the bucket name first
BUCKET=''

aws cloudformation package \
  --template-file template.yml \
  --s3-bucket $BUCKET \
  --output-template-file outputtemplate.yml
</code></pre>
<p>The package command will upload your python file to S3 and output a new
template file with the S3 path replacing the <code>CodeUri</code>.</p>
<pre><code class="language-text">Uploading to &#x3C;ID>  381 / 381.0  (100.00%)
Successfully packaged artifacts and wrote output template to file outputtemplate.yml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /home/vagrant/arcus-lambda/outputtemplate.yml --stack-name &#x3C;YOUR STACK NAME>
</code></pre>
<p>Now deploy it. Since there's no CloudFormation stack created yet, I make a new
one named <code>test-cicd-lambda</code>. Note that in the above command output, the
example they give for deploy is wrong. You need to include the
<code>--capabilities CAPABILITY_IAM</code> argument for this template too.</p>
<pre><code class="language-bash">aws cloudformation deploy \
  --capabilities CAPABILITY_IAM \
  --template-file /home/vagrant/arcus-lambda/outputtemplate.yml \
  --stack-name test-cicd-lambda
</code></pre>
<p>The application is now deployed.</p>
<ul>
<li>Stack is visible in the  <a href="https://ca-central-1.console.aws.amazon.com/cloudformation">CloudFormation console</a></li>
<li>API is created in the <a href="https://ca-central-1.console.aws.amazon.com/apigateway/">API Gateway console</a></li>
<li>Function is created in the <a href="https://ca-central-1.console.aws.amazon.com/lambda/">Lambda console</a>.
The function also has an Application page in the web console now.</li>
</ul>
<hr>
<h1 id="create-a-codebuild-project">Create a CodeBuild Project</h1>
<p>I recently did a whole post on this <a href="/aws-codepipeline-ecr.html">here</a>, so I'm
going to be light on the details. Head over to that post to fill in any blanks.
In that post I was building Docker images and pushing them to ECR. This is
similar, except we push python files to Lambda instead.</p>
<h2 id="write-the-buildspecyml">Write the buildspec.yml</h2>
<p>Put this in the project root, it defines what CodeBuild will do. We'll use an
environment variable for the <code>BUCKET</code> definition.</p>
<p>If you're not sure what to pick for a runtime, check <a href="https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-available.html">this list</a>.</p>
<pre><code class="language-yaml"># buildspec.yml
version: 0.2
phases:
  install:
    runtime-versions:
        python: 3.8
  build:
    commands:
      - aws cloudformation package --template-file template.yml --s3-bucket $BUCKET --output-template-file outputtemplate.yml
artifacts:
  type: zip
  files:
    - template.yml
    - outputtemplate.yml
</code></pre>
<p>Commit the file to source control.</p>
<h2 id="create-a-role-for-codebuild">Create a role for CodeBuild</h2>
<p>I use the role I defined for CodeBuild in my previous post. It's a bit overly
permissive but it works. Other than the policies which get created
automatically, mine has these:</p>
<ul>
<li><code>AWSCodeCommitFullAccess</code></li>
<li><code>AmazonEC2ContainerRegistryFullAccess</code></li>
<li><code>AmazonS3FullAccess</code></li>
<li><code>CloudWatchLogsFullAccess</code></li>
<li><code>AWSCodeBuildAdminAccess</code></li>
</ul>
<h2 id="define-the-build-project">Define the build project</h2>
<ol>
<li>Open the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codebuild/">CodeBuild console</a></li>
<li>Create build project</li>
<li>Name the build and give it a description</li>
<li>Source: Add your repository</li>
<li>Environment
<ol>
<li>Managed Image</li>
<li>Operating system: Ubuntu</li>
<li>Runtimes: Standard</li>
<li>Image: the newest one</li>
<li>Environment type: Linux</li>
<li>Privileged: False
<ol>
<li>Service Role: Existing role - choose the one you made above</li>
</ol>
</li>
<li>Additional Configuration - environment variables
<ul>
<li>name: BUCKET</li>
<li>value: Your bucket name</li>
</ul>
</li>
</ol>
</li>
<li>Buildspec: use buildspec file</li>
<li>Artifacts: no artifacts</li>
<li>Logs: set a group name for CloudWatch</li>
<li>Create build project</li>
</ol>
<p>Test the build before moving on and make sure it works.</p>
<hr>
<h1 id="create-a-pipeline">Create a Pipeline</h1>
<h2 id="create-iam-policy-for-codedeploy">Create IAM Policy for CodeDeploy</h2>
<p>Open the IAM console and create a new policy.
Define the policy using the following JSON.
These wildcards will cover everything you need, though it might be a bit
permissive.</p>
<pre><code class="language-json">{
    "Statement": [
        {
            "Action": [
                "apigateway:*",
                "codedeploy:*",
                "lambda:*",
                "cloudformation:CreateChangeSet",
                "iam:GetRole",
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:PutRolePolicy",
                "iam:AttachRolePolicy",
                "iam:DeleteRolePolicy",
                "iam:DetachRolePolicy",
                "iam:PassRole",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetBucketVersioning"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ],
    "Version": "2012-10-17"
}
</code></pre>
<h2 id="create-iam-role-for-codedeploy">Create IAM Role for CodeDeploy</h2>
<p>In the role wizard of the IAM page:</p>
<ol>
<li>Create role</li>
<li>Trusted Entity: <code>CloudFormation</code></li>
<li>Permissions:
<ol>
<li><code>AWSLambdaExecute</code></li>
<li>Your new policy created above</li>
</ol>
</li>
<li>Role Name: assign a name</li>
</ol>
<h2 id="create-the-pipeline">Create the Pipeline</h2>
<p>When code is committed, you want CodeBuild to run and re-deploy your serverless
application.</p>
<ol>
<li>Open the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codepipeline/pipelines">CodePipeline console</a></li>
<li>Click Create Pipeline</li>
<li>Name the pipeline</li>
<li>Service Role: Create a new service role for the pipeline. AWS will set sane
permissions for it, no special role is needed. I like to have a consistent
naming convention as you end up with a ton of IAM entities after a while.</li>
<li>Role Name: Pick a name</li>
<li>Allow AWS CodePipeline to create a role</li>
<li>Next</li>
<li>Source Provider: AWS CodeCommit</li>
<li>Repository Name: Select the repository name</li>
<li>Branch name: master</li>
<li>Change detection: Amazon CloudWatch Events</li>
<li>Next</li>
<li>Build Provider: AWS CodeBuild</li>
<li>Project Name: Choose the project you just made</li>
<li>Next</li>
<li>Deploy Provider: AWS CloudFormation</li>
<li>Action Mode: Create or replace a change set</li>
<li>Stack Name: choose a name. I used the one from the test above.</li>
<li>Change Set Name: choose a name</li>
<li>Template:
<ul>
<li>Artifact name: <code>BuildArtifact</code></li>
<li>File name: <code>outputtemplate.yml</code></li>
</ul>
</li>
<li>Capabilities: <code>CAPABILITY_IAM</code></li>
<li>Role Name: The role you defined for CodeDeploy above</li>
<li>Next</li>
<li>Create Pipeline</li>
</ol>
<p>That should do it. Now whenever you push code to your master branch, the
serverless application will re-deploy.</p>


<p><strong>This post is linked to from the <a href="/openstack.html">OpenStack Deep Dive Project</a></strong></p>
<hr>
<p>When you launch an instance in OpenStack, you can provide a script to
cloud-init that will be executed at startup time. The glance template needs to
have <a href="https://cloudinit.readthedocs.io/en/latest/">cloud-init</a> installed
(linux) or <a href="https://cloudbase.it/cloudbase-init/">Cloudbase-init</a> (windows).</p>
<h1 id="write-your-powershell-script">Write your Powershell Script</h1>
<p>You need to define the script ahead of time. Here's a super simple script
to add a local administrator:</p>
<pre><code class="language-powershell">#ps1
$name = "MyUser"
$password = "MyPassword"
$password_secure_string = ConvertTo-SecureString -AsPlainText -Force $password
$new_user = New-LocalUser -Name $name -Password $password_secure_string -AccountNeverExpires
Add-LocalGroupMember -Group "Administrators" -Member $new_user
</code></pre>
<hr>
<h1 id="create-the-instance">Create the Instance</h1>
<p>There's more than one way to do just about everything, but here's how I do it.</p>
<pre><code class="language-bash"># source openrc file
source my-openrc.sh

# Collect environment details
openstack flavor list
openstack network list
openstack image list

# Set the VM specs
flavor=""
network=""
image=""
name="CloudInitDemo"
size=60
vol_name="$name-boot"

# Create a boot volume
openstack volume create --image $image --bootable --size $size $vol_name

# create server and specify cloud-init script
script_file="/home/kyle/localAdmin.ps1"
openstack server create \
  --volume $vol_name \
  --flavor $flavor \
  --network $network \
  --user-data $script_file \
  $name
</code></pre>


<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#build-an-api-gateway-resource">Build an API Gateway resource</a>
<ul>
<li><a href="#build-a-lambda-function-for-it-to-run">Build a Lambda function for it to run</a></li>
<li><a href="#define-the-api">Define the API</a></li>
</ul>
</li>
<li><a href="#iam-setup">IAM Setup</a>
<ul>
<li><a href="#create-policy">Create Policy</a></li>
<li><a href="#bind-policy-to-a-user">Bind policy to a user</a></li>
</ul>
</li>
<li><a href="#test-the-api-in-postman">Test the API in Postman</a></li>
<li><a href="#execute-the-api-from-python">Execute the API from Python</a></li>
</ul>
<h1 id="build-an-api-gateway-resource">Build an API Gateway resource</h1>
<h2 id="build-a-lambda-function-for-it-to-run">Build a Lambda function for it to run</h2>
<p>Open the lambda console and create a function. Here's an example python
function that says hello to whoever sent the authenticated request.</p>
<pre><code class="language-python">def lambda_handler(event, context):
    """ run a hello function """
    body = 'NO HTTP METHOD'
    if 'httpMethod' in event and event['httpMethod'] == 'POST':
        request_body = event['body']
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a POST with body: {request_body}'
    elif 'httpMethod' in event and event['httpMethod'] == 'GET':
        request_user = event['requestContext']['identity']['userArn']
        body = f'{request_user} sent a GET'
    return {
        'statusCode': 200,
        'body': json.dumps(body)
    }
</code></pre>
<h2 id="define-the-api">Define the API</h2>
<ol>
<li>Go to the <a href="https://ca-central-1.console.aws.amazon.com/apigateway/">API Gateway Console</a></li>
<li>Create</li>
<li>Protocol: REST</li>
<li>New API</li>
<li>API Name: <code>&#x3C;name></code></li>
<li>Endpoint type: Regional</li>
<li>Create API</li>
</ol>
<p>Create a method:</p>
<ol>
<li>Actions</li>
<li>Create Method</li>
<li>POST</li>
<li>Check mark</li>
<li>Integration Type: Lambda Function</li>
<li>Use Lambda Proxy Integration: Checked</li>
<li>Lambda Region: ca-central-1</li>
<li>Lambda Function: HelloWorld</li>
</ol>
<p>Enable IAM Authorization</p>
<ol>
<li>Click Method Request</li>
<li>Set authorization to <code>AWS_IAM</code></li>
</ol>
<h1 id="iam-setup">IAM Setup</h1>
<h2 id="create-policy">Create Policy</h2>
<p>Collect the ARN of the API Gateway. Click your endpoint, then the method.
The ARN will display under Method Request in the Method Execution diagram.</p>
<p>Navigate to the IAM console and create a new policy.</p>
<pre><code class="language-json">{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "execute-api:Invoke"
            ],
            "Resource": [
								"arn:aws:execute-api:ca-central-1:850047500507:42mydti3lg/*/POST/*"
            ]
        }
    ]
}
</code></pre>
<h2 id="bind-policy-to-a-user">Bind policy to a user</h2>
<p>Here's how I do it:</p>
<ol>
<li>Open the IAM console</li>
<li>If you don't have a group yet, make one</li>
<li>Attach the policy to the group</li>
<li>Attach the group to a user</li>
</ol>
<p>Collect the key and key ID of that user.</p>
<h1 id="test-the-api-in-postman">Test the API in Postman</h1>
<p>Postman is nice since you know it works, and it natively handles building the
AWS request headers. It's the easiest way I know to run a one-off test of your
API.</p>
<p>Install <a href="https://www.getpostman.com/">Postman</a>.</p>
<ol>
<li>New > Request - fill in some names.</li>
<li>Set the request type to POST</li>
<li>Enter the URL</li>
<li>Enter any request data</li>
<li>Go to the Authorization tab and choose AWS signature</li>
<li>Enter the key and key ID from your IAM user</li>
<li>Set the region, in my case ca-central-1</li>
<li>Send</li>
</ol>
<p>You should get a valid response from your API. If you skip any of the
authorization steps you'll get an error instead.</p>
<h1 id="execute-the-api-from-python">Execute the API from Python</h1>
<p>End users don't use postman. Usually they'd be using JavaScript, and in my case
they use Python to access my API. This was <strong>really complicated</strong>.</p>
<p><a href="https://github.com/DavidMuller/aws-requests-auth">Here's an open source library that can do it</a>.</p>
<p>I've written my own version of this for the Breqwatr deployment tool. I didn't
want to depend on the above link. Also I tried to rewrite the client as
functions with the hope of making it easier to read.</p>
<p>You can find it in <a href="https://github.com/breqwatr/breqwatr-deployment-tool">Breqwatr's GitHub</a>
under <code>lib/aws</code>.</p>
<p>Here's the high level flow of how it gets the auth header:</p>
<pre><code class="language-python">def get_authorization_header(time, key_id, secret_key, region, method, host,
                             body, uri, query):
    """ Return an aws authorization header """
    # canonical request is a multi-line string with a particular format
    canonical_headers = get_canonical_headers(host, time)
    payload_hash = get_payload_hash(body)
    canonical_request = get_canonical_request(
         method=method,
         uri=uri,
         query=query,
         canonical_headers=canonical_headers,
         payload_hash=payload_hash)
    # create a string-to-sign from the canonical requests' digest
    cr_digest = hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()
    credential_scope = get_credential_scope(time, region)
    string_to_sign = get_string_to_sign(time, credential_scope, cr_digest)
    # sign the String-To-Sign with a signing key derived from secret iam key
    signing_key = get_signing_key(secret_key, time, region)
    signature = get_signature(signing_key, string_to_sign)
    # return the headers all in one string
    return (
        f'AWS4-HMAC-SHA256 Credential={key_id}/{credential_scope}, '
        f'SignedHeaders=host;x-amz-date, '
        f'Signature={signature}')
</code></pre>


<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#create-ci-token">Create CI token</a></li>
<li><a href="#add-token-as-circleci-env-var">Add token as CircleCI env var</a></li>
<li><a href="#write-circleciconfigyml">Write .circleci/config.yml</a>
<ul>
<li><a href="#example-file">Example File</a></li>
<li><a href="#locally-validating-your-config">Locally validating your config</a></li>
</ul>
</li>
</ul>
<h1 id="create-ci-token">Create CI token</h1>
<p>This will do the usual browser-based login system, so if you're using a VM then
be sure your lolcahost:9005 goes to the server's <ip>:9005.</p>
<pre><code class="language-bash">firebase login:ci
</code></pre>
<h1 id="add-token-as-circleci-env-var">Add token as CircleCI env var</h1>
<ol>
<li>Log into CircleCI</li>
<li>Choose your GitHub project</li>
<li>Use the Hello World build template</li>
<li>Assign that token as <code>$FIREBASE_DEPLOY_TOKEN</code> in the project's environment
variable settings.
<ol>
<li>Navigate to your project in CircleCI</li>
<li>Click the gear on the top-right to open this project's settings</li>
<li>Open environment variables page</li>
<li>Add Variable. Name: <code>FIREBASE_DEPLOY_TOKEN</code>, value is your token.</li>
<li>Add another variable for <code>FIREBASE_PROJECT</code> and set it to your Firebase
project's name.</li>
</ol>
</li>
</ol>
<h1 id="write-circleciconfigyml">Write .circleci/config.yml</h1>
<p>I found <a href="https://medium.com/static-void-academy/easy-peasy-ci-cd-w-circleci-282bc85ddcf5">this guide</a>
useful.</p>
<h2 id="example-file">Example File</h2>
<pre><code class="language-yaml">version: 2
jobs:
  build:
    docker:
      # Using latest can cause failures, but I hope it will help me keep up
      - image: circleci/node:latest
    steps:
      - checkout
      - run: npm install --prefix=./firebase-deploy firebase-tools
      - run: >
          ./firebase-deploy/node_modules/.bin/firebase deploy
          --project=$FIREBASE_PROJECT
          --token=$FIREBASE_DEPLOY_TOKEN
</code></pre>
<h2 id="locally-validating-your-config">Locally validating your config</h2>
<p>You can use CircleCI's local-ci. Note that this was tested on a
<a href="https://github.com/kylep/dev-vm">dev Ubuntu server</a>.</p>
<p><strong>Install</strong> - <a href="https://circleci.com/docs/2.0/local-cli/#installation">official docs</a></p>
<pre><code class="language-bash">snap install docker circleci
snap connect circleci:docker docker
</code></pre>
<p>Test your configuration file. Navigate to your project directory first.</p>
<pre><code class="language-bash">circleci config validate
# or
circleci config validate &#x3C;config file path>
</code></pre>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<h1 id="workflow">Workflow</h1>
<ol>
<li>Commit code to AWS CodeCommit</li>
<li>CloudWatch rule triggers CodePipeline</li>
<li>CodePipeline uses CodeBuild to build the Docker image</li>
<li>CodePipeline pushes the image to ECR</li>
</ol>
<hr>
<h1 id="deploy-code-to-codecommit">Deploy Code to CodeCommit</h1>
<ol>
<li>Open <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codecommit/repositories">CodeCommit</a>.</li>
<li>Create a repository</li>
<li>Go to your IAM page and edit your user.
<ol>
<li>Look for "HTTPS Git credentials for AWS CodeCommit"</li>
<li>click Generate Credentials</li>
<li>Save your credentials.</li>
</ol>
</li>
<li>Grab the HTTPS URL for your repository and <code>git clone</code> it. Use your new
credentials.</li>
<li>Commit your code (including a Dockerfile) to the CodeCommit repository.</li>
</ol>
<hr>
<h1 id="write-buildspecyml">Write buildspec.yml</h1>
<p>AWS CodeBuild will look for a <code>buildspec.yml</code> file to dictate its actions.
This is where we define how the image will be built. Put it in your CodeCommit
project's root directory.</p>
<p>Here's mine. This will build a docker image and tag it with a timestamp, then
push it to ECR as the timestamp and latest. The variables are defined inside
the build as Environment Variables.</p>
<pre><code class="language-yml"># buildspec.yml
version: 0.2

phases:
  install:
    runtime-versions:
      docker: 18
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - $(aws ecr get-login --no-include-email --region ca-central-1)
      - IMAGE_TAG=$(date +%Y.%m.%d.%I.%M)
  build:
    commands:
      - echo Build started on `date`
      - echo "Building the Docker image."
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest
</code></pre>
<hr>
<h1 id="define-the-pipeline">Define the Pipeline</h1>
<h2 id="cloud-build-role">Cloud Build Role</h2>
<p><strong>Bad Practice Alert</strong>: These steps don't apply the principal of least
privilege. You should lock down your role to be allowed to do only the minimum
that it needs to do.</p>
<ol>
<li>In the IAM page, create a new role.</li>
<li>It will be used by CodeBuild.</li>
<li>Assing the following policies (again, more than it needs)
<ul>
<li>CloudWatchLogsFullAccess</li>
<li>AWSCodeBuildAdminAccess</li>
<li>AmazonS3FullAccess</li>
<li>AmazonEC2ContainerRegistryFullAccess</li>
<li>AWSCodeCommitFullAccess</li>
</ul>
</li>
<li>Name: Assign a name, such as <code>CloudBuild-Admin</code></li>
</ol>
<h2 id="define-cloudbuild-project">Define CloudBuild project</h2>
<ol>
<li>Navigate to the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codebuild/projects">CloudBuild Console</a></li>
<li>Create build project</li>
<li>No build badge</li>
<li>Source provider: CodeCommit</li>
<li>Choose your repository</li>
<li>Choose your branch</li>
<li>Managed Image</li>
<li>Operating System: Ubuntu</li>
<li>Runtime: Standard</li>
<li>Image: aws/codebuild/standard:3.0</li>
<li>Image version: latest</li>
<li>Environment type: Linux</li>
<li>Privileged: true</li>
<li>Service Role: Existing service role. Use the one you just made.</li>
<li>Allow CodeBuild to modify this role: False (we gave it admin)</li>
<li>Additional Configuration - Environment variables (click Add to get more)
- IMAGE_REPO_NAME: (your image name)
- AWS_ACCOUNT_ID: (your numeric AWS account iD)
- AWS_DEFAULT_REGION: ca-central-1</li>
<li>Buildspec: Use a buildspec file</li>
<li>Artifacts: No artifacts</li>
<li>Logs: Give a group and stream name for CloudWatch</li>
<li>Create Build Project</li>
</ol>
<p>Now test the build by clicking "Start build". It should create your image.
You'll need to disable artifacts.</p>
<p>Once the build works and your image is pushed successfully to ECR, you can move
on to setting up a pipeline so this happens automatically.</p>
<h2 id="create-pipeline-role">Create Pipeline Role</h2>
<p><strong>Bad Practice Alert</strong>: These steps don't apply the principal of least
privilege. You should lock down your role to be allowed to do only the minimum
that it needs to do.</p>
<ol>
<li>In the IAM page, create a new role.</li>
<li>It will be used by CodeBuild.</li>
<li>Assing the following policies (again, more than it needs)
<ul>
<li>CloudWatchLogsFullAccess</li>
<li>AWSCodeBuildAdminAccess</li>
<li>AmazonS3FullAccess</li>
<li>AmazonEC2ContainerRegistryFullAccess</li>
<li>AWSCodeCommitFullAccess</li>
</ul>
</li>
<li>Name: Assign a name, such as <code>CloudBuild-Admin</code></li>
</ol>
<h2 id="define-the-pipeline-1">Define the pipeline</h2>
<ol>
<li>Open the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codepipeline/pipelines">CodePipeline console</a></li>
<li>Click Create Pipeline</li>
<li>Name the pipeline</li>
<li>Service Role: New service role</li>
<li>Role Name: Pick a name</li>
<li>Allow AWS CodePipeline to create a role</li>
<li>Next</li>
<li>Source Provider: AWS CodeCommit</li>
<li>Repository Name: Select the repository name</li>
<li>Branch name: master</li>
<li>Change detection: Amazon CloudWatch Events</li>
<li>Next</li>
<li>Build Provider: AWS CodeBuild</li>
<li>Region: Canada (Central)</li>
<li>Project Name: Create Project
<ol>
<li>Project Name: Choose a name</li>
<li>Description: Add a description</li>
<li>Environment image: Managed Image</li>
<li>Operating System: Ubuntu</li>
<li>Runtimes: Standard</li>
<li>Image: aws/codebuild/standard:3.0</li>
<li>Image version: Always use the latest</li>
<li>Environment type: Linux</li>
<li>Privileged: True</li>
<li>Service Role: New service role</li>
<li>Role Name: default name is fine</li>
<li>Advanced Configuration - Environment variables (click Add to get more)
<ul>
<li>IMAGE_REPO_NAME: (your image name)</li>
<li>AWS_ACCOUNT_ID: (your numeric AWS account iD)</li>
<li>AWS_DEFAULT_REGION: ca-central-1</li>
</ul>
</li>
<li>Build specifications: Use a buildspec file</li>
<li>Buildspec name: Leave it blank so it uses buildspec.yml</li>
<li>CloudWatch logs: True</li>
<li>Group name: pick a name</li>
<li>Stream name: pick a name</li>
<li>S3 logs: false</li>
<li>Continue to CodePipeline</li>
</ol>
</li>
<li>Next</li>
<li>Skip deploy stage, pushing to ECR is enough</li>
<li>Create pipeline</li>
</ol>
<p>This will re-run the build that you tested before. It should pass again.</p>


<h1 id="getting-the-certificate">Getting the certificate</h1>
<p>So far as I can tell, certbot works in the following way:</p>
<ol>
<li>Certbot is executed from the web-server CLI, and told to download a cert for
an FQDN. Let's say it's getting a cert from <code>example.com</code>.</li>
<li>Certbot opens up a port, I think 443, and listens for requests from the
LetsEncrypt service.</li>
<li>Certbot calls out to LetsEncrypt's service, letting the service know it's
listening on <code>example.com</code> and ready to prove it owns that domain.</li>
<li>LetsEncrypt sends some secret, or maybe collects some secret from the
listening certbot service, proving that certbot's server does indeed own
this domain name.</li>
<li>LetsEncrypt signs a cert and makes it available. Certbot downloads the cert
to the local server.</li>
</ol>
<h2 id="configure-dns">Configure DNS</h2>
<p>First, ensure that your DNS A record, such as <code>example.com</code>, resolves to
the IP address your server is using. This needs to be an internet accessible
IP address, not an internal RFC-1918 address.</p>
<h2 id="install-certbot">Install Certbot</h2>
<p>This guide assumes you're using Ubuntu Server.</p>
<pre><code class="language-bash">add-apt-repository ppa:certbot/certbot
apt-get update
apt-get install certbot
</code></pre>
<h2 id="download-the-certificate">Download the certificate</h2>
<p>Set the <code>site</code> variable here to your DNS entry which will point to this server.</p>
<pre><code class="language-bash">site="example.com"
certbot certonly --standalone --preferred-challenges http -d $site
</code></pre>
<h2 id="use-the-certificate">Use the certificate</h2>
<p>You can find the certificates as files on your server.</p>
<pre><code class="language-bash">> cd /etc/letsencrypt/live/$site
README  cert.pem  chain.pem  fullchain.pem  privkey.pem
</code></pre>
<p>Copy these certs to your web service and use them like any other cert.</p>
<p>Note that if you want to use these certs for HAProxy, you need to combine
<code>cert.pem</code> and <code>privkey.pem</code> into a single file.</p>
<pre><code class="language-bash">cat cert.pem privkey.pem > haproxy-$site.pem
</code></pre>


<h1 id="create-a-firebase-project">Create a Firebase Project</h1>
<p>Open the <a href="https://console.firebase.google.com/">Firebase console</a> and follow
the wizard. It's pretty straightforward. I bound my project to a Google Cloud
Platform project I was already using and used the Blaze plan.</p>
<p>The wizard is intuitive, I won't cover its steps here.</p>
<p>Navigate into your project and "Add Firebase to your web app". Once that's done
you can use the CLI to deploy the app.</p>
<hr>
<h1 id="install-firebase-cli">Install Firebase CLI</h1>
<p>Installing Firebase CLI requires the <code>npm</code> command from NodeJS.
If you want to configure your dev environment in Ansible, check out my
<a href="http://localhost:8000/nodejs-ansible.html">Installing Node.js with Ansible</a>
post.</p>
<pre><code class="language-bash"># Add the key for the nodesource Apt repository
wget -qO https://deb.nodesource.com/gpgkey/nodesource.gpg.key | sudo apt-key add -

# Add the nodesource apt repo
echo "deb https://deb.nodesource.com/node_13.x bionic main" > /etc/apt/sources.d/nodesource
apt-get update

# Install NodeJS to get the npm command
apt-get install -y nodejs

# Install firebase-tools
npm install firebase-tools
</code></pre>
<h2 id="logging-in">Logging In</h2>
<p>If your CLI and browser are on the same system, just run <code>firebase login</code> and
ignore the rest of this section.</p>
<p>I haven't found a way to log into Firebase CLI that doesn't need a web browser.
The limitation was inconvenient since I develop inside an Ubuntu server VM.</p>
<p>The options I figure would work are:</p>
<ol>
<li>Using a local Vagrant VM, forward the port that the oauth site tries to
redirect to on localhost. This was the easiest for my local dev environment,
so it's what I did. You copy the URL from your CLI into your workstation's
browser, then have it redirect <code>http://localhost:9005/</code> to your VM. Here's
an example of how to forward the port in a Vagrantfile:
<code>config.vm.network :forwarded_port, host: 9005, guest: 9005</code></li>
<li>Install Chrome in Ubuntu server then launch it with
<a href="/x11-forwarding-ubuntu.html">X11 Forwarding</a></li>
<li>Create a reverse SSH tunnel from a port on your laptop to the development
server. <a href="/dial-home-device.html">Here's a post where I've done SSH port forwarding</a></li>
<li>Use the <code>firebase login:ci</code> command and the
<code>--token</code> argument. This option was my fallback, but you still need to
install <code>firebase-cli</code> on your actual workstation with a browser to use it.</li>
</ol>
<p>Once successfully authenticated your browser will show a message saying
<code>Woohoo! Firebase CLI Login Successful</code>.</p>
<hr>
<h1 id="deploy-hello-world">Deploy Hello World</h1>
<p>Make an index file in a directory for this app</p>
<pre><code class="language-bash">mkdir app
cd app
mkdir public
echo "&#x3C;h1>Hello World!&#x3C;/h1>" > public/index.html
</code></pre>
<h2 id="firebase-init">firebase init</h2>
<p>Initialize the directory as an application. This will present you with an
intuitive wizard.</p>
<pre><code class="language-bash">firebase init

# When prompted,
#  choose only Hosting
#  keep your index file
#  decide if you'll use a single page app
</code></pre>
<h3 id="single-page-app-setting">Single-page app setting</h3>
<p>Running <code>firebase init</code> generates a <code>firebase.json</code> file for your project.
In single-page mode, this file will rewrite all traffic to index.html. You can
edit the file to change this behaviour. The rewrite block in <code>"hosting":</code> looks
like this:</p>
<pre><code class="language-json">    "rewrites": [{
      "source": "**",
      "destination": "/index.html"
    }],
</code></pre>
<h2 id="firebase-deploy">firebase deploy</h2>
<p>Deploy the default app</p>
<pre><code class="language-bash">firebase deploy
</code></pre>
<p>The deploy command will print a Hosting URL. The Hello World site can be loaded
from there.</p>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<h1 id="create--terminate-an-ec2-instance-from-python">Create &#x26; Terminate an EC2 Instance from Python</h1>
<p>The IAM user tested for this task has the <code>AmazonEC2FullAccess</code> policy.</p>
<h2 id="create">Create</h2>
<p>Before you begin, determine the following:</p>
<ul>
<li>Image AMI: The example I provide here works, it's for Ubuntu 18.04 in Canada</li>
<li>Security group name</li>
<li>Instance name</li>
<li>Instance type</li>
</ul>
<pre><code class="language-python">import boto3

key_id = 'XXXXXXXXXXXXXX'
key_secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXX'

session = boto3.Session(
    aws_access_key_id=key_id,
    aws_secret_access_key=key_secret)

# Define the VM properties
image_id='ami-0d0eaed20348a3389' # bionic ca-central LTS
instance_name='Test9'
security_group_name = 'SupportService'
instance_type = 't2.micro'

# Put the properties into the expected format
tag_spec = [{
    'ResourceType': 'instance',
    'Tags': [{
        'Key': 'Name',
        'Value': instance_name }]}]
security_groups=[security_group_name]

# Reserve the instance
ec2_client = session.client('ec2')
reservation = ec2_client.run_instances(
        ImageId=image_id,
        MinCount=1,
        MaxCount=1,
        InstanceType=instance_type,
        KeyName='bwsupport',
        TagSpecifications=tag_spec,
        SecurityGroups=security_groups )

# Now the VM is creating. Optionally you can wait and inspect the instace
instance_data = reservation['Instances'][0]
iid = instance_data['InstanceId']
print(f'Reserved instance "{instance_name}" as {iid}. Waiting until running.')
ec2_resource = session.resource('ec2')
instance = ec2_resource.Instance(iid)
instance.wait_until_running()
state = instance.state['Name']
print(f'Instancei {iid} is {state} - Public IP: {instance.public_ip_address}')
</code></pre>
<h2 id="terminate">Terminate</h2>
<p>You can use the same code as above to get the <code>instance</code> variable.
From there, it's this easy:</p>
<pre><code class="language-python">instance.terminate()
instance.wait_until_terminated()
</code></pre>


Kyle Pericak

"It works in my environment"

Categories

Tags