<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to restrict access to a Lambda function using AWS IAM
roles. It builds on the function written <a href="/aws-lambda.html">in my last post</a></p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#create-a-policy-for-your-lambda-function">Create a policy for your lambda function</a>
<ul>
<li><a href="#get-the-arn-of-your-function">Get the ARN of your function</a></li>
<li><a href="#create-the-policy">Create the policy</a></li>
</ul>
</li>
<li><a href="#invoke-lambda-from-python">Invoke Lambda from Python</a></li>
</ul>
<h1 id="create-a-policy-for-your-lambda-function">Create a policy for your lambda function</h1>
<h2 id="get-the-arn-of-your-function">Get the ARN of your function</h2>
<p>Navigate to your function in the AWS Lambda Console. On the top right, the ARN
is displayed. Copy it, you'll use it in the policy.</p>
<h2 id="create-the-policy">Create the policy</h2>
<p>Open the IAM Console and create a new policy.
The JSON looks like this. Note the ARN copied from the Function.</p>
<pre><code class="language-JSON">{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LambdaInvokeHelloWorld",
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction",
                "lambda:InvokeAsync"
            ],
            "Resource": "arn:aws:lambda:ca-central-1:850047500507:function:HelloWorld"
        }
    ]
}
</code></pre>
<p>Name the policy and assign it to your user. This user should have a Key ID and
Secret Key.</p>
<hr>
<h1 id="invoke-lambda-from-python">Invoke Lambda from Python</h1>
<p>Note that the payloads are in byte format.</p>
<pre><code class="language-python">import json
import boto3

key_id = 'XXXXXXXXXXXXXXXXX'
key_secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

session = boto3.Session(
    aws_access_key_id=key_id,
    aws_secret_access_key=key_secret)

client = session.client('lambda')

body = {'name': 'Kyle'}
payload = str.encode(json.dumps(body))
response = client.invoke(FunctionName='HelloWorld', Payload=payload)
print(response['Payload'].read())
</code></pre>
<p>If the user identified by that key has the above permission, they'll be able
to execute this function. Otherwise they'll get a nice permissions error.</p>


<p>This post covers the basics of using AWS Lambda functions.</p>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#pricing">Pricing</a></li>
<li><a href="#hello-world-lambda-function">Hello World Lambda Function</a>
<ul>
<li><a href="#create-the-function">Create the function</a></li>
<li><a href="#define-a-test-event">Define a test event</a></li>
<li><a href="#write-the-function-code">Write the function code</a></li>
<li><a href="#define-a-trigger">Define a trigger</a>
<ul>
<li><a href="#test-the-api">Test the API</a></li>
</ul>
</li>
<li><a href="#what-are-layers">...What are layers?</a></li>
</ul>
</li>
</ul>
<h1 id="pricing">Pricing</h1>
<p>For the most accurate data, see the <a href="https://aws.amazon.com/lambda/pricing/">official pricing documentation</a>.</p>
<p>Lambda charges you for each request invocation, and also for how long the
request took to run. These prices are probably in USD.</p>
<ul>
<li><strong>Region</strong>: Canada (Central)</li>
<li><strong>Requests</strong>: $0.20 / 1M Requests, or $0.0000002 / Request</li>
<li><strong>Duration (General)</strong>: $0.000016667 for every GB-second</li>
<li><strong>Duration (128MB)</strong>: $0.00000208/sec, or ~$0.0075/hour.
I think if it as 3/4 of a cent per hour.</li>
</ul>
<hr>
<h1 id="hello-world-lambda-function">Hello World Lambda Function</h1>
<h2 id="create-the-function">Create the function</h2>
<ol>
<li>In your browser, open the AWS console <a href="https://console.aws.amazon.com/lambda">Lambda Page</a>.</li>
<li>Click the orange Create Function button</li>
<li>Author from scratch</li>
<li>Function name: HelloWorld</li>
<li>Runtime: Python 3.7</li>
<li>Permissions: Create a new role with basic lambda permissions</li>
<li>click Create Function on the bottom right</li>
</ol>
<h2 id="define-a-test-event">Define a test event</h2>
<ol>
<li>In the dropdowns along the top, one is called "Select a test event".</li>
<li>Click it, and choose Configure test events</li>
<li>Event template: Hello World</li>
<li>Event name: <code>HelloWorldTest1</code></li>
<li>Enter some sample data and click Create.</li>
</ol>
<p><strong>Sample Data</strong>:</p>
<pre><code class="language-JSON">{
  "body": "{\"name\": \"Test User\"}"
}
</code></pre>
<h2 id="write-the-function-code">Write the function code</h2>
<p>Click on the orange lambda icon with the text "HelloWorld" to open the
"Function code" form. Expand it to full screen (button on top right) so the
Save and Test buttons appear along the top.</p>
<p>Write your Hello World function. This was the shortest one I could come up with
that wouldn't throw 500's if you called it with the wrong data.</p>
<pre><code class="language-python">import json

def lambda_handler(event, context):
    name = 'UNDEFINED'
    if 'body' in event:
        try:
            ebody = json.loads(event['body'])
            name = ebody['name']
        except:
            pass
    greeting = f'Hello {name}!'
    body = json.dumps(greeting)
    return {
        'statusCode': 200,
        'body': body
    }
</code></pre>
<p>Click Save, then Test. You should get a response with body <code>'Hello TestUser'</code>.</p>
<h2 id="define-a-trigger">Define a trigger</h2>
<ol>
<li>Click the "+ Add trigger" button.</li>
<li>API Gateway</li>
<li>REST API</li>
<li>Security: Open (in this case)</li>
<li>Additional Settings: Leave the defaults</li>
</ol>
<p>Now if you click on the purple API Gateway button it will open the API Gateway
form. In there, you can get the URL. Copy the link address.</p>
<h3 id="test-the-api">Test the API</h3>
<p>From bash:</p>
<pre><code class="language-bash">curl \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{"name": "MyName"}' \
  https://wg8p74asah.execute-api.us-east-1.amazonaws.com/default/HelloWorld
</code></pre>
<p>It should reply "Hello MyName".</p>
<h2 id="what-are-layers">...What are layers?</h2>
<p>Prominently displayed beneath the function's name in the Designer form
is a button to manage Layers.</p>
<p><a href="https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html">Here's their official documentation</a>.</p>
<p>Layers are dependencies like archives and libraries.
Any one function can use up to 5 of them, and they have a 250MB limit.
They land in <code>/opt</code> when extracted.</p>
<p>We don't need layers for this Hello World app.</p>


<p><strong>Note:</strong> This is usually a bad idea. You should really get the team who runs
the CA to send you the certificate in case some <em>bad guy</em> is doing the MITM and
not the local security team. Use this procedure with caution.</p>
<hr>
<h1 id="get-the-signing-certificate">Get the signing certificate</h1>
<p>Use <code>openssl</code> to print the certificate data.</p>
<pre><code class="language-bash">openssl s_client -connect google.com:443 -showcerts
</code></pre>
<p>In the output you'll see, among other things, some certificates.
They look like this:</p>
<pre><code class="language-text">-----BEGIN CERTIFICATE-----
blaBLAbla
-----END CERTIFICATE-----
</code></pre>
<p>Find the one signed by your local CA. Copy and paste it into a new file,
such as <code>example.com.crt</code>. The <code>.crt</code> file extension is required.
Don't copy anything before or after the BEGIN and END lines.</p>
<hr>
<h1 id="trust-the-certificate">Trust the certificate</h1>
<p>As root, move the certificate file to <code>/usr/local/share/ca-certificates</code>,
then run <code>update-ca-certificates</code>.</p>
<pre><code class="language-bash">mv example.com.crt /usr/local/share/ca-certificates
update-ca-certificates
</code></pre>
<p>That's it. Now your system will trust certs signed by that CA too.</p>


<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#prepare-the-image-in-guest">Prepare the image in-guest</a>
<ul>
<li><a href="#remove-vmware-tools">Remove VMWare Tools</a></li>
<li><a href="#install-cloudbase-init--sysprep-if-needed">Install cloudbase-init &#x26; sysprep (if needed)</a></li>
<li><a href="#shut-down-the-vm--export-the-drive">Shut down the VM &#x26; Export the drive</a></li>
</ul>
</li>
<li><a href="#inject-the-kvm-drivers">Inject the KVM Drivers</a></li>
</ul>
<h1 id="prepare-the-image-in-guest">Prepare the image in-guest</h1>
<p>From VMWare or HyperV, launch your template Windows VM and log in as an
administrative user. Make any firewall changes you need, install any packages,
and ensure that RDP is enabled.</p>
<h2 id="remove-vmware-tools">Remove VMWare Tools</h2>
<p>If the guest has VMWare tools installed, remove them. They won't be needed
where we're going.</p>
<h2 id="install-cloudbase-init--sysprep-if-needed">Install cloudbase-init &#x26; sysprep (if needed)</h2>
<p>This isn't so much for KVM as it is for OpenStack, but cloudbase-init is the
Windows version of cloud-init. This will set your hostname and such when the VM
is created.</p>
<ol>
<li><a href="https://cloudbase.it/cloudbase-init/">Download the Cloudbase-init installer</a></li>
<li>Run through the installer.
<ol>
<li>Don't run as Local System.</li>
<li>Don't name your user Administrator, the named user is made by this
service to run the startup tasks.</li>
</ol>
</li>
<li>At the end of the installation, you'll be prompted to sysprep. If this VM
will be re-used as a template and not just a one-off VM, then sysprep it.</li>
</ol>
<h2 id="shut-down-the-vm--export-the-drive">Shut down the VM &#x26; Export the drive</h2>
<p>Shut down the VM and grab the VMDK or VHDX file it booted from. Copy that file
to a directory on your computer, such as <code>C:\Temp</code>.</p>
<hr>
<h1 id="inject-the-kvm-drivers">Inject the KVM Drivers</h1>
<ol>
<li><a href="https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/virtio-win.iso">Download the VirtIO ISO</a></li>
<li>Mount the ISO to your workstation. In these examples, it's E:</li>
<li>Open PowerShell as admin and verify that the file is where you put it. You
can show some basic info about it like this: <code>get-vhd C:\Temp\Win10.vhdx</code></li>
<li>Create a mount-point for Dism: <code>mkdir C:\mount</code></li>
<li>Mount the volume with Dism:
<code>Dism /mount-image /ImageFile:C:\Temp\Win10.vhdx /Index:1 /MountDir:C:\mount</code></li>
<li>Inject the <code>viostor</code> driver:
<code>Dism /image:C:\mount /Add-Driver /Driver:E:\viostor\w10\amd64 /Recurse</code></li>
<li>Inject the <code>Baloon</code> driver:
<code>Dism /image:C:\mount /Add-Driver /Driver:E:\Balloon\w10\amd64 /Recurse</code></li>
<li>Inject the <code>NetKVM</code> driver:
<code>Dism /image:C:\mount /Add-Driver /Driver:E:\NetKVM\w10\amd64 /Recurse</code></li>
<li>Confirm the drivers injected: <code>Dism /image:C:\mount /Get-Drivers</code></li>
<li>Unmount the image: <code>Dism /Unmount-image /MountDir:c:\mount /commit</code></li>
</ol>
<p>Now the image is ready to be converted to raw/qcow2 and uploaded to OpenStack.</p>
<p><strong>Note</strong>: CloudBase has released a Windows version of <code>qemu-img</code> so you can do
the conversions on Windows now without VirtualBox tools. You can download it
<a href="https://cloudbase.it/qemu-img-windows/">here</a></p>


<p>When you use a tool like <code>dd</code> or <a href="https://www.balena.io/etcher/">Etcher</a> to
turn a USB
stick into boot media, usually for installing an operating system, any attempt
at formatting that disk later will be met with a drive that only has a few GB
total. This is particularly annoying when you use something like a 100+GB USB
stick for that.</p>
<p>The cause of this problem is that a smaller partition is created on the drive
than the drive could potentially hold. To fix the drive you need to
removing all the partitions.</p>
<p>List your devices. Note your USB stick (example: <code>/dev/disk2</code>)</p>
<pre><code class="language-bash">diskutil list
</code></pre>
<p>Try to fix it using diskutil:</p>
<pre><code class="language-bash">diskutil eraseDisk free EMPTY /dev/disk2
</code></pre>
<p>Often that's good enough, and the drive now shows up in Disk Utility with its full capacity.</p>
<hr>
<p>If that fails, you can overwrite the partition/filesystem's metadata at the start of the disk with zeros.</p>
<p>If the drive is mounted, unmount it.</p>
<pre><code class="language-bash">diskutil umountDisk /dev/disk2
</code></pre>
<p>Delete the partition. Note that I tried <code>gpt</code> first but it threw an error
saying <code>gpt show: error: bogus map</code>. Also, as far as I can tell, <code>diskutil</code>
won't actually erase a partition and instead will only reformat it. Strangely,
even <code>fdisk</code> seems to have hidden the option to delete a partition. Guess Apple
<strong>REALLY</strong> doesn't trust its users. That's too bad, since I ended up doing
something even more dangerous.</p>
<p>Overwrite the partition and filesystem data in the first 1GB of the USB drive
with zeros. Be <strong>really really sure</strong> that you select the right disk here.
You can 100% toast your workstation's boot disk if you accidentally specify it.</p>
<pre><code class="language-bash">sudo dd if=/dev/zero of=/dev/disk2 count=1 bs=1GB
</code></pre>
<p>If that doesn't work, set <code>bs=1MB</code> and remove the <code>count=1</code>. You might need to
delete the end of it instead. That'll take a while.</p>


<p>This post covers how I use Ansible to install Node.js. I googled around for a
solution to this and thought the solutions I found looked way too complicated.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#the-problem">The Problem</a>
<ul>
<li><a href="#what-about-nvm">What about NVM?</a></li>
<li><a href="#chris-leanodejs">chris-lea/node.js</a></li>
<li><a href="#curl-x--bash">curl <em>x</em> | bash</a></li>
</ul>
</li>
<li><a href="#the-solution">The Solution</a></li>
<li><a href="#ansible-example">Ansible Example</a>
<ul>
<li><a href="#add-nodesource-repository-to-apts-sourceslist">Add nodesource repository to Apt's sources.list</a></li>
<li><a href="#install-nodejs">Install Node.js</a></li>
<li><a href="#use-npm-normally">Use NPM normally</a></li>
</ul>
</li>
</ul>
<h1 id="the-problem">The Problem</h1>
<p>In theory, installing Node.js should be as easy as running
<code>apt-get install nodejs</code>. The problem is that Node.js versions seem to move
pretty quickly and the one distributed by Cannonical in Ubuntu Bionic is
already too old to run some applications. In my case I was trying to install
<code>firebase-tools</code>, the CLI for Google Firebase, and it wouldn't run with Apt's
version of the package.</p>
<h2 id="what-about-nvm">What about NVM?</h2>
<p>NVM, or Node Version Manager, is a nice little application for installing
various versions of Node.js. It kind of reminds me of python virtualenv.
The problem with NVM is that it relies on setting environment variables and
defining the <code>nvm</code> function using the <code>.bashrc</code> file. Ansible doesn't read
that file when it runs.</p>
<p>In theory, the Ansible <code>npm</code> module supports accepting an executable path,
specifically to support NVM, but I had to do some ugly shell commands to
collect that path and even afterwards it wouldn't actually work. Specifically
running the <code>npm</code> binary deployed by NVM without scoping the <code>.bashrc</code> file
throws stack traces instead of installing your app.</p>
<h2 id="chris-leanodejs">chris-lea/node.js</h2>
<p>A lot of the sites I found were adding the Apt PPA <code>'ppa:chris-lea/node.js'</code>.
I don't know what that is, but it doesn't sit well with me. Just seems like
something that wont age well, might be insecure, and is probably fragile.</p>
<p>To make matters worse, when I tried it out it didn't seem to support Bionic.</p>
<h2 id="curl-x--bash">curl <em>x</em> | bash</h2>
<p>Some examples I saw mentioned just curling various URLs and piping them to bash
to run a script.
Sometimes I'm OK with doing this, but that sort of installation method doesn't
really seem appropriate for an Ansible playbook. If nothing else it'd be hard
to make those tasks idempotent. Plus, we can do better.</p>
<hr>
<h1 id="the-solution">The Solution</h1>
<p>If you look at the <a href="https://github.com/nodesource/distributions">nodesource/distributions GitHub page</a>,
they have some really nice documentation in their <code>README.md</code>. Under the
<strong>Manual Installation</strong> section you can see that they have a nice, stable
looking Apt repository URL. This worked great for me, and is trivial to
set up Ansible to work with.</p>
<hr>
<h1 id="ansible-example">Ansible Example</h1>
<h2 id="add-nodesource-repository-to-apts-sourceslist">Add nodesource repository to Apt's sources.list</h2>
<pre><code class="language-yaml">- name: "Add nodejs apt key"
  apt_key:
    url: https://deb.nodesource.com/gpgkey/nodesource.gpg.key
    state: present

- name: "Add nodejs 13.x ppa for apt repo"
  apt_repository:
    repo: deb https://deb.nodesource.com/node_13.x bionic main
    update_cache: yes
</code></pre>
<h2 id="install-nodejs">Install Node.js</h2>
<pre><code class="language-yaml">- name: "Install nodejs"
  apt:
    update_cache: yes
    name: nodejs
    state: present
</code></pre>
<h2 id="use-npm-normally">Use NPM normally</h2>
<p>With this version of Node.js, I was able to install and run the application.</p>
<pre><code class="language-yaml">- name: "Install NPM-distributed command-line tools"
  npm:
    global: yes
    name: "{{ item }}"
  with_items:
    - firebase-tools
</code></pre>


<p><strong>This post is linked to from the <a href="/blog-website.html">Blog Website Project</a></strong></p>
<hr>
<p>This static website is hosted using Google Cloud Storage, which is great except
that it doesn't support HTTPS. There's no really good excuse to not support
HTTPS on a website today.</p>
<p>To close that gap, CloudFlare can handle the HTTPS negotiation for you and it
takes no time at all to set up.</p>
<h1 id="configure-https-in-cloudflare">Configure HTTPS in CloudFlare</h1>
<p>This guide assumes CloudFlare is already acting as a CDN for your site.</p>
<ol>
<li>Go to the CloudFlare DNS page and find the A record or CNAME for your FQDN</li>
<li>Ensure that the cloud icon is yellow and not grey, indicating that the CDN
is in use and not bypassing CloudFlare.</li>
<li>Click on the padlock labelled SSL/TLS</li>
<li>Go to the Overview tab</li>
<li>Select Flexible.... Basically that's it, you're good.
HTTPS is now running on your site. CloudFlare is a really great product!
There's more that can be done, though.</li>
<li>Go to the Edge Certificates tab</li>
<li>Set "Always Use HTTPS" to On</li>
</ol>
<p>In my testing the "Always Use HTTPS" redirect change didn't take effect
immediately. After about 15 minutes 301's were coming back as expected.</p>
<pre><code class="language-text">vagrant@dev-env:~/$ curl -i http://kyle.pericak.com
HTTP/1.1 301 Moved Permanently
Date: Tue, 03 Dec 2019 00:52:25 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Tue, 03 Dec 2019 01:52:25 GMT
Location: https://kyle.pericak.com/
Server: cloudflare
CF-RAY: 53f18ecdec10b677-YWG
</code></pre>
<p><a href="https://support.cloudflare.com/hc/en-us/articles/204144518">You can find more details and official documentation here.</a></p>


<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<p>In this project I'll be trying out each of the Google Cloud Platform's
services. I'll be updating it pretty regularly.</p>
<hr>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes, or will include, the following posts.
If any aren't finished, check back later! I'll also certainly be adding more
as time goes on.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left"></td>
<td align="left"><strong>Continuous Integration</strong></td>
</tr>
<tr>
<td align="left">2019-08-06</td>
<td align="left"><a href="/google-cloud-build.html">Google Cloud Build: Basics</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Database</strong></td>
</tr>
<tr>
<td align="left">2019-10-14</td>
<td align="left"><a href="/gcp-firestore-python.html">Google Cloud Firestore Basics - Python</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Containers</strong></td>
</tr>
<tr>
<td align="left">2019-08-05</td>
<td align="left"><a href="/google-container-registry.html">Google Container Registry: Basics</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Cloud Storage</strong></td>
</tr>
<tr>
<td align="left">2019-08-09</td>
<td align="left"><a href="/google-cloud-storage-website.html">Hosting websites on Cloud Storage</a></td>
</tr>
</tbody>
</table>


<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<p>In this project I'll be trying many of AWS's services.</p>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes the following posts.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left"></td>
<td align="left"><strong>Running Servers on AWS</strong></td>
</tr>
<tr>
<td align="left">2020-01-09</td>
<td align="left"><a href="/aws-ec2-python.html">Create &#x26; Terminate EC2 Instances from Python</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Running Containers on AWS</strong></td>
</tr>
<tr>
<td align="left">2019-08-26</td>
<td align="left"><a href="/aws-ecr.html">Save a Docker image to Elastic Container Registry (ECR)</a></td>
</tr>
<tr>
<td align="left">2019-09-10</td>
<td align="left"><a href="/aws-fargate.html">Run a Docker container Using AWS Fargate</a></td>
</tr>
<tr>
<td align="left">2020-01-15</td>
<td align="left"><a href="/aws-codepipeline-ecr.html">CodePipeline: Continuous Delivery to AWS ECR</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Cloud Storage</strong></td>
</tr>
<tr>
<td align="left">2019-09-26</td>
<td align="left"><a href="/aws-s3-python.html">Using AWS S3 from Python</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Serverless Applications</strong></td>
</tr>
<tr>
<td align="left">2020-01-08</td>
<td align="left"><a href="/aws-lambda.html">Lambda basics</a></td>
</tr>
<tr>
<td align="left">2020-01-09</td>
<td align="left"><a href="/aws-lambda-iam.html">IAM for Lambda</a></td>
</tr>
<tr>
<td align="left">2020-01-21</td>
<td align="left"><a href="/aws-api-gateway-iam.html">IAM Authentication using API Gateway</a></td>
</tr>
</tbody>
</table>


<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<h4 id="table-of-contents">Table of contents</h4>
<ul>
<li><a href="#what-is-openstack">What is OpenStack?</a></li>
<li><a href="#project-posts--progress">Project Posts &#x26; Progress</a></li>
<li><a href="#why-use-an-openstack-private-cloud">Why Use an OpenStack Private Cloud?</a></li>
<li><a href="#complexity-and-openstack-ad">Complexity and OpenStack (Ad)</a></li>
<li><a href="#installing-openstack">Installing OpenStack</a>
<ul>
<li><a href="#non-production-installation-options">Non-Production Installation Options</a>
<ul>
<li><a href="#devstack">DevStack</a></li>
<li><a href="#small-physical-server--kolla-ansible">Small Physical Server + Kolla-Ansible</a></li>
<li><a href="#virtual-server--kolla-ansible">Virtual Server + Kolla-Ansible</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#containerized-openstack">Containerized OpenStack</a></li>
</ul>
<h1 id="what-is-openstack">What is OpenStack?</h1>
<p><a href="https://www.openstack.org/">OpenStack</a> is a collection of open-source projects
that work together to form a private cloud solution. It's by far the
biggest open source private cloud option available. The <a href="https://www.openstack.org/foundation/">OpenStack Foundation</a>
has over 105,000 community members from 187 countries around the world.</p>
<p>At a high level, you can think of it as an abstraction layer that sits on top
of other popular cloud technologies such as storage clusters, hypervisors, and
network solutions. For example, KVM (the popular Linux hypervisor) is not part
of OpenStack, but it's used by <strong>Nova</strong>, a component of OpenStack which can
orchestrate against KVM, VMWare, HyperV, and Xen.</p>
<hr>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes the following posts.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">2019-08-13</td>
<td align="left"><a href="/openstack-aio-ka-metal.html">Install OpenStack on Metal - Intel NUC</a></td>
</tr>
<tr>
<td align="left">2019-08-11</td>
<td align="left"><a href="/openstack-aio-ka-vm.html">Install OpenStack inside a VM</a></td>
</tr>
<tr>
<td align="left">2019-08-26</td>
<td align="left"><a href="/openstack-kolla-custom-plugin.html">Modifying OpenStack Kolla Docker Images</a></td>
</tr>
<tr>
<td align="left">2019-12-03</td>
<td align="left"><a href="/openstack-ansible.html">Operating OpenStack from Ansible</a></td>
</tr>
<tr>
<td align="left">2020-01-25</td>
<td align="left"><a href="/openstack-cloudinit-powershell.html">OpenStack Cloud-Init Powershell Example</a></td>
</tr>
<tr>
<td align="left">2019-12-17</td>
<td align="left"><a href="windows-kvm-drivers">Injecting KVM Drivers into Windows</a></td>
</tr>
</tbody>
</table>
<hr>
<h1 id="why-use-an-openstack-private-cloud">Why Use an OpenStack Private Cloud?</h1>
<ol>
<li><strong>Price at scale:</strong> At small scales, private clouds lose on price
against the public clouds. As you start to ramp up though, the public clouds
start to really nickle and dime you. Private clouds have higher up-front
costs but they win out in the long-run.</li>
<li><strong>Predictable Cost:</strong> Spin up a few services on AWS or Azure. How much will
they cost you each month? As it stands, you're guess is going to be wrong.
After the free trial periods and special offers stop muddying the water,
you'll usually discover you're spending more than you thought.</li>
<li><strong>Hardware Re-Use:</strong> Many IT organizations already have hardware sitting
around running expensive hypervisor software. OpenStack is free and can use
that existing hardware, giving you more for less.</li>
<li><strong>Data Sovereignty:</strong> For legal and security reasons, many organizations
must know exactly where their data is sitting at rest. With private clouds,
there's no ambiguity.</li>
<li><strong>Behind Your Firewall:</strong> So you spent a ton of money on some next
generation firewall technologies with intrusion detection/prevention, and
trained the staff up to use it. You run an arguably well secured
data-center. Running OpenStack keeps the cloud behind your existing security
measures.</li>
<li><strong>Latency:</strong> Particularly for edge clouds and VDI solutions, having the
cloud located right on-premises can hugely improve user experience.</li>
<li><strong>Shadow IT:</strong> Developers are going to develop, even if your organization
can't get them the tools they need in time. By offering your own in-house
cloud, you can help prevent
<a href="https://en.wikipedia.org/wiki/Shadow_IT">shadow IT</a> from emerging.</li>
<li><strong>Control:</strong> This one is a double-edged sword. Having more control over your
cloud can be its own benefit, but there's also value in letting the pros
handle it.</li>
<li><strong>Business as usual, without VMWare:</strong> Don't get me wrong, VMWare make a
great product... I've never heard anyone say they were happy with their
VMWare bill. It's insanely expensive when there's an open source alternative
that can check all the same boxes.</li>
</ol>
<hr>
<h1 id="complexity-and-openstack-ad">Complexity and OpenStack (Ad)</h1>
<p>OpenStack is significantly more difficult to get started with than the public
clouds. You can't really install OpenStack without first understanding what
each of the main components does, and even then each one has its own quirks for
their installation. If you want to install in production, you also need to
figure out how to get it all highly available.</p>
<p>Unless you've got a huge team that's eager to tackle this complexity, I'm going
to use this opportunity to plug the software &#x26; solution I've helped build for
<a href="https://breqwatr.com">Breqwatr</a>. They'll stand up your cloud in a couple days
flat, teach your team how to use it, and keep the lights on - all while
charging significantly less than VMWare would cost for the same hardware.</p>
<p>Breqwatr, and companies like it, can be your trusted cloud partner and abstract
away that complexity by handling the installation, upgrades, monitoring, and
day-to-day operation of the cloud.</p>
<hr>
<h1 id="installing-openstack">Installing OpenStack</h1>
<h2 id="non-production-installation-options">Non-Production Installation Options</h2>
<p>If you're not looking for a production deployment and instead want to try
OpenStack out to learn it, there are a few good options.</p>
<h3 id="devstack">DevStack</h3>
<p>The standard developer option is probably <a href="https://docs.openstack.org/devstack/latest/">DevStack</a>.</p>
<p>In truth I haven't used this since I've always needed my OpenStack installs to
mirror a production-ready environment. If I try it out, I'll replace the above
link with one to a guide I write myself.</p>
<h3 id="small-physical-server--kolla-ansible">Small Physical Server + Kolla-Ansible</h3>
<p>Another option is to install OpenStack on a physical server. I've documented
my experience doing this using images from the <a href="https://github.com/openstack/kolla">Kolla project</a>,
and Ansible automation from the <a href="https://github.com/openstack/kolla-ansible">Kolla-Ansible project</a>.</p>
<p>The nice thing about this approach is you can use a server at home, and once
you're done it can be a VM host as your private house cloud.</p>
<p><strong><a href="/openstack-aio-ka-metal.html">Installing OpenStack on a physical server</a></strong></p>
<h3 id="virtual-server--kolla-ansible">Virtual Server + Kolla-Ansible</h3>
<p>This option is great for development use if you already have a cloud available
or want to use a public cloud for testing OpenStack. The VMs you launch on it
will perform terribly though, and there are some networking complications.</p>
<p>Don't bother trying to launch Windows VMs at all in a nested virtualisation
scenario, you'll be lucky if they even boot.</p>
<p><strong><a href="/openstack-aio-ka-vm.html">Installing OpenStack on a Virtual Server</a></strong></p>
<hr>
<h1 id="containerized-openstack">Containerized OpenStack</h1>
<p>Managing OpenStack clusters is a pain. It has so many services, each with their
own dependencies and operating instructions, there's a lot that can go wrong.
One thing that's really helped with that is to containerize each service and
deploy them all using Docker.</p>
<p>The <a href="https://github.com/openstack/kolla">Kolla project</a> does such a great job
of that, that there's no need to do it yourself. That is, until you need to
change an image, for instance to install a Cinder plugin.</p>
<p><strong><a href="/openstack-kolla-custom-plugin.html">Modifying OpenStack Kolla Docker Images</a></strong></p>


<p>As of Vim7, spell-checking is a built-in feature.</p>
<p>The spell checker will highlight incorrectly spelled words in red.
Light blue highlights indicate words which are only found in another dictionary
(such as "color" from <code>en_us</code>).</p>
<hr>
<h1 id="commands">Commands</h1>
<h2 id="enabledisable">Enable/Disable</h2>
<p>Enable spellcheck</p>
<ul>
<li><code>set</code> for all buffers, <code>setlocal</code> for the current one</li>
<li><code>en_ca</code> will use Canadian spelling.</li>
</ul>
<pre><code class="language-text">:set spell spelllang=en_ca
:setlocal spell spelllang=en_ca
</code></pre>
<p>Or if you're not picky, the <code>spelllang</code> argument can be omitted. I'm not sure
what the default is, but I figure it's probable <code>en</code> which uses all the English
dictionaries together.</p>
<pre><code>:set spell
</code></pre>
<p>Disable spellcheck:</p>
<pre><code class="language-text">:set nospell
</code></pre>
<h2 id="using-spell-check">Using Spell-Check</h2>
<p>Move to the next misspelled word</p>
<pre><code>]s
</code></pre>
<p>previous misspelled word</p>
<pre><code>[s
</code></pre>
<p>Highlight a misspelled word. Show possible replacements.
Press return with no input to go back.</p>
<pre><code>z=
</code></pre>
<p>Accept the spelling of a word:</p>
<pre><code class="language-text">zg
</code></pre>
<p>Mark a word's spelling as wrong</p>
<pre><code class="language-text">zw
</code></pre>


<p><strong>This post is linked to from the <a href="/gcp.html">GCP: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to interact with Google's Cloud Firestore, using examples
written in Python. The concepts should apply to any language.</p>
<p>Firestore is the newer version of Datastore. You can find the differences
<a href="https://cloud.google.com/firestore/docs/firestore-or-datastore">here</a>.
In this post, the  database is configured in Native mode.
Other than some extra language support, Native mode's coolest new feature seems
to be the ability to listen to a post. From what I can tell, this could replace
the need for pub/sub in some scenarios.</p>
<p>Throughout the post I'll use a project named "example". Be sure to replace this
with your own project name.</p>
<hr>
<h1 id="pricing">Pricing</h1>
<p>Official Firestore pricing data is <a href="https://cloud.google.com/firestore/pricing">here</a>.</p>
<p><strong>Note that Firestore mode is almost twice the price of Datastore mode.</strong>
Datastore pricing can be found <a href="https://cloud.google.com/datastore/pricing">here</a>.</p>
<p>Every day a number of free operations are allowed, which for a small
application makes Datastore very affordable. Even after the free tier though,
the pricing is pretty good. Firestore and Datastore have the same free tiers.</p>
<h2 id="summary-of-pricing-datastore-vs-firestore">Summary of pricing: Datastore vs Firestore</h2>
<ul>
<li>Storage: $/GB/Month
<ul>
<li>Datastore: 0.108</li>
<li>Firestore: 0.180</li>
</ul>
</li>
<li>Reads: $/100,000
<ul>
<li>Datastore: 0.036</li>
<li>Firestore: 0.060</li>
</ul>
</li>
<li>Writes: $/100,000
<ul>
<li>Datastore: $0.108</li>
<li>Firestore: $0.180</li>
</ul>
</li>
<li>Deletes: $/100,000
<ul>
<li>Datastore: 0.012</li>
<li>Firestore: 0.020</li>
</ul>
</li>
</ul>
<p>Max entity size and document size are both 1MB, so you could realistically just
use JSON strings in entities instead to save money if you don't need any of
Firestore's other features.</p>
<hr>
<h1 id="datastorefirestore-concepts">Datastore/Firestore Concepts</h1>
<p>Both options use a non-relational/NoSQL database. NoSQL databases in general
allow for the direct serialization of a programs objects/dictionaries.</p>
<h2 id="firestore">Firestore</h2>
<p><a href="https://cloud.google.com/firestore/docs/data-model">Official Firestore data model docs</a></p>
<p>Firestore uses a document database organized into documents and collections.
This is a change from the older Datastore model of Entities organized by Kinds.</p>
<p>Documents support nested data structures ("Maps"). Think of them like JSON
objects, but with more types supported. Documents are limited to 1MB each.</p>
<p>Documents are stored in Collections.
Collections are containers for documents, like a filesystem's directories.
Documents can also contain their own nested collections, which store other
documents. Nesting has a depth limit of 100.</p>
<p>Documents are identified by References. In python, a reference looks like
this:</p>
<pre><code class="language-python">collection('collection1').document('document1')
</code></pre>
<h2 id="datastore">Datastore</h2>
<p>In Datastore, objects are called Entities. The key-value data pairs stored by
entities are called properties.</p>
<p>Entities are uniquely identified by keys.
Entity Keys consists of the following components:</p>
<ul>
<li><em>Namespace</em>: Basically a tenant identifier, like a user ID. By default, the
namespace is an empty string. Any entity created as a "child" will inherit
its parent's namespace.</li>
<li><em>Kind</em>: A string describing the category or purpose of the entity.</li>
<li><em>Identifier</em>: Unlike namespace and kind which are shared between like
entities, the identifier is unique to each entity. This is either a string
called a "key name", or an integer numeric ID.</li>
<li><em>Ancestor Path</em>: An optional component of the key, identifying its parents to
form a database hierarchy.</li>
</ul>
<hr>
<h1 id="create-iam-service-account">Create IAM Service Account</h1>
<p>Before your code can access Datastore/Firestore, access needs to be configured
in the <a href="https://console.cloud.google.com/iam-admin">Identity Access Management</a>
tool.</p>
<p>First, create a <a href="https://console.cloud.google.com/iam-admin/serviceaccounts">Service Account</a>:</p>
<ol>
<li>Open the Service Account page</li>
<li>Select your project</li>
<li>Click "Create Service Account"</li>
<li>Service Account Details:
<ol>
<li>Service account name: <code>PythonDatastoreAdmin</code></li>
<li>Service account ID: <code>pythondatastoreadmin</code></li>
<li>Service account description: SA with Full Datastore Access for Python</li>
</ol>
</li>
<li>Grant this service account access to project (optional)
<ol>
<li>Select a role: Datastore > Cloud Datastore Owner</li>
<li>Continue</li>
</ol>
</li>
<li>Grant users access to this service account (optional)
<ol>
<li>Download a key file for Python to use. Use JSON as the key type.</li>
<li>Done</li>
</ol>
</li>
</ol>
<p>I downloaded my key to <code>$HOME/gcp_keys/datastore_admin.json</code></p>
<hr>
<h1 id="python-environment-setup">Python Environment Setup</h1>
<p>Build a virtualenv. If you're in a git repo, don't forget to ignore this
directory.</p>
<pre><code class="language-bash">python3 -m venv env3
source env3/bin/activate
</code></pre>
<p>Write the requirement file</p>
<p><code>vi requirements.txt</code></p>
<pre><code class="language-text">google-cloud-firestore
</code></pre>
<p>Install the requirements</p>
<pre><code class="language-bash">pip install -r rqeuirements.txt
</code></pre>
<hr>
<h1 id="python-code-examples">Python Code Examples</h1>
<p>A fake project name of "example" will be used. Be sure to use a valid one.</p>
<p>Some notes about the Firestore python API:</p>
<ul>
<li><code>firestore.client().collection</code> returns a <a href="https://googleapis.dev/python/firestore/latest/collection.html">CollectionReference</a>,
which allows you to operate on the named collection.</li>
<li><code>firestore.client().collection().list_documents</code> returns a <a href="google.api_core.page_iterator.GRPCIterator">Page Iterator</a>.
Casting a page iterator to a list will trigger a read operation, returning a
list of <code>DocumentReference</code> objects.</li>
<li><code>firestore.client().collection().document</code> returns
a <a href="https://googleapis.dev/python/firestore/latest/document.html">DocumentReferece</a>,
which interacts with the actual Firestore documents. The <code>DocumentReference</code>
object can operate (get/set/create) on the actual Document.</li>
<li><code>firestore.client().collection().document().get()</code> will return
a <a href="https://developers.google.com/android/reference/com/google/firebase/firestore/DocumentSnapshot">DocumentSnapshot</a>
of the document. The document snapshot is how you read data from a document.
Document snapshots are a representation of the document state taken at the
instant of their creation.</li>
<li><code>firestore.client().collection().document().create()</code> makes a new document.
If the object already  existed, a <code>google.api_core.exceptions.AlreadyExists</code>
exception is thrown.</li>
<li>Write operations (create &#x26; set) return a <a href="https://cloud.google.com/firestore/docs/reference/rest/v1/WriteResult">WriteResult</a>.</li>
</ul>
<pre><code class="language-python">from os import environ
from google.cloud import firestore

# The key file is an exported JSON file from the IAM tool
key_file_path = '{}/gcp_keys/datastore_admin.json'.format(environ['HOME'])

# When running this on a local system, the cloud.google package from GCP
# expects the GOOGLE_APPLICATION_CREDENTIALS environment variable to be set
# to the exported IAM key file's path
environ['GOOGLE_APPLICATION_CREDENTIALS'] = key_file_path

# google.cloud.datastore.Client expects the project name as an argument
project = 'example'

# Instantiate the datastore client. This step authenticates to GCP.
# See the Credential Error note below if authentication fails
client = firestore.Client(project)

# Get a reference to the test collection. This returns a
# google.cloud.firestore_v1.collection.CollectionReference object
ref_col_test = client.collection('test')

# Get a reference to a document which may or may not exist. This returns a
# google.cloud.firestore_v1.document.DocumentReference object
ref_doc_document1 = ref_col_test.document('document1')

# Perform the actual database query to get the document. get() returns a
# google.cloud.firestore_v1.document.DocumentSnapshot object.
snapshot_document1 = ref_doc_document1.get()

# Note that there is no data
assert document1.to_dict() is None

# You can use document.exists to check if the document already exists.
# Here's an example of doing a safe insert of some sample data:
sample_data = {
    'nums': {'num1': 100, 'num2': 200},
    'strs': {'animal1': 'dog', 'animal2': 'cat'}}
if snapshot_document1.exists is False:
    createe_write_result = ref_doc_document1.create(sample_data)

document1_data = ref_doc_document1.get().to_dict()

# Finally, update the data
sample_data['strs']['animal3'] = 'bird'
set_write_result = ref_doc_document1.set(sample_data)

# List the documents in the collection. The iterator returns a paginated list
# of document references. You can then operate on those references.
doc_list_iterator = ref_col_test.list_documents()
doc_ref_list = list(doc_list_iterator)
sample_document_snapshot = doc_ref_list[0].get()

# Show that the changes were applied to the data. This will return 'bird'
animal3 = sample_document_snapshot.to_dict()['strs']['animal3']
</code></pre>
<h2 id="troubleshooting">Troubleshooting</h2>
<h3 id="credentials-error">Credentials Error</h3>
<p>If your local google cloud authentication isn't set up correctly, you'll get an
error saying:</p>
<pre><code class="language-text">google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
</code></pre>
<p>If that happens, review the IAM Service account steps.</p>
<h3 id="iterator-has-already-started">Iterator has already started</h3>
<p>If you try and cast a <code>GRPCIterator</code> to list twice, you get a <code>ValueError</code>.</p>
<pre><code class="language-text">ValueError: ('Iterator has already started', &#x3C;google.api_core.page_iterator.GRPCIterator object)
</code></pre>
<p>The <code>ValueError</code> is thrown because <code>._started</code> is <code>True</code> in the iterator. It has
already done its thing and won't work again. I didn't see any good way to
restart the iterator without discarding it.
You can see the details in the <a href="https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/page_iterator.py">page iterator source code</a>.</p>
<p>Best course is just to get a new iterator from the reference object.
See in the following example how the ID changes. The <code>doc_list_iterator</code> var is
being pointed at a new iterator. The old iterator gets gc'd by Python as usual.</p>
<pre><code>>>> id(doc_list_iterator)
4422123408
>>> doc_list_iterator = ref_col_test.list_documents()
>>> id(doc_list_iterator)
4422123472
</code></pre>
<p>It will work as expected.</p>


<p>This post covers getting started with a very simple Google Cloud Functions API,
a great serverless solution with affordable pricing. This API will pull the
IP address from the HTTP request, and return <code>Hello &#x3C;source IP></code>.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#pricing-details">Pricing Details</a></li>
<li><a href="#deploy-hello-world-function">Deploy Hello World Function</a>
<ul>
<li><a href="#mainpy">main.py</a></li>
<li><a href="#requirementspy">requirements.py</a></li>
</ul>
</li>
<li><a href="#accessing-the-function">Accessing the Function</a>
<ul>
<li><a href="#from-the-console-web-ui">From the Console Web UI</a></li>
<li><a href="#from-the-request-url">From the Request URL</a></li>
</ul>
</li>
</ul>
<h1 id="pricing-details">Pricing Details</h1>
<p>Cloud Functions pricing is pretty affordable for most use cases, assuming you
write your functions in such a way that they don't get invoked a few too many
millions of times.</p>
<p>Check the official <a href="https://cloud.google.com/functions/pricing">Google docs</a>
for accurate and up-to-date pricing, but here's what it looked like at the time
of my writing this.</p>
<p>Pricing is broken into a few fees:</p>
<ul>
<li><em>Invocations</em>: A flat fee per function execution. $0.0000004 per invocation,
excluding the first 2 million.</li>
<li><em>Compute Time</em>: At the lowest clock speed (200MHz x 128MB RAM), compute costs
$0.000000231/100ms, ceil 100.</li>
<li><em>Networking</em>: $0.12/GB egress for data, while ingress is free.</li>
</ul>
<hr>
<h1 id="deploy-hello-world-function">Deploy Hello World Function</h1>
<p>Open <a href="https://console.cloud.google.com/functions/">Google Cloud Functions</a> in
your browser.</p>
<p>Create a function.</p>
<ul>
<li>Name: HelloWorld</li>
<li>Memory Allocated: 128 MB</li>
<li>Trigger: HTTP</li>
<li>Authentication: Allow unauthenticated invocations</li>
<li>Source code: Inline Editor</li>
<li>Runtime: Python 3.7</li>
<li>Other settings: Leave them as defaults</li>
</ul>
<h2 id="mainpy">main.py</h2>
<p>The API function defined by <code>Function to execute</code> accepts a
<a href="https://flask.palletsprojects.com/en/1.1.x/api/#incoming-request-data">flask.Request</a>
argument.</p>
<pre><code class="language-python">def hello_world(request):
    """" Return a Hello message to sender of the API call """
		return 'Hello {}!'.format(request.remote_addr)
</code></pre>
<h2 id="requirementspy">requirements.py</h2>
<p>You can leave this blank. It's a list of libraries that will be used.</p>
<hr>
<h1 id="accessing-the-function">Accessing the Function</h1>
<h2 id="from-the-console-web-ui">From the Console Web UI</h2>
<ol>
<li>Navigate to the <a href="https://console.cloud.google.com/functions">Functions page</a>.</li>
<li>Click on the 3 dots next to the function name, and choose "Test function".</li>
<li>Populate and JSON for the GET request in "Triggering event". The above
example doesn't accept any input data so <code>{}</code> is all you need.</li>
<li>Click "Test the function"</li>
</ol>
<h2 id="from-the-request-url">From the Request URL</h2>
<ol>
<li>Navigate to the <a href="https://console.cloud.google.com/functions">Functions page</a>.</li>
<li>Click the function's name to navigate to the "Function details" page</li>
<li>Go to the 'Trigger" tab</li>
<li>Find the URL. Copy it.</li>
</ol>
<p>You can now test the URL with curl, your browser, httpie, python, etc.</p>
<pre><code class="language-bash">https://&#x3C;region>-&#x3C;project>.cloudfunctions.net/HelloWorld
</code></pre>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to upload and download files from Amazon's S3 using
Python. Setup is done from the web UI.</p>
<h1 id="create-a-bucket">Create a Bucket</h1>
<p>Go to the <a href="https://s3.console.aws.amazon.com">AWS S3 page</a>.</p>
<p>Click on "+ Create Bucket" and fill out the form. For access, choose
Block all public access. For the rest of the options, the defaults are fine.</p>
<hr>
<h1 id="create-iam-credentials">Create IAM Credentials</h1>
<p>IAM stands for Identity and Access Management.</p>
<h2 id="create-a-policy">Create a Policy</h2>
<p>From the left nav-bar, go to policies. The default policies don't grant
programmatic access, so you need to make a new one.</p>
<p>For admin rights, I just used the visual policy builder and granted it
access to everything. I'll come back and update this guide with how to lock
that down later.</p>
<hr>
<h1 id="add-the-policy-to-a-group">Add the policy to a group</h1>
<p>Go to the groups on the left. Either make a new one or edit an existing one.
Under Permissions, click Attach Policy. If you want, you can also add
<code>AmazonS3FullAccess</code> to see the difference between the two.</p>
<h2 id="createedit-a-user">Create/Edit a User</h2>
<p>On the left nav-bar, go to Users > Add User. Or edit a user that exists.
Fill in the wizard and assign the group.</p>
<hr>
<h1 id="use-python-to-interact-with-s3">Use Python to interact with S3</h1>
<h2 id="install-boto3">Install Boto3</h2>
<p>Python needs boto3 installed to interact with AWS.</p>
<pre><code class="language-bash">pip install boto3
</code></pre>
<h2 id="uploading-and-downloading-files">Uploading and Downloading Files</h2>
<pre><code class="language-python">import boto3

# Fill in the key and key id, or read them from stdin/file/whatever
key_id = ''
key = ''

# Create the authenticated s3 client
session = boto3.Session(aws_access_key_id=key_id, aws_secret_access_key=key)
client = session.client('s3')

# Define the file/object
filename = '/example/file/path'
bucket_name = 'my_bucket'
key = 'my_key'

# Upload Example
client.upload_file(filename, bucket_name, key)

# Download Example
client.download_file(bucket_name, object_name, path)
</code></pre>


Kyle Pericak

"It works in my environment"

Categories

Tags