<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<h1>Workflow</h1>
<ol>
<li>Commit code to AWS CodeCommit</li>
<li>CloudWatch rule triggers CodePipeline</li>
<li>CodePipeline uses CodeBuild to build the Docker image</li>
<li>CodePipeline pushes the image to ECR</li>
</ol>
<hr>
<h1>Deploy Code to CodeCommit</h1>
<ol>
<li>Open <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codecommit/repositories">CodeCommit</a>.</li>
<li>Create a repository</li>
<li>Go to your IAM page and edit your user.
<ol>
<li>Look for "HTTPS Git credentials for AWS CodeCommit"</li>
<li>click Generate Credentials</li>
<li>Save your credentials.</li>
</ol>
</li>
<li>Grab the HTTPS URL for your repository and <code>git clone</code> it. Use your new
credentials.</li>
<li>Commit your code (including a Dockerfile) to the CodeCommit repository.</li>
</ol>
<hr>
<h1>Write buildspec.yml</h1>
<p>AWS CodeBuild will look for a <code>buildspec.yml</code> file to dictate its actions.
This is where we define how the image will be built. Put it in your CodeCommit
project's root directory.</p>
<p>Here's mine. This will build a docker image and tag it with a timestamp, then
push it to ECR as the timestamp and latest. The variables are defined inside
the build as Environment Variables.</p>
<pre><code class="language-yml"># buildspec.yml
version: 0.2

phases:
  install:
    runtime-versions:
      docker: 18
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - $(aws ecr get-login --no-include-email --region ca-central-1)
      - IMAGE_TAG=$(date +%Y.%m.%d.%I.%M)
  build:
    commands:
      - echo Build started on `date`
      - echo "Building the Docker image."
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest
</code></pre>
<hr>
<h1>Define the Pipeline</h1>
<h2>Cloud Build Role</h2>
<p><strong>Bad Practice Alert</strong>: These steps don't apply the principal of least
privilege. You should lock down your role to be allowed to do only the minimum
that it needs to do.</p>
<ol>
<li>In the IAM page, create a new role.</li>
<li>It will be used by CodeBuild.</li>
<li>Assing the following policies (again, more than it needs)
<ul>
<li>CloudWatchLogsFullAccess</li>
<li>AWSCodeBuildAdminAccess</li>
<li>AmazonS3FullAccess</li>
<li>AmazonEC2ContainerRegistryFullAccess</li>
<li>AWSCodeCommitFullAccess</li>
</ul>
</li>
<li>Name: Assign a name, such as <code>CloudBuild-Admin</code></li>
</ol>
<h2>Define CloudBuild project</h2>
<ol>
<li>Navigate to the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codebuild/projects">CloudBuild Console</a></li>
<li>Create build project</li>
<li>No build badge</li>
<li>Source provider: CodeCommit</li>
<li>Choose your repository</li>
<li>Choose your branch</li>
<li>Managed Image</li>
<li>Operating System: Ubuntu</li>
<li>Runtime: Standard</li>
<li>Image: aws/codebuild/standard:3.0</li>
<li>Image version: latest</li>
<li>Environment type: Linux</li>
<li>Privileged: true</li>
<li>Service Role: Existing service role. Use the one you just made.</li>
<li>Allow CodeBuild to modify this role: False (we gave it admin)</li>
<li>Additional Configuration - Environment variables (click Add to get more)
- IMAGE_REPO_NAME: (your image name)
- AWS_ACCOUNT_ID: (your numeric AWS account iD)
- AWS_DEFAULT_REGION: ca-central-1</li>
<li>Buildspec: Use a buildspec file</li>
<li>Artifacts: No artifacts</li>
<li>Logs: Give a group and stream name for CloudWatch</li>
<li>Create Build Project</li>
</ol>
<p>Now test the build by clicking "Start build". It should create your image.
You'll need to disable artifacts.</p>
<p>Once the build works and your image is pushed successfully to ECR, you can move
on to setting up a pipeline so this happens automatically.</p>
<h2>Create Pipeline Role</h2>
<p><strong>Bad Practice Alert</strong>: These steps don't apply the principal of least
privilege. You should lock down your role to be allowed to do only the minimum
that it needs to do.</p>
<ol>
<li>In the IAM page, create a new role.</li>
<li>It will be used by CodeBuild.</li>
<li>Assing the following policies (again, more than it needs)
<ul>
<li>CloudWatchLogsFullAccess</li>
<li>AWSCodeBuildAdminAccess</li>
<li>AmazonS3FullAccess</li>
<li>AmazonEC2ContainerRegistryFullAccess</li>
<li>AWSCodeCommitFullAccess</li>
</ul>
</li>
<li>Name: Assign a name, such as <code>CloudBuild-Admin</code></li>
</ol>
<h2>Define the pipeline</h2>
<ol>
<li>Open the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codepipeline/pipelines">CodePipeline console</a></li>
<li>Click Create Pipeline</li>
<li>Name the pipeline</li>
<li>Service Role: New service role</li>
<li>Role Name: Pick a name</li>
<li>Allow AWS CodePipeline to create a role</li>
<li>Next</li>
<li>Source Provider: AWS CodeCommit</li>
<li>Repository Name: Select the repository name</li>
<li>Branch name: master</li>
<li>Change detection: Amazon CloudWatch Events</li>
<li>Next</li>
<li>Build Provider: AWS CodeBuild</li>
<li>Region: Canada (Central)</li>
<li>Project Name: Create Project
<ol>
<li>Project Name: Choose a name</li>
<li>Description: Add a description</li>
<li>Environment image: Managed Image</li>
<li>Operating System: Ubuntu</li>
<li>Runtimes: Standard</li>
<li>Image: aws/codebuild/standard:3.0</li>
<li>Image version: Always use the latest</li>
<li>Environment type: Linux</li>
<li>Privileged: True</li>
<li>Service Role: New service role</li>
<li>Role Name: default name is fine</li>
<li>Advanced Configuration - Environment variables (click Add to get more)
<ul>
<li>IMAGE_REPO_NAME: (your image name)</li>
<li>AWS_ACCOUNT_ID: (your numeric AWS account iD)</li>
<li>AWS_DEFAULT_REGION: ca-central-1</li>
</ul>
</li>
<li>Build specifications: Use a buildspec file</li>
<li>Buildspec name: Leave it blank so it uses buildspec.yml</li>
<li>CloudWatch logs: True</li>
<li>Group name: pick a name</li>
<li>Stream name: pick a name</li>
<li>S3 logs: false</li>
<li>Continue to CodePipeline</li>
</ol>
</li>
<li>Next</li>
<li>Skip deploy stage, pushing to ECR is enough</li>
<li>Create pipeline</li>
</ol>
<p>This will re-run the build that you tested before. It should pass again.</p>


<h1>Getting the certificate</h1>
<p>So far as I can tell, certbot works in the following way:</p>
<ol>
<li>Certbot is executed from the web-server CLI, and told to download a cert for
an FQDN. Let's say it's getting a cert from <code>example.com</code>.</li>
<li>Certbot opens up a port, I think 443, and listens for requests from the
LetsEncrypt service.</li>
<li>Certbot calls out to LetsEncrypt's service, letting the service know it's
listening on <code>example.com</code> and ready to prove it owns that domain.</li>
<li>LetsEncrypt sends some secret, or maybe collects some secret from the
listening certbot service, proving that certbot's server does indeed own
this domain name.</li>
<li>LetsEncrypt signs a cert and makes it available. Certbot downloads the cert
to the local server.</li>
</ol>
<h2>Configure DNS</h2>
<p>First, ensure that your DNS A record, such as <code>example.com</code>, resolves to
the IP address your server is using. This needs to be an internet accessible
IP address, not an internal RFC-1918 address.</p>
<h2>Install Certbot</h2>
<p>This guide assumes you're using Ubuntu Server.</p>
<pre><code class="language-bash">add-apt-repository ppa:certbot/certbot
apt-get update
apt-get install certbot
</code></pre>
<h2>Download the certificate</h2>
<p>Set the <code>site</code> variable here to your DNS entry which will point to this server.</p>
<pre><code class="language-bash">site="example.com"
certbot certonly --standalone --preferred-challenges http -d $site
</code></pre>
<h2>Use the certificate</h2>
<p>You can find the certificates as files on your server.</p>
<pre><code class="language-bash">> cd /etc/letsencrypt/live/$site
README  cert.pem  chain.pem  fullchain.pem  privkey.pem
</code></pre>
<p>Copy these certs to your web service and use them like any other cert.</p>
<p>Note that if you want to use these certs for HAProxy, you need to combine
<code>cert.pem</code> and <code>privkey.pem</code> into a single file.</p>
<pre><code class="language-bash">cat cert.pem privkey.pem > haproxy-$site.pem
</code></pre>


<h1>Create a Firebase Project</h1>
<p>Open the <a href="https://console.firebase.google.com/">Firebase console</a> and follow
the wizard. It's pretty straightforward. I bound my project to a Google Cloud
Platform project I was already using and used the Blaze plan.</p>
<p>The wizard is intuitive, I won't cover its steps here.</p>
<p>Navigate into your project and "Add Firebase to your web app". Once that's done
you can use the CLI to deploy the app.</p>
<hr>
<h1>Install Firebase CLI</h1>
<p>Installing Firebase CLI requires the <code>npm</code> command from NodeJS.
If you want to configure your dev environment in Ansible, check out my
<a href="http://localhost:8000/nodejs-ansible.html">Installing Node.js with Ansible</a>
post.</p>
<pre><code class="language-bash"># Add the key for the nodesource Apt repository
wget -qO https://deb.nodesource.com/gpgkey/nodesource.gpg.key | sudo apt-key add -

# Add the nodesource apt repo
echo "deb https://deb.nodesource.com/node_13.x bionic main" > /etc/apt/sources.d/nodesource
apt-get update

# Install NodeJS to get the npm command
apt-get install -y nodejs

# Install firebase-tools
npm install firebase-tools
</code></pre>
<h2>Logging In</h2>
<p>If your CLI and browser are on the same system, just run <code>firebase login</code> and
ignore the rest of this section.</p>
<p>I haven't found a way to log into Firebase CLI that doesn't need a web browser.
The limitation was inconvenient since I develop inside an Ubuntu server VM.</p>
<p>The options I figure would work are:</p>
<ol>
<li>Using a local Vagrant VM, forward the port that the oauth site tries to
redirect to on localhost. This was the easiest for my local dev environment,
so it's what I did. You copy the URL from your CLI into your workstation's
browser, then have it redirect <code>http://localhost:9005/</code> to your VM. Here's
an example of how to forward the port in a Vagrantfile:
<code>config.vm.network :forwarded_port, host: 9005, guest: 9005</code></li>
<li>Install Chrome in Ubuntu server then launch it with
<a href="/x11-forwarding-ubuntu.html">X11 Forwarding</a></li>
<li>Create a reverse SSH tunnel from a port on your laptop to the development
server. <a href="/dial-home-device.html">Here's a post where I've done SSH port forwarding</a></li>
<li>Use the <code>firebase login:ci</code> command and the
<code>--token</code> argument. This option was my fallback, but you still need to
install <code>firebase-cli</code> on your actual workstation with a browser to use it.</li>
</ol>
<p>Once successfully authenticated your browser will show a message saying
<code>Woohoo! Firebase CLI Login Successful</code>.</p>
<hr>
<h1>Deploy Hello World</h1>
<p>Make an index file in a directory for this app</p>
<pre><code class="language-bash">mkdir app
cd app
mkdir public
echo "&#x3C;h1>Hello World!&#x3C;/h1>" > public/index.html
</code></pre>
<h2>firebase init</h2>
<p>Initialize the directory as an application. This will present you with an
intuitive wizard.</p>
<pre><code class="language-bash">firebase init

# When prompted,
#  choose only Hosting
#  keep your index file
#  decide if you'll use a single page app
</code></pre>
<h3>Single-page app setting</h3>
<p>Running <code>firebase init</code> generates a <code>firebase.json</code> file for your project.
In single-page mode, this file will rewrite all traffic to index.html. You can
edit the file to change this behaviour. The rewrite block in <code>"hosting":</code> looks
like this:</p>
<pre><code class="language-json">    "rewrites": [{
      "source": "**",
      "destination": "/index.html"
    }],
</code></pre>
<h2>firebase deploy</h2>
<p>Deploy the default app</p>
<pre><code class="language-bash">firebase deploy
</code></pre>
<p>The deploy command will print a Hosting URL. The Hello World site can be loaded
from there.</p>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<h1>Create &#x26; Terminate an EC2 Instance from Python</h1>
<p>The IAM user tested for this task has the <code>AmazonEC2FullAccess</code> policy.</p>
<h2>Create</h2>
<p>Before you begin, determine the following:</p>
<ul>
<li>Image AMI: The example I provide here works, it's for Ubuntu 18.04 in Canada</li>
<li>Security group name</li>
<li>Instance name</li>
<li>Instance type</li>
</ul>
<pre><code class="language-python">import boto3

key_id = 'XXXXXXXXXXXXXX'
key_secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXX'

session = boto3.Session(
    aws_access_key_id=key_id,
    aws_secret_access_key=key_secret)

# Define the VM properties
image_id='ami-0d0eaed20348a3389' # bionic ca-central LTS
instance_name='Test9'
security_group_name = 'SupportService'
instance_type = 't2.micro'

# Put the properties into the expected format
tag_spec = [{
    'ResourceType': 'instance',
    'Tags': [{
        'Key': 'Name',
        'Value': instance_name }]}]
security_groups=[security_group_name]

# Reserve the instance
ec2_client = session.client('ec2')
reservation = ec2_client.run_instances(
        ImageId=image_id,
        MinCount=1,
        MaxCount=1,
        InstanceType=instance_type,
        KeyName='bwsupport',
        TagSpecifications=tag_spec,
        SecurityGroups=security_groups )

# Now the VM is creating. Optionally you can wait and inspect the instace
instance_data = reservation['Instances'][0]
iid = instance_data['InstanceId']
print(f'Reserved instance "{instance_name}" as {iid}. Waiting until running.')
ec2_resource = session.resource('ec2')
instance = ec2_resource.Instance(iid)
instance.wait_until_running()
state = instance.state['Name']
print(f'Instancei {iid} is {state} - Public IP: {instance.public_ip_address}')
</code></pre>
<h2>Terminate</h2>
<p>You can use the same code as above to get the <code>instance</code> variable.
From there, it's this easy:</p>
<pre><code class="language-python">instance.terminate()
instance.wait_until_terminated()
</code></pre>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to restrict access to a Lambda function using AWS IAM
roles. It builds on the function written <a href="/aws-lambda.html">in my last post</a></p>
<hr>
<h3>Table of Contents</h3>
<ul>
<li><a href="#create-a-policy-for-your-lambda-function">Create a policy for your lambda function</a>
<ul>
<li><a href="#get-the-arn-of-your-function">Get the ARN of your function</a></li>
<li><a href="#create-the-policy">Create the policy</a></li>
</ul>
</li>
<li><a href="#invoke-lambda-from-python">Invoke Lambda from Python</a></li>
</ul>
<h1>Create a policy for your lambda function</h1>
<h2>Get the ARN of your function</h2>
<p>Navigate to your function in the AWS Lambda Console. On the top right, the ARN
is displayed. Copy it, you'll use it in the policy.</p>
<h2>Create the policy</h2>
<p>Open the IAM Console and create a new policy.
The JSON looks like this. Note the ARN copied from the Function.</p>
<pre><code class="language-JSON">{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LambdaInvokeHelloWorld",
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction",
                "lambda:InvokeAsync"
            ],
            "Resource": "arn:aws:lambda:ca-central-1:850047500507:function:HelloWorld"
        }
    ]
}
</code></pre>
<p>Name the policy and assign it to your user. This user should have a Key ID and
Secret Key.</p>
<hr>
<h1>Invoke Lambda from Python</h1>
<p>Note that the payloads are in byte format.</p>
<pre><code class="language-python">import json
import boto3

key_id = 'XXXXXXXXXXXXXXXXX'
key_secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

session = boto3.Session(
    aws_access_key_id=key_id,
    aws_secret_access_key=key_secret)

client = session.client('lambda')

body = {'name': 'Kyle'}
payload = str.encode(json.dumps(body))
response = client.invoke(FunctionName='HelloWorld', Payload=payload)
print(response['Payload'].read())
</code></pre>
<p>If the user identified by that key has the above permission, they'll be able
to execute this function. Otherwise they'll get a nice permissions error.</p>


<p>This post covers the basics of using AWS Lambda functions.</p>
<h3>Table of Contents</h3>
<ul>
<li><a href="#pricing">Pricing</a></li>
<li><a href="#hello-world-lambda-function">Hello World Lambda Function</a>
<ul>
<li><a href="#create-the-function">Create the function</a></li>
<li><a href="#define-a-test-event">Define a test event</a></li>
<li><a href="#write-the-function-code">Write the function code</a></li>
<li><a href="#define-a-trigger">Define a trigger</a>
<ul>
<li><a href="#test-the-api">Test the API</a></li>
</ul>
</li>
<li><a href="#what-are-layers">...What are layers?</a></li>
</ul>
</li>
</ul>
<h1>Pricing</h1>
<p>For the most accurate data, see the <a href="https://aws.amazon.com/lambda/pricing/">official pricing documentation</a>.</p>
<p>Lambda charges you for each request invocation, and also for how long the
request took to run. These prices are probably in USD.</p>
<ul>
<li><strong>Region</strong>: Canada (Central)</li>
<li><strong>Requests</strong>: $0.20 / 1M Requests, or $0.0000002 / Request</li>
<li><strong>Duration (General)</strong>: $0.000016667 for every GB-second</li>
<li><strong>Duration (128MB)</strong>: $0.00000208/sec, or ~$0.0075/hour.
I think if it as 3/4 of a cent per hour.</li>
</ul>
<hr>
<h1>Hello World Lambda Function</h1>
<h2>Create the function</h2>
<ol>
<li>In your browser, open the AWS console <a href="https://console.aws.amazon.com/lambda">Lambda Page</a>.</li>
<li>Click the orange Create Function button</li>
<li>Author from scratch</li>
<li>Function name: HelloWorld</li>
<li>Runtime: Python 3.7</li>
<li>Permissions: Create a new role with basic lambda permissions</li>
<li>click Create Function on the bottom right</li>
</ol>
<h2>Define a test event</h2>
<ol>
<li>In the dropdowns along the top, one is called "Select a test event".</li>
<li>Click it, and choose Configure test events</li>
<li>Event template: Hello World</li>
<li>Event name: <code>HelloWorldTest1</code></li>
<li>Enter some sample data and click Create.</li>
</ol>
<p><strong>Sample Data</strong>:</p>
<pre><code class="language-JSON">{
  "body": "{\"name\": \"Test User\"}"
}
</code></pre>
<h2>Write the function code</h2>
<p>Click on the orange lambda icon with the text "HelloWorld" to open the
"Function code" form. Expand it to full screen (button on top right) so the
Save and Test buttons appear along the top.</p>
<p>Write your Hello World function. This was the shortest one I could come up with
that wouldn't throw 500's if you called it with the wrong data.</p>
<pre><code class="language-python">import json

def lambda_handler(event, context):
    name = 'UNDEFINED'
    if 'body' in event:
        try:
            ebody = json.loads(event['body'])
            name = ebody['name']
        except:
            pass
    greeting = f'Hello {name}!'
    body = json.dumps(greeting)
    return {
        'statusCode': 200,
        'body': body
    }
</code></pre>
<p>Click Save, then Test. You should get a response with body <code>'Hello TestUser'</code>.</p>
<h2>Define a trigger</h2>
<ol>
<li>Click the "+ Add trigger" button.</li>
<li>API Gateway</li>
<li>REST API</li>
<li>Security: Open (in this case)</li>
<li>Additional Settings: Leave the defaults</li>
</ol>
<p>Now if you click on the purple API Gateway button it will open the API Gateway
form. In there, you can get the URL. Copy the link address.</p>
<h3>Test the API</h3>
<p>From bash:</p>
<pre><code class="language-bash">curl \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{"name": "MyName"}' \
  https://wg8p74asah.execute-api.us-east-1.amazonaws.com/default/HelloWorld
</code></pre>
<p>It should reply "Hello MyName".</p>
<h2>...What are layers?</h2>
<p>Prominently displayed beneath the function's name in the Designer form
is a button to manage Layers.</p>
<p><a href="https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html">Here's their official documentation</a>.</p>
<p>Layers are dependencies like archives and libraries.
Any one function can use up to 5 of them, and they have a 250MB limit.
They land in <code>/opt</code> when extracted.</p>
<p>We don't need layers for this Hello World app.</p>


<p><strong>Note:</strong> This is usually a bad idea. You should really get the team who runs
the CA to send you the certificate in case some <em>bad guy</em> is doing the MITM and
not the local security team. Use this procedure with caution.</p>
<hr>
<h1>Get the signing certificate</h1>
<p>Use <code>openssl</code> to print the certificate data.</p>
<pre><code class="language-bash">openssl s_client -connect google.com:443 -showcerts
</code></pre>
<p>In the output you'll see, among other things, some certificates.
They look like this:</p>
<pre><code class="language-text">-----BEGIN CERTIFICATE-----
blaBLAbla
-----END CERTIFICATE-----
</code></pre>
<p>Find the one signed by your local CA. Copy and paste it into a new file,
such as <code>example.com.crt</code>. The <code>.crt</code> file extension is required.
Don't copy anything before or after the BEGIN and END lines.</p>
<hr>
<h1>Trust the certificate</h1>
<p>As root, move the certificate file to <code>/usr/local/share/ca-certificates</code>,
then run <code>update-ca-certificates</code>.</p>
<pre><code class="language-bash">mv example.com.crt /usr/local/share/ca-certificates
update-ca-certificates
</code></pre>
<p>That's it. Now your system will trust certs signed by that CA too.</p>


<h3>Table of Contents</h3>
<ul>
<li><a href="#prepare-the-image-in-guest">Prepare the image in-guest</a>
<ul>
<li><a href="#remove-vmware-tools">Remove VMWare Tools</a></li>
<li><a href="#install-cloudbase-init--sysprep-if-needed">Install cloudbase-init &#x26; sysprep (if needed)</a></li>
<li><a href="#shut-down-the-vm--export-the-drive">Shut down the VM &#x26; Export the drive</a></li>
</ul>
</li>
<li><a href="#inject-the-kvm-drivers">Inject the KVM Drivers</a></li>
</ul>
<h1>Prepare the image in-guest</h1>
<p>From VMWare or HyperV, launch your template Windows VM and log in as an
administrative user. Make any firewall changes you need, install any packages,
and ensure that RDP is enabled.</p>
<h2>Remove VMWare Tools</h2>
<p>If the guest has VMWare tools installed, remove them. They won't be needed
where we're going.</p>
<h2>Install cloudbase-init &#x26; sysprep (if needed)</h2>
<p>This isn't so much for KVM as it is for OpenStack, but cloudbase-init is the
Windows version of cloud-init. This will set your hostname and such when the VM
is created.</p>
<ol>
<li><a href="https://cloudbase.it/cloudbase-init/">Download the Cloudbase-init installer</a></li>
<li>Run through the installer.
<ol>
<li>Don't run as Local System.</li>
<li>Don't name your user Administrator, the named user is made by this
service to run the startup tasks.</li>
</ol>
</li>
<li>At the end of the installation, you'll be prompted to sysprep. If this VM
will be re-used as a template and not just a one-off VM, then sysprep it.</li>
</ol>
<h2>Shut down the VM &#x26; Export the drive</h2>
<p>Shut down the VM and grab the VMDK or VHDX file it booted from. Copy that file
to a directory on your computer, such as <code>C:\Temp</code>.</p>
<hr>
<h1>Inject the KVM Drivers</h1>
<ol>
<li><a href="https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/virtio-win.iso">Download the VirtIO ISO</a></li>
<li>Mount the ISO to your workstation. In these examples, it's E:</li>
<li>Open PowerShell as admin and verify that the file is where you put it. You
can show some basic info about it like this: <code>get-vhd C:\Temp\Win10.vhdx</code></li>
<li>Create a mount-point for Dism: <code>mkdir C:\mount</code></li>
<li>Mount the volume with Dism:
<code>Dism /mount-image /ImageFile:C:\Temp\Win10.vhdx /Index:1 /MountDir:C:\mount</code></li>
<li>Inject the <code>viostor</code> driver:
<code>Dism /image:C:\mount /Add-Driver /Driver:E:\viostor\w10\amd64 /Recurse</code></li>
<li>Inject the <code>Baloon</code> driver:
<code>Dism /image:C:\mount /Add-Driver /Driver:E:\Balloon\w10\amd64 /Recurse</code></li>
<li>Inject the <code>NetKVM</code> driver:
<code>Dism /image:C:\mount /Add-Driver /Driver:E:\NetKVM\w10\amd64 /Recurse</code></li>
<li>Confirm the drivers injected: <code>Dism /image:C:\mount /Get-Drivers</code></li>
<li>Unmount the image: <code>Dism /Unmount-image /MountDir:c:\mount /commit</code></li>
</ol>
<p>Now the image is ready to be converted to raw/qcow2 and uploaded to OpenStack.</p>
<p><strong>Note</strong>: CloudBase has released a Windows version of <code>qemu-img</code> so you can do
the conversions on Windows now without VirtualBox tools. You can download it
<a href="https://cloudbase.it/qemu-img-windows/">here</a></p>


<p>When you use a tool like <code>dd</code> or <a href="https://www.balena.io/etcher/">Etcher</a> to
turn a USB
stick into boot media, usually for installing an operating system, any attempt
at formatting that disk later will be met with a drive that only has a few GB
total. This is particularly annoying when you use something like a 100+GB USB
stick for that.</p>
<p>The cause of this problem is that a smaller partition is created on the drive
than the drive could potentially hold. To fix the drive you need to
removing all the partitions.</p>
<p>List your devices. Note your USB stick (example: <code>/dev/disk2</code>)</p>
<pre><code class="language-bash">diskutil list
</code></pre>
<p>Try to fix it using diskutil:</p>
<pre><code class="language-bash">diskutil eraseDisk free EMPTY /dev/disk2
</code></pre>
<p>Often that's good enough, and the drive now shows up in Disk Utility with its full capacity.</p>
<hr>
<p>If that fails, you can overwrite the partition/filesystem's metadata at the start of the disk with zeros.</p>
<p>If the drive is mounted, unmount it.</p>
<pre><code class="language-bash">diskutil umountDisk /dev/disk2
</code></pre>
<p>Delete the partition. Note that I tried <code>gpt</code> first but it threw an error
saying <code>gpt show: error: bogus map</code>. Also, as far as I can tell, <code>diskutil</code>
won't actually erase a partition and instead will only reformat it. Strangely,
even <code>fdisk</code> seems to have hidden the option to delete a partition. Guess Apple
<strong>REALLY</strong> doesn't trust its users. That's too bad, since I ended up doing
something even more dangerous.</p>
<p>Overwrite the partition and filesystem data in the first 1GB of the USB drive
with zeros. Be <strong>really really sure</strong> that you select the right disk here.
You can 100% toast your workstation's boot disk if you accidentally specify it.</p>
<pre><code class="language-bash">sudo dd if=/dev/zero of=/dev/disk2 count=1 bs=1GB
</code></pre>
<p>If that doesn't work, set <code>bs=1MB</code> and remove the <code>count=1</code>. You might need to
delete the end of it instead. That'll take a while.</p>


<p>This post covers how I use Ansible to install Node.js. I googled around for a
solution to this and thought the solutions I found looked way too complicated.</p>
<hr>
<h3>Table of Contents</h3>
<ul>
<li><a href="#the-problem">The Problem</a>
<ul>
<li><a href="#what-about-nvm">What about NVM?</a></li>
<li><a href="#chris-leanodejs">chris-lea/node.js</a></li>
<li><a href="#curl-x--bash">curl <em>x</em> | bash</a></li>
</ul>
</li>
<li><a href="#the-solution">The Solution</a></li>
<li><a href="#ansible-example">Ansible Example</a>
<ul>
<li><a href="#add-nodesource-repository-to-apts-sourceslist">Add nodesource repository to Apt's sources.list</a></li>
<li><a href="#install-nodejs">Install Node.js</a></li>
<li><a href="#use-npm-normally">Use NPM normally</a></li>
</ul>
</li>
</ul>
<h1>The Problem</h1>
<p>In theory, installing Node.js should be as easy as running
<code>apt-get install nodejs</code>. The problem is that Node.js versions seem to move
pretty quickly and the one distributed by Cannonical in Ubuntu Bionic is
already too old to run some applications. In my case I was trying to install
<code>firebase-tools</code>, the CLI for Google Firebase, and it wouldn't run with Apt's
version of the package.</p>
<h2>What about NVM?</h2>
<p>NVM, or Node Version Manager, is a nice little application for installing
various versions of Node.js. It kind of reminds me of python virtualenv.
The problem with NVM is that it relies on setting environment variables and
defining the <code>nvm</code> function using the <code>.bashrc</code> file. Ansible doesn't read
that file when it runs.</p>
<p>In theory, the Ansible <code>npm</code> module supports accepting an executable path,
specifically to support NVM, but I had to do some ugly shell commands to
collect that path and even afterwards it wouldn't actually work. Specifically
running the <code>npm</code> binary deployed by NVM without scoping the <code>.bashrc</code> file
throws stack traces instead of installing your app.</p>
<h2>chris-lea/node.js</h2>
<p>A lot of the sites I found were adding the Apt PPA <code>'ppa:chris-lea/node.js'</code>.
I don't know what that is, but it doesn't sit well with me. Just seems like
something that wont age well, might be insecure, and is probably fragile.</p>
<p>To make matters worse, when I tried it out it didn't seem to support Bionic.</p>
<h2>curl <em>x</em> | bash</h2>
<p>Some examples I saw mentioned just curling various URLs and piping them to bash
to run a script.
Sometimes I'm OK with doing this, but that sort of installation method doesn't
really seem appropriate for an Ansible playbook. If nothing else it'd be hard
to make those tasks idempotent. Plus, we can do better.</p>
<hr>
<h1>The Solution</h1>
<p>If you look at the <a href="https://github.com/nodesource/distributions">nodesource/distributions GitHub page</a>,
they have some really nice documentation in their <code>README.md</code>. Under the
<strong>Manual Installation</strong> section you can see that they have a nice, stable
looking Apt repository URL. This worked great for me, and is trivial to
set up Ansible to work with.</p>
<hr>
<h1>Ansible Example</h1>
<h2>Add nodesource repository to Apt's sources.list</h2>
<pre><code class="language-yaml">- name: "Add nodejs apt key"
  apt_key:
    url: https://deb.nodesource.com/gpgkey/nodesource.gpg.key
    state: present

- name: "Add nodejs 13.x ppa for apt repo"
  apt_repository:
    repo: deb https://deb.nodesource.com/node_13.x bionic main
    update_cache: yes
</code></pre>
<h2>Install Node.js</h2>
<pre><code class="language-yaml">- name: "Install nodejs"
  apt:
    update_cache: yes
    name: nodejs
    state: present
</code></pre>
<h2>Use NPM normally</h2>
<p>With this version of Node.js, I was able to install and run the application.</p>
<pre><code class="language-yaml">- name: "Install NPM-distributed command-line tools"
  npm:
    global: yes
    name: "{{ item }}"
  with_items:
    - firebase-tools
</code></pre>


<p><strong>This post is linked to from the <a href="/blog-website.html">Blog Website Project</a></strong></p>
<hr>
<p>This static website is hosted using Google Cloud Storage, which is great except
that it doesn't support HTTPS. There's no really good excuse to not support
HTTPS on a website today.</p>
<p>To close that gap, CloudFlare can handle the HTTPS negotiation for you and it
takes no time at all to set up.</p>
<h1>Configure HTTPS in CloudFlare</h1>
<p>This guide assumes CloudFlare is already acting as a CDN for your site.</p>
<ol>
<li>Go to the CloudFlare DNS page and find the A record or CNAME for your FQDN</li>
<li>Ensure that the cloud icon is yellow and not grey, indicating that the CDN
is in use and not bypassing CloudFlare.</li>
<li>Click on the padlock labelled SSL/TLS</li>
<li>Go to the Overview tab</li>
<li>Select Flexible.... Basically that's it, you're good.
HTTPS is now running on your site. CloudFlare is a really great product!
There's more that can be done, though.</li>
<li>Go to the Edge Certificates tab</li>
<li>Set "Always Use HTTPS" to On</li>
</ol>
<p>In my testing the "Always Use HTTPS" redirect change didn't take effect
immediately. After about 15 minutes 301's were coming back as expected.</p>
<pre><code class="language-text">vagrant@dev-env:~/$ curl -i http://kyle.pericak.com
HTTP/1.1 301 Moved Permanently
Date: Tue, 03 Dec 2019 00:52:25 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Tue, 03 Dec 2019 01:52:25 GMT
Location: https://kyle.pericak.com/
Server: cloudflare
CF-RAY: 53f18ecdec10b677-YWG
</code></pre>
<p><a href="https://support.cloudflare.com/hc/en-us/articles/204144518">You can find more details and official documentation here.</a></p>


<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<p>In this project I'll be trying out each of the Google Cloud Platform's
services. I'll be updating it pretty regularly.</p>
<hr>
<h1>Project Posts &#x26; Progress</h1>
<p>This project includes, or will include, the following posts.
If any aren't finished, check back later! I'll also certainly be adding more
as time goes on.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left"></td>
<td align="left"><strong>Continuous Integration</strong></td>
</tr>
<tr>
<td align="left">2019-08-06</td>
<td align="left"><a href="/google-cloud-build.html">Google Cloud Build: Basics</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Database</strong></td>
</tr>
<tr>
<td align="left">2019-10-14</td>
<td align="left"><a href="/gcp-firestore-python.html">Google Cloud Firestore Basics - Python</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Containers</strong></td>
</tr>
<tr>
<td align="left">2019-08-05</td>
<td align="left"><a href="/google-container-registry.html">Google Container Registry: Basics</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Cloud Storage</strong></td>
</tr>
<tr>
<td align="left">2019-08-09</td>
<td align="left"><a href="/google-cloud-storage-website.html">Hosting websites on Cloud Storage</a></td>
</tr>
</tbody>
</table>


<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<p>In this project I'll be trying many of AWS's services.</p>
<h1>Project Posts &#x26; Progress</h1>
<p>This project includes the following posts.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left"></td>
<td align="left"><strong>Running Servers on AWS</strong></td>
</tr>
<tr>
<td align="left">2020-01-09</td>
<td align="left"><a href="/aws-ec2-python.html">Create &#x26; Terminate EC2 Instances from Python</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Running Containers on AWS</strong></td>
</tr>
<tr>
<td align="left">2019-08-26</td>
<td align="left"><a href="/aws-ecr.html">Save a Docker image to Elastic Container Registry (ECR)</a></td>
</tr>
<tr>
<td align="left">2019-09-10</td>
<td align="left"><a href="/aws-fargate.html">Run a Docker container Using AWS Fargate</a></td>
</tr>
<tr>
<td align="left">2020-01-15</td>
<td align="left"><a href="/aws-codepipeline-ecr.html">CodePipeline: Continuous Delivery to AWS ECR</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Cloud Storage</strong></td>
</tr>
<tr>
<td align="left">2019-09-26</td>
<td align="left"><a href="/aws-s3-python.html">Using AWS S3 from Python</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Serverless Applications</strong></td>
</tr>
<tr>
<td align="left">2020-01-08</td>
<td align="left"><a href="/aws-lambda.html">Lambda basics</a></td>
</tr>
<tr>
<td align="left">2020-01-09</td>
<td align="left"><a href="/aws-lambda-iam.html">IAM for Lambda</a></td>
</tr>
<tr>
<td align="left">2020-01-21</td>
<td align="left"><a href="/aws-api-gateway-iam.html">IAM Authentication using API Gateway</a></td>
</tr>
</tbody>
</table>


<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<h4>Table of contents</h4>
<ul>
<li><a href="#what-is-openstack">What is OpenStack?</a></li>
<li><a href="#project-posts--progress">Project Posts &#x26; Progress</a></li>
<li><a href="#why-use-an-openstack-private-cloud">Why Use an OpenStack Private Cloud?</a></li>
<li><a href="#complexity-and-openstack-ad">Complexity and OpenStack (Ad)</a></li>
<li><a href="#installing-openstack">Installing OpenStack</a>
<ul>
<li><a href="#non-production-installation-options">Non-Production Installation Options</a>
<ul>
<li><a href="#devstack">DevStack</a></li>
<li><a href="#small-physical-server--kolla-ansible">Small Physical Server + Kolla-Ansible</a></li>
<li><a href="#virtual-server--kolla-ansible">Virtual Server + Kolla-Ansible</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#containerized-openstack">Containerized OpenStack</a></li>
</ul>
<h1>What is OpenStack?</h1>
<p><a href="https://www.openstack.org/">OpenStack</a> is a collection of open-source projects
that work together to form a private cloud solution. It's by far the
biggest open source private cloud option available. The <a href="https://www.openstack.org/foundation/">OpenStack Foundation</a>
has over 105,000 community members from 187 countries around the world.</p>
<p>At a high level, you can think of it as an abstraction layer that sits on top
of other popular cloud technologies such as storage clusters, hypervisors, and
network solutions. For example, KVM (the popular Linux hypervisor) is not part
of OpenStack, but it's used by <strong>Nova</strong>, a component of OpenStack which can
orchestrate against KVM, VMWare, HyperV, and Xen.</p>
<hr>
<h1>Project Posts &#x26; Progress</h1>
<p>This project includes the following posts.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">2019-08-13</td>
<td align="left"><a href="/openstack-aio-ka-metal.html">Install OpenStack on Metal - Intel NUC</a></td>
</tr>
<tr>
<td align="left">2019-08-11</td>
<td align="left"><a href="/openstack-aio-ka-vm.html">Install OpenStack inside a VM</a></td>
</tr>
<tr>
<td align="left">2019-08-26</td>
<td align="left"><a href="/openstack-kolla-custom-plugin.html">Modifying OpenStack Kolla Docker Images</a></td>
</tr>
<tr>
<td align="left">2019-12-03</td>
<td align="left"><a href="/openstack-ansible.html">Operating OpenStack from Ansible</a></td>
</tr>
<tr>
<td align="left">2020-01-25</td>
<td align="left"><a href="/openstack-cloudinit-powershell.html">OpenStack Cloud-Init Powershell Example</a></td>
</tr>
<tr>
<td align="left">2019-12-17</td>
<td align="left"><a href="windows-kvm-drivers">Injecting KVM Drivers into Windows</a></td>
</tr>
</tbody>
</table>
<hr>
<h1>Why Use an OpenStack Private Cloud?</h1>
<ol>
<li><strong>Price at scale:</strong> At small scales, private clouds lose on price
against the public clouds. As you start to ramp up though, the public clouds
start to really nickle and dime you. Private clouds have higher up-front
costs but they win out in the long-run.</li>
<li><strong>Predictable Cost:</strong> Spin up a few services on AWS or Azure. How much will
they cost you each month? As it stands, you're guess is going to be wrong.
After the free trial periods and special offers stop muddying the water,
you'll usually discover you're spending more than you thought.</li>
<li><strong>Hardware Re-Use:</strong> Many IT organizations already have hardware sitting
around running expensive hypervisor software. OpenStack is free and can use
that existing hardware, giving you more for less.</li>
<li><strong>Data Sovereignty:</strong> For legal and security reasons, many organizations
must know exactly where their data is sitting at rest. With private clouds,
there's no ambiguity.</li>
<li><strong>Behind Your Firewall:</strong> So you spent a ton of money on some next
generation firewall technologies with intrusion detection/prevention, and
trained the staff up to use it. You run an arguably well secured
data-center. Running OpenStack keeps the cloud behind your existing security
measures.</li>
<li><strong>Latency:</strong> Particularly for edge clouds and VDI solutions, having the
cloud located right on-premises can hugely improve user experience.</li>
<li><strong>Shadow IT:</strong> Developers are going to develop, even if your organization
can't get them the tools they need in time. By offering your own in-house
cloud, you can help prevent
<a href="https://en.wikipedia.org/wiki/Shadow_IT">shadow IT</a> from emerging.</li>
<li><strong>Control:</strong> This one is a double-edged sword. Having more control over your
cloud can be its own benefit, but there's also value in letting the pros
handle it.</li>
<li><strong>Business as usual, without VMWare:</strong> Don't get me wrong, VMWare make a
great product... I've never heard anyone say they were happy with their
VMWare bill. It's insanely expensive when there's an open source alternative
that can check all the same boxes.</li>
</ol>
<hr>
<h1>Complexity and OpenStack (Ad)</h1>
<p>OpenStack is significantly more difficult to get started with than the public
clouds. You can't really install OpenStack without first understanding what
each of the main components does, and even then each one has its own quirks for
their installation. If you want to install in production, you also need to
figure out how to get it all highly available.</p>
<p>Unless you've got a huge team that's eager to tackle this complexity, I'm going
to use this opportunity to plug the software &#x26; solution I've helped build for
<a href="https://breqwatr.com">Breqwatr</a>. They'll stand up your cloud in a couple days
flat, teach your team how to use it, and keep the lights on - all while
charging significantly less than VMWare would cost for the same hardware.</p>
<p>Breqwatr, and companies like it, can be your trusted cloud partner and abstract
away that complexity by handling the installation, upgrades, monitoring, and
day-to-day operation of the cloud.</p>
<hr>
<h1>Installing OpenStack</h1>
<h2>Non-Production Installation Options</h2>
<p>If you're not looking for a production deployment and instead want to try
OpenStack out to learn it, there are a few good options.</p>
<h3>DevStack</h3>
<p>The standard developer option is probably <a href="https://docs.openstack.org/devstack/latest/">DevStack</a>.</p>
<p>In truth I haven't used this since I've always needed my OpenStack installs to
mirror a production-ready environment. If I try it out, I'll replace the above
link with one to a guide I write myself.</p>
<h3>Small Physical Server + Kolla-Ansible</h3>
<p>Another option is to install OpenStack on a physical server. I've documented
my experience doing this using images from the <a href="https://github.com/openstack/kolla">Kolla project</a>,
and Ansible automation from the <a href="https://github.com/openstack/kolla-ansible">Kolla-Ansible project</a>.</p>
<p>The nice thing about this approach is you can use a server at home, and once
you're done it can be a VM host as your private house cloud.</p>
<p><strong><a href="/openstack-aio-ka-metal.html">Installing OpenStack on a physical server</a></strong></p>
<h3>Virtual Server + Kolla-Ansible</h3>
<p>This option is great for development use if you already have a cloud available
or want to use a public cloud for testing OpenStack. The VMs you launch on it
will perform terribly though, and there are some networking complications.</p>
<p>Don't bother trying to launch Windows VMs at all in a nested virtualisation
scenario, you'll be lucky if they even boot.</p>
<p><strong><a href="/openstack-aio-ka-vm.html">Installing OpenStack on a Virtual Server</a></strong></p>
<hr>
<h1>Containerized OpenStack</h1>
<p>Managing OpenStack clusters is a pain. It has so many services, each with their
own dependencies and operating instructions, there's a lot that can go wrong.
One thing that's really helped with that is to containerize each service and
deploy them all using Docker.</p>
<p>The <a href="https://github.com/openstack/kolla">Kolla project</a> does such a great job
of that, that there's no need to do it yourself. That is, until you need to
change an image, for instance to install a Cinder plugin.</p>
<p><strong><a href="/copenstack-kolla-custom-plugin.html">Modifying OpenStack Kolla Docker Images</a></strong></p>


Kyle Pericak

"It works in my environment"

Categories

Tags