<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<h1>Workflow</h1>
<ol>
<li>Commit code to AWS CodeCommit</li>
<li>CloudWatch rule triggers CodePipeline</li>
<li>CodePipeline uses CodeBuild to build the Docker image</li>
<li>CodePipeline pushes the image to ECR</li>
</ol>
<hr>
<h1>Deploy Code to CodeCommit</h1>
<ol>
<li>Open <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codecommit/repositories">CodeCommit</a>.</li>
<li>Create a repository</li>
<li>Go to your IAM page and edit your user.
<ol>
<li>Look for "HTTPS Git credentials for AWS CodeCommit"</li>
<li>click Generate Credentials</li>
<li>Save your credentials.</li>
</ol>
</li>
<li>Grab the HTTPS URL for your repository and <code>git clone</code> it. Use your new
credentials.</li>
<li>Commit your code (including a Dockerfile) to the CodeCommit repository.</li>
</ol>
<hr>
<h1>Write buildspec.yml</h1>
<p>AWS CodeBuild will look for a <code>buildspec.yml</code> file to dictate its actions.
This is where we define how the image will be built. Put it in your CodeCommit
project's root directory.</p>
<p>Here's mine. This will build a docker image and tag it with a timestamp, then
push it to ECR as the timestamp and latest. The variables are defined inside
the build as Environment Variables.</p>
<pre><code class="language-yml"># buildspec.yml
version: 0.2

phases:
  install:
    runtime-versions:
      docker: 18
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - $(aws ecr get-login --no-include-email --region ca-central-1)
      - IMAGE_TAG=$(date +%Y.%m.%d.%I.%M)
  build:
    commands:
      - echo Build started on `date`
      - echo "Building the Docker image."
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest
</code></pre>
<hr>
<h1>Define the Pipeline</h1>
<h2>Cloud Build Role</h2>
<p><strong>Bad Practice Alert</strong>: These steps don't apply the principal of least
privilege. You should lock down your role to be allowed to do only the minimum
that it needs to do.</p>
<ol>
<li>In the IAM page, create a new role.</li>
<li>It will be used by CodeBuild.</li>
<li>Assing the following policies (again, more than it needs)
<ul>
<li>CloudWatchLogsFullAccess</li>
<li>AWSCodeBuildAdminAccess</li>
<li>AmazonS3FullAccess</li>
<li>AmazonEC2ContainerRegistryFullAccess</li>
<li>AWSCodeCommitFullAccess</li>
</ul>
</li>
<li>Name: Assign a name, such as <code>CloudBuild-Admin</code></li>
</ol>
<h2>Define CloudBuild project</h2>
<ol>
<li>Navigate to the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codebuild/projects">CloudBuild Console</a></li>
<li>Create build project</li>
<li>No build badge</li>
<li>Source provider: CodeCommit</li>
<li>Choose your repository</li>
<li>Choose your branch</li>
<li>Managed Image</li>
<li>Operating System: Ubuntu</li>
<li>Runtime: Standard</li>
<li>Image: aws/codebuild/standard:3.0</li>
<li>Image version: latest</li>
<li>Environment type: Linux</li>
<li>Privileged: true</li>
<li>Service Role: Existing service role. Use the one you just made.</li>
<li>Allow CodeBuild to modify this role: False (we gave it admin)</li>
<li>Additional Configuration - Environment variables (click Add to get more)
- IMAGE_REPO_NAME: (your image name)
- AWS_ACCOUNT_ID: (your numeric AWS account iD)
- AWS_DEFAULT_REGION: ca-central-1</li>
<li>Buildspec: Use a buildspec file</li>
<li>Artifacts: No artifacts</li>
<li>Logs: Give a group and stream name for CloudWatch</li>
<li>Create Build Project</li>
</ol>
<p>Now test the build by clicking "Start build". It should create your image.
You'll need to disable artifacts.</p>
<p>Once the build works and your image is pushed successfully to ECR, you can move
on to setting up a pipeline so this happens automatically.</p>
<h2>Create Pipeline Role</h2>
<p><strong>Bad Practice Alert</strong>: These steps don't apply the principal of least
privilege. You should lock down your role to be allowed to do only the minimum
that it needs to do.</p>
<ol>
<li>In the IAM page, create a new role.</li>
<li>It will be used by CodeBuild.</li>
<li>Assing the following policies (again, more than it needs)
<ul>
<li>CloudWatchLogsFullAccess</li>
<li>AWSCodeBuildAdminAccess</li>
<li>AmazonS3FullAccess</li>
<li>AmazonEC2ContainerRegistryFullAccess</li>
<li>AWSCodeCommitFullAccess</li>
</ul>
</li>
<li>Name: Assign a name, such as <code>CloudBuild-Admin</code></li>
</ol>
<h2>Define the pipeline</h2>
<ol>
<li>Open the <a href="https://ca-central-1.console.aws.amazon.com/codesuite/codepipeline/pipelines">CodePipeline console</a></li>
<li>Click Create Pipeline</li>
<li>Name the pipeline</li>
<li>Service Role: New service role</li>
<li>Role Name: Pick a name</li>
<li>Allow AWS CodePipeline to create a role</li>
<li>Next</li>
<li>Source Provider: AWS CodeCommit</li>
<li>Repository Name: Select the repository name</li>
<li>Branch name: master</li>
<li>Change detection: Amazon CloudWatch Events</li>
<li>Next</li>
<li>Build Provider: AWS CodeBuild</li>
<li>Region: Canada (Central)</li>
<li>Project Name: Create Project
<ol>
<li>Project Name: Choose a name</li>
<li>Description: Add a description</li>
<li>Environment image: Managed Image</li>
<li>Operating System: Ubuntu</li>
<li>Runtimes: Standard</li>
<li>Image: aws/codebuild/standard:3.0</li>
<li>Image version: Always use the latest</li>
<li>Environment type: Linux</li>
<li>Privileged: True</li>
<li>Service Role: New service role</li>
<li>Role Name: default name is fine</li>
<li>Advanced Configuration - Environment variables (click Add to get more)
<ul>
<li>IMAGE_REPO_NAME: (your image name)</li>
<li>AWS_ACCOUNT_ID: (your numeric AWS account iD)</li>
<li>AWS_DEFAULT_REGION: ca-central-1</li>
</ul>
</li>
<li>Build specifications: Use a buildspec file</li>
<li>Buildspec name: Leave it blank so it uses buildspec.yml</li>
<li>CloudWatch logs: True</li>
<li>Group name: pick a name</li>
<li>Stream name: pick a name</li>
<li>S3 logs: false</li>
<li>Continue to CodePipeline</li>
</ol>
</li>
<li>Next</li>
<li>Skip deploy stage, pushing to ECR is enough</li>
<li>Create pipeline</li>
</ol>
<p>This will re-run the build that you tested before. It should pass again.</p>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page contains operations-related mini-guides and minor posts.</p>
<hr>
<h3>Table of Contents</h3>
<ul>
<li><a href="#find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</a></li>
<li><a href="#convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</a></li>
<li><a href="#journalctl-commands">Journalctl commands</a></li>
<li><a href="#get-real-ram-use-of-processes">Get real RAM use of processes</a>
<ul>
<li><a href="#of-ceph">of ceph</a></li>
<li><a href="#of-vms">of VMs</a></li>
</ul>
</li>
<li><a href="#bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</a></li>
<li><a href="#create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</a>
<ul>
<li><a href="#use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</a></li>
</ul>
</li>
<li><a href="#test-mtu-from-ubuntu">Test MTU from Ubuntu</a></li>
<li><a href="#find-big-files-on-mac-os">Find big files on Mac OS</a></li>
<li><a href="#renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</a></li>
<li><a href="#create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</a></li>
<li><a href="#passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</a></li>
<li><a href="#vim-tasks">VIM Tasks</a>
<ul>
<li><a href="#find--replace">Find &#x26; Replace</a></li>
<li><a href="#spell-check-commands">Spell-Check commands</a></li>
</ul>
</li>
<li><a href="#use-an-insecure-docker-registry">Use an Insecure Docker Registry</a></li>
<li><a href="#ansible-ignore-errors">Ansible ignore errors</a></li>
<li><a href="#how-to-tail--f-on-dmesg">How to tail -f on dmesg</a></li>
<li><a href="#ubuntu-downgrade-a-package">Ubuntu Downgrade a package</a></li>
<li><a href="#make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</a></li>
<li><a href="#bash-print-all-but-first-line">Bash: Print all but first line</a></li>
<li><a href="#bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</a></li>
</ul>
<h1>Find which active-passive bond member is active</h1>
<pre><code>cat /proc/net/bonding/bond0
</code></pre>
<hr>
<h1>Convert p7b certificate to pem format</h1>
<pre><code class="language-bash">openssl pkcs7 -print_certs -in certnew.p7b -out cert.pem
</code></pre>
<hr>
<h1>Journalctl commands</h1>
<p>Watch the logs as they come in:</p>
<pre><code class="language-bash">journalctl -f
</code></pre>
<p>When journalctl logs too much and you need to pair it down:</p>
<pre><code class="language-bash"># to 30 days
journalctl --vacuum-time=30d

# to 500MB
journalctl --vacuum-size=500M
</code></pre>
<p>Show for a specific service</p>
<pre><code class="language-bash">journalctl ceph*
</code></pre>
<hr>
<h1>Get real RAM use of processes</h1>
<p>You can always use top or htop or whatever. I use this to specifically get the
info about those processes.</p>
<h2>of ceph</h2>
<pre><code>ps aux | grep ceph | grep -v -e grep -e qemu -e "\[" | while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); id=$(echo $line | awk '{print $16}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tid: $id,    \tRAM: ($size)"; done
</code></pre>
<h2>of VMs</h2>
<p>for KVM</p>
<pre><code>ps aux | grep qemu | grep -v -e grep| while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); guest=$(echo $line | awk '{print $13}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tguest: $guest,   \t\tRAM: ($size)" ; done
</code></pre>
<h1>Bypass Chrome's self-signed cert warning</h1>
<p>As of Mac Catalina, developing a site using self-signed certs is a bigger pain
than before. Yeah you can go get the key and add it, but that's a nuisance.
It looks like the Chrome team built something they call an
"interstitial bypass keyword" to deal with that. That's one cool feature name.</p>
<p><strong>Obvious disclaimer</strong> If you're not a developer doing this to access your own
site, don't do this.</p>
<p>To bypass the warning, type</p>
<pre><code>thisisunsafe
</code></pre>
<p>Apparently it used to be <code>badidea</code> but they rotated it since it got too well
known. They'll probably do it again someday. I suspect people are using this
on sites they "trust" without really knowing any better and getting hurt by
it.</p>
<hr>
<h1>Create a virtual loopback volume on Ubuntu</h1>
<p>Useful for simulating a real drive when building a test environment. I use this to test OpenStack in a VM as a Ceph or LVM back-end.</p>
<pre><code class="language-bash"># create empty file of all zeros
sudo dd if=/dev/zero of=/root/virtual-disk bs=1M count=512
# show list of loopback devices. Note its not there
losetup --list
# create the loopback device
losetup /dev/loop0 /root/virtual-disk
# confirm it worked
losetup --list
</code></pre>
<p>You can now treat it like a normal volume. Mount it, put a filesystem on it, whatever. Example:</p>
<pre><code class="language-bash">pvcreate /dev/loop0
vgcreate cinder-volumes /dev/loop0
</code></pre>
<h2>Use Systemd to re-map loopback on reboot</h2>
<p>I found <a href="https://unix.stackexchange.com/questions/418322/persistent-lvm-device-with-loopback-devices-by-fstab">this useful systemd unit example</a> that didn't quite work for me, but was close. Here's mine:</p>
<p><code>vi /etc/systemd/system/loops-setup.service</code></p>
<pre><code class="language-text">[Unit]
Description=Setup loopback devices

DefaultDependencies=no
Conflicts=umount.target

Requires=lvm2-lvmetad.service mnt-host.mount
Before=local-fs.target umount.target
After=lvm2-lvmetad.service mnt-host.mount

[Service]
ExecStart=/sbin/losetup /dev/loop0 /root/virtual-disk
ExecStop=/sbin/losetup -d /dev/loop0

RemainAfterExit=yes
Type=oneshot

[Install]
WantedBy=local-fs-pre.target
</code></pre>
<p>Then enable it</p>
<pre><code class="language-bash">systemctl enable loops-setup
</code></pre>
<p>Reboot, then check <code>vgdisplay</code> (if that's your use case). It'll be there.</p>
<hr>
<h1>Test MTU from Ubuntu</h1>
<p>You would not believe the strange problems that an MTU issue can be the root cause of.</p>
<pre><code class="language-bash">ping -M do -s 1472 &#x3C;remote server>
ping -M do -s 8900 &#x3C;remote server>
</code></pre>
<hr>
<h1>Find big files on Mac OS</h1>
<p>My SSD is always almost full. Here are useful commands for cleaning up:</p>
<pre><code class="language-bash"># Find all the files over 1G
sudo find / -type f -size +1G -exec ls -lh {} \; | awk '{ print $9 ": " $5 }' 2>/dev/null

# List files in Downloads directory by size
ls -lhS ~/Downloads
</code></pre>
<h1>Renew NTP Lease Ubuntu</h1>
<pre><code class="language-bash">service ntp stop
ntpd -gq
service ntp start
</code></pre>
<hr>
<h1>Create AWS ECR Registry If Not Found</h1>
<pre><code class="language-bash">aws ecr describe-repositories --repository-names $repo_name \
    || aws ecr create-repository --repository-name $repo_name
</code></pre>
<hr>
<h1>Passwordless sudo on Ubuntu Server</h1>
<p>Run <code>visudo</code> as root.</p>
<pre><code class="language-bash"># as root
export EDITOR=vim
visudo
</code></pre>
<p>Add the user (replace <code>exampleUser</code>) at the end.</p>
<pre><code class="language-text">exampleUser ALL=(ALL) NOPASSWD: ALL
</code></pre>
<p>For those who don't use vim, a quick reminder:
<code>i</code> enter to insert mode , <code>[esc]</code> to exit insert mode, <code>:wq</code> to save and quit.</p>
<hr>
<h1>VIM Tasks</h1>
<h2>Find &#x26; Replace</h2>
<pre><code class="language-bash"># replace findThis with ReplaceWithThis
:%s/findThis/ReplaceWithThis/g
</code></pre>
<h2>Spell-Check commands</h2>
<pre><code class="language-text"># enable/disable
:set spell spelllang=en_ca
:setlocal spell spelllang=en_ca
:set nospell

# next/last
]s
[s

# suggest change
z=

# add/remove word from dictionary
zg
zw`
</code></pre>
<hr>
<h1>Use an Insecure Docker Registry</h1>
<p>To use a registry with no HTTP cert:</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
    "insecure-registries" : ["myregistrydomain.com:5000"]
}
</code></pre>
<hr>
<h1>Ansible ignore errors</h1>
<p>The command is <code>ignore_errors: yes</code></p>
<p>Just throw this into the task that's failing. It will still fail, but the play
won't stop.</p>
<p>Example:</p>
<pre><code class="language-yml">- name: Ensure the docker service is running
  script: ../tools/validate-docker-execute.sh
    ignore_errors: yes
</code></pre>
<hr>
<h1>How to tail -f on dmesg</h1>
<pre><code class="language-bash">dmesg -wH
</code></pre>
<hr>
<h1>Ubuntu Downgrade a package</h1>
<pre><code class="language-bash"># Find the available versions under Versions:
apt-cache showpkg &#x3C;package>

# Install one of those
apt-get install &#x3C;package>=&#x3C;version>
</code></pre>
<hr>
<h1>Make Ansible run Python 3 on target nodes</h1>
<p>Useful for OS that ship with python3 but not python2.
Edit the inventory, find a group of the groups and apply this var.</p>
<pre><code class="language-ini">[baremetal:vars]
ansible_python_interpreter=/usr/bin/python3
</code></pre>
<hr>
<h1>Bash: Print all but first line</h1>
<p>Use <code>tail -n +2</code>.
This was unintuitive to me because +1 seems like it should do it.</p>
<pre><code class="language-bash"># Example
docker images | awk '{print $3}' | tail -n +2
</code></pre>
<hr>
<h1>Bash: Run code whenever a script exits</h1>
<p>Use trap.</p>
<ul>
<li><a href="http://redsymbol.net/articles/bash-exit-traps/">This is a nice post covering lots of examples</a>.</li>
<li><a href="https://github.com/GoogleCloudPlatform/endpoints-quickstart/blob/master/scripts/deploy_api.sh">Here's where I first saw it used</a></li>
</ul>
<pre><code class="language-bash">temp_file=$(mktemp)

cleanup() {
  rm "$temp_file"
}

trap cleanup EXIT
</code></pre>


<p>How to run a docker container on AWS ECS (Elastic Compute Cloud)  using the AWS
web console. A container from the Elastic Container Registry will be launched.
It will run on Amazon's <a href="https://aws.amazon.com/fargate/">Fargate Container-as-a-Service</a>.</p>
<hr>
<h3>Table of Contents</h3>
<ul>
<li><a href="#launch-a-container-on-fargate">Launch a container on Fargate</a></li>
<li><a href="#publish-containers-network-to-the-internet">Publish container's network to the internet</a></li>
<li><a href="#using-a-custom-domain-name">Using a custom domain name</a></li>
</ul>
<h1>Launch a container on Fargate</h1>
<ol>
<li>Upload an image to AWS ECR. <a href="/aws-ecr.html">See my guide on using AWS ECR</a>.</li>
<li>Open <a href="https://ca-central-1.console.aws.amazon.com/ecs/">AWS ECS</a></li>
<li>Under Container Definition, select Configure</li>
<li>Fill in the form
<ol>
<li>For the image URL, use the URL of the image uploaded to
ECR. I was testing a reverse proxy image, so it looked like this:
<code>000000000000.dkr.ecr.ca-central-1.amazonaws.com/test/revproxy:latest</code></li>
<li>I have my nginx reverse proxy a 256MB soft limit for RAM</li>
<li>For my reverse proxy I made port mappings for port 80 and 443</li>
<li>No need to fill out any of the advanced fields</li>
</ol>
</li>
<li>Fill out the Task Definition section, click Next</li>
<li>Define your service
<ol>
<li>Without a load-balancer, the only way to access your container will be by
using <a href="https://aws.amazon.com/blogs/compute/access-private-applications-on-aws-fargate-using-amazon-api-gateway-privatelink/">PrivateLink</a>.</li>
<li>During the initial cluster create wizard you can choose an Application
Load Balancer. That seems to work well.</li>
</ol>
</li>
<li>Name the Fargate cluster, next</li>
<li>Click create</li>
</ol>
<p>It will take a little while. Once its finished you can click on View Service.</p>
<hr>
<h1>Publish container's network to the internet</h1>
<p>Navigate to the newly created service and go to the Details tab. From there you
can see a Load Balancing section. It should have a link to the new load
balancer.</p>
<p>You can also access this page from the left nav-bar by going to Load Balancing,
then choosing the new load balancer.</p>
<p>In the Description tab here, you can find the DNS name that AWS has assigned
to this load balancer. Unfortunately you can't depend on the IP address behind
this FQDN to stay the same.</p>
<hr>
<h1>Using a custom domain name</h1>
<p>I'm working on that. Its not intuitive. If you know how, I'd like to hear about
it..</p>


<p>Docker supports shared memory. The default is only 64MB though which is TINY.</p>
<p>The small size became a problem during a containerized OpenStack upgrade.</p>
<p>This will be a really short post because I haven't done much playing with the
shared memory yet, I just think its really neat.</p>
<hr>
<h1>Change the /dev/shm size on the system</h1>
<p>This lets you change the size of shared memory on a host.</p>
<pre><code class="language-bash">mount -o remount,size=1G /dev/shm
</code></pre>
<hr>
<h1>Update Docker Daemon's Default Shared Memory Size</h1>
<p>This will update the default across the whole Docker service.</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
  "default-shm-size": "512M"
}
</code></pre>
<hr>
<h1>Set a Containers Shared Memory Size</h1>
<p>Here's how to change it for a specific container at runtime.</p>
<pre><code class="language-bash">docker run -it --shm-size=512M
</code></pre>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to make a Docker registry on AWS Elastic Container
Registry (ECR).
To secure the registry, least-privilege roles are created and assigned to
service account users in the AWS AIM tool</p>
<hr>
<h3>Table of contents</h3>
<ul>
<li><a href="#create-an-aws-ecr-repository">Create an AWS ECR Repository</a></li>
<li><a href="#create-an-aws-iam-user-for-the-registry">Create an AWS IAM User For the Registry</a>
<ul>
<li><a href="#create-groups-for-ecr-access">Create Groups for ECR Access</a>
<ul>
<li><a href="#make-a-group-with-write-access">Make a group with write access</a></li>
<li><a href="#make-a-read-only-group">Make a read-only group</a></li>
</ul>
</li>
<li><a href="#create-users-for-ecr-access">Create Users for ECR Access</a></li>
</ul>
</li>
<li><a href="#connect-docker-to-the-new--registry">Connect Docker to the new  Registry</a>
<ul>
<li><a href="#install-the-aws-cli">Install the AWS CLI</a></li>
<li><a href="#configure-the-aws-cli-user">Configure the AWS CLI USER</a></li>
<li><a href="#get-the-docker-login-command">Get the docker login command</a></li>
</ul>
</li>
<li><a href="#push-images-to-the-ecr-registry">Push Images to the ECR Registry</a></li>
</ul>
<h1>Create an AWS ECR Repository</h1>
<p>Navigate to the
<a href="https://ca-central-1.console.aws.amazon.com/ecr/repositories">AWS ECR Repositories page</a>
and click Create a repository.</p>
<p>Fill in the form to name your repository.</p>
<h1>Create an AWS IAM User For the Registry</h1>
<p>Its a good idea to have user accounts with write access, and others with
read-only access. This way you can push to the registry with your privileged
user but you never need to expose those credentials outside your development
or CI/CD environment.</p>
<h2>Create Groups for ECR Access</h2>
<p>First, review the AWS Group access levels <a href="https://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr_managed_policies.html">here</a>.</p>
<h3>Make a group with write access</h3>
<p>Users in this group can push to existing registries but they can't use the CLI
to make new ones.</p>
<ol>
<li>Go to the <a href="https://console.aws.amazon.com/iam">AWS IAM page</a></li>
<li>Click Groups on the sidebar</li>
<li>Name the group, ex: <code>ecr-write</code></li>
<li>Attach Policy: <code>AmazonEC2ContainerRegistryPowerUser</code>, next</li>
<li>Create Group</li>
</ol>
<h3>Make a read-only group</h3>
<p>Its the same procedure as above, but use the read-only policy instead:
<code>AmazonEC2ContainerRegistryReadOnly</code>.</p>
<p>Consider also making an admin group with the
<code>AmazonEC2ContainerRegistryFullAccess</code> permission.</p>
<h2>Create Users for ECR Access</h2>
<p>Make two users, one for pushing and one for pulling. You can share the user
will read access to anyone who needs to consume the images.</p>
<p>To make a user:</p>
<ol>
<li>Go to the <a href="https://console.aws.amazon.com/iam">AWS IAM page</a></li>
<li>Click Users on the sidebar</li>
<li>Add User</li>
<li>Enter user name, check Programmatic Access, next</li>
<li>Choose from the groups you just created to grant read/write access, next</li>
<li>Skip the tags, next</li>
<li>Create User. Be sure to save the secret access key in your password manager.</li>
</ol>
<h1>Connect Docker to the new  Registry</h1>
<h2>Install the AWS CLI</h2>
<p>The one from apt is really old and doesn't work. Use pip.</p>
<pre><code class="language-bash">apt-get install -y python python-pip
pip install aws awscli
</code></pre>
<p>Test it out</p>
<pre><code class="language-bash">python -m awscli help
</code></pre>
<h2>Configure the AWS CLI USER</h2>
<p>When prompted, enter your key, secret key, region (ca-central-1), and output
format (none):</p>
<pre><code class="language-bash">aws configure
</code></pre>
<h2>Get the docker login command</h2>
<p>Use the AWS CLI to fetch and execute the docker login command. You can also
export this command to a file to run it, but there's no point since the token
gets invalidated after 12 hours.</p>
<pre><code class="language-bash">$(aws ecr get-login --no-include-email --region ca-central-1)
</code></pre>
<hr>
<h1>Push Images to the ECR Registry</h1>
<p>If you click on the Push Commands button in the ECR registry page you'll get
examples programmatically generated for your registry.</p>
<p>Pushing is kind of unintuitive. Unlike a normal <code>registry:2</code> docker registry,
ECR won't automatically create your repositories for you. You have to manually
create each one. You can use the web UI, or log in as an admin user and run the
following:</p>
<pre><code class="language-bash">aws ecr create-repository --repository-name kolla/ubuntu-source-base
</code></pre>
<p>For scripting, this is a handy way of creating the repo if it doesn't exist:</p>
<pre><code class="language-bash">aws ecr describe-repositories --repository-names $repo_name \
  || aws ecr create-repository --repository-name $repo_name
</code></pre>
<p>Then you can push.</p>
<pre><code class="language-bash">docker push $myId.dkr.ecr.ca-central-1.amazonaws.com/$repo:$tag
</code></pre>
<p>If you first create the repository, you get an error like this: <code>name unknown: The repository with name 'kolla/ubuntu-source-base' does not exist in the registry with id '...'</code></p>


<h3>Table of Contents</h3>
<ul>
<li><a href="#pricing">Pricing</a></li>
<li><a href="#write-the-cloud-build-file">Write the Cloud Build file</a></li>
<li><a href="#iam-roles">IAM Roles</a>
<ul>
<li><a href="#docker-image-to-cloud-registry">Docker Image to Cloud Registry</a></li>
<li><a href="#static-website-to-google-cloud-storage">Static Website to Google Cloud Storage</a></li>
</ul>
</li>
<li><a href="#define-cloud-build-triggers">Define Cloud Build Triggers</a></li>
</ul>
<h1>Pricing</h1>
<p>Current pricing data can be found <a href="https://cloud.google.com/cloud-build/pricing">here</a>.</p>
<ul>
<li>For the first 120 builds in a day, cloud build is <strong>free</strong>.</li>
<li>each build after the free quota costs $0.003 US / build-minute when using the
single core build server with no additional SSD storage</li>
</ul>
<h1>Write the Cloud Build file</h1>
<p>Google Cloud Build uses a file, <code>cloudbuild.yml</code>, to define the tasks that will
be executed when a build is triggered.</p>
<p>The online references for the syntax are pretty good:</p>
<ul>
<li><a href="https://cloud.google.com/cloud-build/docs/build-config">Google's Cloud-Build Documentation</a></li>
<li><a href="https://cloud.google.com/cloud-build/docs/configuring-builds/substitute-variable-values">Variables Available in cloudbuild.yaml</a></li>
</ul>
<p>Below are some example configuration files used for this site.</p>
<h1>IAM Roles</h1>
<p><a href="https://cloud.google.com/cloud-build/docs/securing-builds/configure-access-control">Official Documentation</a></p>
<ul>
<li>Cloud Build Editor</li>
<li>Cloud Build Viewer</li>
</ul>
<h2>Docker Image to Cloud Registry</h2>
<p>Cloud Build is great for keeping your Docker Registry images aligned with your
version control code.</p>
<p>In this example I use Cloud Build to build and push a <a href="/docker-pelican-image">pelican image</a>
to the <a href="/google-container-registry">Google Container Registry</a>.</p>
<p>Be sure to keep the Dockerfile code in version control. Aside from being a good
idea in general, Cloud Build can also use Git commits as a build trigger.
<a href="https://github.com/kylep/pelican">Here's my Pelican Docker image repository</a>.</p>
<pre><code class="language-yaml">steps:
- name: 'gcr.io/cloud-builders/docker'
  args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/pelican', '.' ]
images:
- 'gcr.io/$PROJECT_ID/pelican'
</code></pre>
<h2>Static Website to Google Cloud Storage</h2>
<p>Another use for Cloud Builder is to generate static website files and then sync
them up with Cloud Storage every time the source control files or template are
changed.</p>
<p>Here's an example that triggers when I commit code to my
<a href="https://github.com/kylep/kyle.pericak.com">blog content git repository</a>. It
copies the content into <code>/workspace</code> and uses the above pelican image to render
the content and output it to the <code>output/</code> directory.</p>
<p>Once the site content is generated, the next step uses the <code>gcloud</code> image's
<code>gsutil -m rsync</code> command to upload the files to the Google Storage Bucket.</p>
<hr>
<h1>Define Cloud Build Triggers</h1>
<p>Here's the procedure I used to link a Cloud Build trigger to my Pelican image.</p>
<ol>
<li>Go to <a href="https://console.cloud.google.com/cloud-build">Google Cloud-Build</a>.</li>
<li>On the left side-bar, navigate to Triggers</li>
<li>Add Trigger</li>
<li>Select source: GitHub</li>
<li>Authenticate: Log in if needed</li>
<li>Select Repository: This is the repo that will be watched for changes</li>
<li>Name: This is the name of the trigger</li>
<li>Description: Something like <code>Push to master branch</code></li>
<li>Trigger Type: <code>Branch</code></li>
<li>Branch: <code>master</code></li>
<li>Build Configuration: <code>cloudbuild.yaml</code></li>
<li><strong>Create Trigger</strong></li>
</ol>
<p>With this trigger in place, all pushes to the pelican project will trigger the
rebuilding of its image, followed by updating the image in GCR with the latest
changes.</p>


<h3>Table of Contents</h3>
<ul>
<li><a href="#why-use-google-container-registry">Why use Google Container Registry</a></li>
<li><a href="#pricing">Pricing</a></li>
<li><a href="#command-line">Command Line</a></li>
<li><a href="#authentication">Authentication</a>
<ul>
<li><a href="#create-a-service-account">Create a service account</a></li>
<li><a href="#authenticate-your-local-docker-client">Authenticate your local Docker client</a></li>
</ul>
</li>
<li><a href="#using-the-google-container-registry">Using the Google Container Registry</a>
<ul>
<li><a href="#pull-image">Pull Image</a></li>
<li><a href="#push-image">Push Image</a>
<ul>
<li><a href="#access-denied">Access Denied</a></li>
</ul>
</li>
<li><a href="#delete-an-image">Delete an Image</a></li>
</ul>
</li>
</ul>
<h1>Why use Google Container Registry</h1>
<p>The Google Container Registry is a SaaS version of the public <a href="https://hub.docker.com/_/registry">Registry</a>
docker image. It solves a number of problems that the public Docker image has.
Namely:</p>
<ul>
<li><strong>Authentication</strong>: GCR has built-in access controls, where the Registry image
does not. With the Docker Registry you need to put something like NGINX in
front of it to provide access control.</li>
<li><strong>HTTPS</strong>: Like with the authentication problem, Registry needs a reverse proxy
in front of it to serve a certificate. Also, you need to worry about
obtaining and maintaining the valid cert files. Without a valid cert, each
Docker client needs to list the registry as an insecure registry, which is
both insecure and inconvenient.</li>
<li><strong>Serverless</strong>: To host a traditional Docker registry, you need a public
server running Docker. With Google's Container Registry, there's no server to
maintain.</li>
<li><strong>Vulnerability Scanning</strong>: GCR can <a href="https://cloud.google.com/container-registry/docs/get-image-vulnerabilities">scan your images for security issues</a>.</li>
</ul>
<hr>
<h1>Pricing</h1>
<p>The pricing is well documented <a href="https://cloud.google.com/container-registry/pricing">here</a>,
and it has a lot of variables.</p>
<p>Basically though, it boils down to the following approximate costs:</p>
<ul>
<li><strong>Storage</strong>: $0.026 USD / GB / month</li>
<li><strong>Network</strong>: ~ $0.12 USD / GB</li>
<li><strong>Vulnerability Scanning</strong>: $0.26 per scanned image</li>
</ul>
<hr>
<h1>Command Line</h1>
<p>GCR can be managed from the web UI or the command line. I try and keep my steps
limited to the web UI, but the command line is really easy to use. You can find
its documentation <a href="https://cloud.google.com/sdk/gcloud/reference/container/images">here</a>.</p>
<hr>
<h1>Authentication</h1>
<p>Before you start, be sure to install:</p>
<ul>
<li><a href="https://cloud.google.com/sdk/">Google Cloud SDK</a></li>
<li><a href="https://docs.docker.com/install/">Docker</a></li>
</ul>
<p>To interact with Google Container Registry you need to first authenticate your
local Docker client. Usually this is done with the <code>docker login &#x3C;registry></code>
command, but that's not how it's done with Google container registry. Instead
we'll use the <code>gcloud</code> command line.</p>
<h2>Create a service account</h2>
<p>While this isn't strictly necessary, it's good practice.</p>
<p>The container registry uses GCS buckets to store the images. It doesn't have
dedicated roles, and instead uses the GCS roles. You can find a detailed list
of the roles <a href="https://cloud.google.com/container-registry/docs/access-control">here</a>.
The two important roles are <code>roles/storage.admin</code> for write and
<code>roles/storage.objectViewer</code> for read.</p>
<p>To create the SA from the Google Cloud Web UI:</p>
<ol>
<li>Open the <a href="https://console.cloud.google.com/iam-admin/serviceaccounts">IAM Service Accounts page</a></li>
<li>If needed, select or create your project</li>
<li>Click <code>Create Service Account</code></li>
<li>Create a member, such as <code>gcr-readonly</code> or <code>gcr-admin</code>.
When possible, its nice to have the account name and account ID match.</li>
<li>Assign a useful description to the service-account.</li>
<li>Assign a role. <code>Storage Object Viewer</code> grants read access and
<code>Storage Admin</code>.</li>
<li>Create and download a key. Name it something like <code>gcr-readonly.json</code>.</li>
</ol>
<h2>Authenticate your local Docker client</h2>
<p>Use the <code>gcloud</code> tool to deploy some <code>credHelpers</code> to <code>~/.docker/config.json</code>.</p>
<pre><code class="language-bash">gcloud auth configure-docker --project &#x3C;project name>
</code></pre>
<p>Next up, it's time to authenticate.</p>
<p>You could run <code>gcloud auth login</code>, but that doesn't use the service account we
just created and also requires a browser to open a generated link.</p>
<p>Instead, use the service account's JSON file generated in the above step with
the <code>--key-file</code> argument as follows:</p>
<pre><code class="language-bash">gcloud auth activate-service-account --key-file /vagrant/auth/gcr-readonly.json
</code></pre>
<p>If successful, you'll get a confirmation message:</p>
<pre><code class="language-bash">Activated service account credentials for: [gcr-readonly@*.gserviceaccount.com]
</code></pre>
<hr>
<h1>Using the Google Container Registry</h1>
<p>The GCR works just like any other registry once you've ran the <code>gcloud auth</code>
command. You don't need to run any <code>docker login</code> command.</p>
<h2>Pull Image</h2>
<p>Here's how to pull a Pelican image from GCR:</p>
<pre><code class="language-bash">docker pull gcr.io/myProject/pelican
</code></pre>
<h2>Push Image</h2>
<p>It's pretty straightforward. Replace <code>myProject</code> with your project name.</p>
<p>In this example I add a tag to my pelican image, calling it <code>pelican2</code>, and
push it up to the registry as a new image. You'd normally make your own Docker
image.</p>
<pre><code class="language-bash">sudo docker tag gcr.io/myProject/pelican gcr.io/myProject/pelican2
sudo docker push gcr.io/myProject/pelican2
</code></pre>
<p>If you didn't get an error, then the image is now pushed.</p>
<h3>Access Denied</h3>
<p>If you receive the following message, your user account lacks write access:</p>
<pre><code class="language-text">denied: Token exchange failed for project 'myProject'. Caller does not have permission 'storage.buckets.get'.
</code></pre>
<p>To resolve this, create a service account  with the role <code>roles/storage.admin</code>.
If there's a finer grained access level to permit this, I'd be interested to
hear about it.</p>
<pre><code class="language-bash">gcloud auth activate-service-account --key-file /vagrant/auth/gcr-admin.json
sudo docker push gcr.io/kylepericak/pelican2
</code></pre>
<h2>Delete an Image</h2>
<p>At the time of writing this, there's a warning in the cloud UI saying that you
can't delete images unless you use the CLI.</p>
<p>Here's how to do that. Be sure to replace <code>myProject</code> with your project.</p>
<pre><code class="language-bash"># List current images
gcloud config set project myProject

# Ignore this warning if you're using a service account:
# WARNING: You do not appear to have access to project [myProject] or it does not exist.

# Delete the image
cloud container images delete gcr.io/myProject/pelican2
</code></pre>


<p><strong>This post is linked to from the <a href="/blog-website.html">Blog Website Project</a></strong></p>
<hr>
<p>Nobody wants to test in production.</p>
<p>This guide covers how to preview your Pelican content locally as you write it.
Every time you save a markdown file or update the Pelican theme, the entire
site will re-generate and be hosted by a local web-server for you to review.</p>
<h1>Install Dependencies</h1>
<ul>
<li><a href="https://docs.docker.com/install/">Docker</a></li>
<li><a href="https://git-scm.com/book/en/v2/Getting-Started-Installing-Git">Git</a></li>
<li>Text Editor: Your choice, I use <a href="https://github.com/vim/vim">Vim</a></li>
<li>Browser: Chrome has nice developer tools but anything works</li>
</ul>
<hr>
<h1>Get the Pelican Image</h1>
<p><strong>Option 1: Build your own</strong></p>
<p>This is the approach I suggest. You can see the guide I wrote explaining how to
build a Pelican image <a href="/docker-pelican-image.html">here</a>.</p>
<p><strong>Option 2: Use a pre-made image</strong>
I've shared my image for public use <a href="https://hub.docker.com/r/kpericak/pelican">here</a>.
If you plan on using it, you're best bet is to check how I use it on GitHub.</p>
<hr>
<h1>Launch a Persistent Pelican Container</h1>
<p>Once you've either pulled or built the image, it will be listed in your Docker
image list.</p>
<pre><code class="language-bash">docker image list
</code></pre>
<p>Set the following variables. Note the paths should be absolute (vs relative).</p>
<ul>
<li><code>$image_name</code>: docker image name</li>
<li><code>$conf_file</code>: path to <code>pelicanconf.py</code> configuration file</li>
<li><code>$content_dir</code>: path to <code>content/</code> directory, where markdown files reside</li>
<li><code>$output_dir</code>: path to <code>output/</code> directory, where generated files will land.</li>
<li><code>$listen_port</code>: Which port to listen on for the dev web-server</li>
</ul>
<p>Optionally, you can set a theme path too. If you omit this and its associated
volume mount then the theme built into the Pelican image will be used instead.
This enables theme development.</p>
<ul>
<li><code>$theme_path</code>: Path to the Pelican theme</li>
</ul>
<p>Start the Docker container in daemon mode (<code>-d</code>)  as persistent container.
By setting the run command to <code>tail -f /dev/null</code>, the container will never
stop itself and instead sit idle until told to do otherwise.</p>
<p>Mount the directories specified by those variables into the container.</p>
<p>Forward <code>$listen_port</code> to the workstation so the site can be opened in a local
browser.</p>
<pre><code class="language-bash">image_name=""
conf_file=""
content_dir=""
output_dir=""
theme_path=""
listen_port=8000

docker run -d \
  --name pelican \
  -v $conf_file:/pelicanconf.py \
  -v $conftent_dir:/content \
  -v $output_dir:/output \
  -v $theme_path:/theme" \
  -p 0.0.0.0:$listen_port:8000 \
  image_name \
  tail -f /dev/null
</code></pre>
<hr>
<h1>Launch Development Web-Server</h1>
<p>At this point you have a pelican container running, but it isn't doing
anything. The files are all mounted in, and it's prepared to forward its local
port 8000 to your specified workstation port, so all that's left is to start
Pelican's built-in development service.</p>
<p>This service will watch both the markdown files and the theme files for
changes, then reload the site files. It will also host the website on a
non-production web-server.</p>
<pre><code class="language-bash">docker exec -it pelican \
  pelican \
    -s /pelicanconf.py \
    --debug \
    --autoreload \
    --listen \
    --port 8000 \
    --bind 0.0.0.0 \
    /content \
    -o /output
</code></pre>


<p><strong>This post is linked to from the <a href="/blog-website.html">Blog Website Project</a></strong></p>
<hr>
<p>Pelican is a free and open source static website generator written in Python.
It accepts a theme, a config file, and a directory of markdown documents as
input and uses them to generate the HTML and CSS files that make up a static
website.</p>
<p>This post covers how to build a Docker image that contains the Pelican software
and the theme which I chose to use with it. This image is reusable and
<a href="https://hub.docker.com/r/kpericak/pelican">freely available on Docker Hub</a>.</p>
<h1>Dockerfile</h1>
<p>As with any Docker image, the most important piece is the Dockerfile.</p>
<p>You can see the current file I'm using <a href="https://github.com/kylep/pelican/blob/master/Dockerfile">here</a>.</p>
<p>In this Dockerfile, Pelican, its theme, and plugins are installed to the image.</p>
<pre><code class="language-text">FROM ubuntu:bionic
ADD requirements.txt ./
RUN apt-get update \
    &#x26;&#x26; apt-get install -y --no-install-recommends \
         git \
         python \
         python-pip \
         python-setuptools \
    &#x26;&#x26; rm -rf /var/lib/apt/lists/* \
    &#x26;&#x26; pip install -r requirements.txt \
    &#x26;&#x26; git clone https://github.com/kylep/voidy-bootstrap.git /theme \
    &#x26;&#x26; mkdir -p /tmp-plugins/ /plugins \
    &#x26;&#x26; git clone --recurse \
           https://github.com/getpelican/pelican-plugins.git /tmp-plugins \
    &#x26;&#x26; cp -r /tmp-plugins/series /plugins/ \
    &#x26;&#x26; rm -r /tmp-plugins
</code></pre>
<h3>FROM ubuntu:bionic</h3>
<p>This section specifies the Docker base image. The above docker file uses
<a href="https://hub.docker.com/_/ubuntu/">the official Docker Hub Ubuntu image</a>.</p>
<h3>ADD requirements.txt</h3>
<p>The requirements.txt file lists the python packages which Pelican depends upon.
Docker's <code>ADD</code> will copy that file into the image at the specified path.</p>
<p>The minimum packages required are:</p>
<pre><code class="language-text">pelican
markdown
</code></pre>
<h3>RUN ...</h3>
<p>The run task is basically a bash script all squished into one line. While its
harder to read with all the line continuation and <code>&#x26;&#x26;</code> symbols, using them
leads to Docker only building a single layer.</p>
<p><a href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/">See the best practices for writing docker images</a>.</p>
<p>Since this is an Ubuntu image, <code>apt</code> is the package manager. It installs Git
and some python packages. Git is required to pull the Pelican theme from
Github.</p>
<p>Only the minimum Python packages are installed by apt, since pip is the
preferred python package manager.</p>
<p><code>rm -rf /var/lib/apt/lists/*</code> removes the leftover apt artifacts, helping to
shrink the Docker image.</p>
<p>After the theme is pulled from GitHub, Pelican's open source plugins are also
pulled. The ones desired are copied to <code>/plugins</code>, then the (huge) checkout is
deleted to reduce image size.</p>
<hr>
<h1>Running the Docker Image</h1>
<p>Locally, you need to mount the content into the container. Something like this
will work:</p>
<pre><code class="language-bash">docker run \
  --name pelican \
  -v $(pwd):/workspace \
  gcr.io/kylepericak/pelican:latest \
  pelican -o /workspace/output -s /workspace/pelicanconf.py
</code></pre>
<p>The above command will load the <code>content/</code> and <code>pelicanconf.py</code> files into
Pelican, and write the static website to <code>output/</code>.</p>


Kyle Pericak

"It works in my environment"

Categories

Tags