<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#prepare-the-image-in-guest">Prepare the image in-guest</a>
<ul>
<li><a href="#remove-vmware-tools">Remove VMWare Tools</a></li>
<li><a href="#install-cloudbase-init--sysprep-if-needed">Install cloudbase-init &#x26; sysprep (if needed)</a></li>
<li><a href="#shut-down-the-vm--export-the-drive">Shut down the VM &#x26; Export the drive</a></li>
</ul>
</li>
<li><a href="#inject-the-kvm-drivers">Inject the KVM Drivers</a></li>
</ul>
<h1 id="prepare-the-image-in-guest">Prepare the image in-guest</h1>
<p>From VMWare or HyperV, launch your template Windows VM and log in as an
administrative user. Make any firewall changes you need, install any packages,
and ensure that RDP is enabled.</p>
<h2 id="remove-vmware-tools">Remove VMWare Tools</h2>
<p>If the guest has VMWare tools installed, remove them. They won't be needed
where we're going.</p>
<h2 id="install-cloudbase-init--sysprep-if-needed">Install cloudbase-init &#x26; sysprep (if needed)</h2>
<p>This isn't so much for KVM as it is for OpenStack, but cloudbase-init is the
Windows version of cloud-init. This will set your hostname and such when the VM
is created.</p>
<ol>
<li><a href="https://cloudbase.it/cloudbase-init/">Download the Cloudbase-init installer</a></li>
<li>Run through the installer.
<ol>
<li>Don't run as Local System.</li>
<li>Don't name your user Administrator, the named user is made by this
service to run the startup tasks.</li>
</ol>
</li>
<li>At the end of the installation, you'll be prompted to sysprep. If this VM
will be re-used as a template and not just a one-off VM, then sysprep it.</li>
</ol>
<h2 id="shut-down-the-vm--export-the-drive">Shut down the VM &#x26; Export the drive</h2>
<p>Shut down the VM and grab the VMDK or VHDX file it booted from. Copy that file
to a directory on your computer, such as <code>C:\Temp</code>.</p>
<hr>
<h1 id="inject-the-kvm-drivers">Inject the KVM Drivers</h1>
<ol>
<li><a href="https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/virtio-win.iso">Download the VirtIO ISO</a></li>
<li>Mount the ISO to your workstation. In these examples, it's E:</li>
<li>Open PowerShell as admin and verify that the file is where you put it. You
can show some basic info about it like this: <code>get-vhd C:\Temp\Win10.vhdx</code></li>
<li>Create a mount-point for Dism: <code>mkdir C:\mount</code></li>
<li>Mount the volume with Dism:
<code>Dism /mount-image /ImageFile:C:\Temp\Win10.vhdx /Index:1 /MountDir:C:\mount</code></li>
<li>Inject the <code>viostor</code> driver:
<code>Dism /image:C:\mount /Add-Driver /Driver:E:\viostor\w10\amd64 /Recurse</code></li>
<li>Inject the <code>Baloon</code> driver:
<code>Dism /image:C:\mount /Add-Driver /Driver:E:\Balloon\w10\amd64 /Recurse</code></li>
<li>Inject the <code>NetKVM</code> driver:
<code>Dism /image:C:\mount /Add-Driver /Driver:E:\NetKVM\w10\amd64 /Recurse</code></li>
<li>Confirm the drivers injected: <code>Dism /image:C:\mount /Get-Drivers</code></li>
<li>Unmount the image: <code>Dism /Unmount-image /MountDir:c:\mount /commit</code></li>
</ol>
<p>Now the image is ready to be converted to raw/qcow2 and uploaded to OpenStack.</p>
<p><strong>Note</strong>: CloudBase has released a Windows version of <code>qemu-img</code> so you can do
the conversions on Windows now without VirtualBox tools. You can download it
<a href="https://cloudbase.it/qemu-img-windows/">here</a></p>


<p>When you use a tool like <code>dd</code> or <a href="https://www.balena.io/etcher/">Etcher</a> to
turn a USB
stick into boot media, usually for installing an operating system, any attempt
at formatting that disk later will be met with a drive that only has a few GB
total. This is particularly annoying when you use something like a 100+GB USB
stick for that.</p>
<p>The cause of this problem is that a smaller partition is created on the drive
than the drive could potentially hold. To fix the drive you need to
removing all the partitions.</p>
<p>List your devices. Note your USB stick (example: <code>/dev/disk2</code>)</p>
<pre><code class="language-bash">diskutil list
</code></pre>
<p>Try to fix it using diskutil:</p>
<pre><code class="language-bash">diskutil eraseDisk free EMPTY /dev/disk2
</code></pre>
<p>Often that's good enough, and the drive now shows up in Disk Utility with its full capacity.</p>
<hr>
<p>If that fails, you can overwrite the partition/filesystem's metadata at the start of the disk with zeros.</p>
<p>If the drive is mounted, unmount it.</p>
<pre><code class="language-bash">diskutil umountDisk /dev/disk2
</code></pre>
<p>Delete the partition. Note that I tried <code>gpt</code> first but it threw an error
saying <code>gpt show: error: bogus map</code>. Also, as far as I can tell, <code>diskutil</code>
won't actually erase a partition and instead will only reformat it. Strangely,
even <code>fdisk</code> seems to have hidden the option to delete a partition. Guess Apple
<strong>REALLY</strong> doesn't trust its users. That's too bad, since I ended up doing
something even more dangerous.</p>
<p>Overwrite the partition and filesystem data in the first 1GB of the USB drive
with zeros. Be <strong>really really sure</strong> that you select the right disk here.
You can 100% toast your workstation's boot disk if you accidentally specify it.</p>
<pre><code class="language-bash">sudo dd if=/dev/zero of=/dev/disk2 count=1 bs=1GB
</code></pre>
<p>If that doesn't work, set <code>bs=1MB</code> and remove the <code>count=1</code>. You might need to
delete the end of it instead. That'll take a while.</p>


<p>This post covers how I use Ansible to install Node.js. I googled around for a
solution to this and thought the solutions I found looked way too complicated.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#the-problem">The Problem</a>
<ul>
<li><a href="#what-about-nvm">What about NVM?</a></li>
<li><a href="#chris-leanodejs">chris-lea/node.js</a></li>
<li><a href="#curl-x--bash">curl <em>x</em> | bash</a></li>
</ul>
</li>
<li><a href="#the-solution">The Solution</a></li>
<li><a href="#ansible-example">Ansible Example</a>
<ul>
<li><a href="#add-nodesource-repository-to-apts-sourceslist">Add nodesource repository to Apt's sources.list</a></li>
<li><a href="#install-nodejs">Install Node.js</a></li>
<li><a href="#use-npm-normally">Use NPM normally</a></li>
</ul>
</li>
</ul>
<h1 id="the-problem">The Problem</h1>
<p>In theory, installing Node.js should be as easy as running
<code>apt-get install nodejs</code>. The problem is that Node.js versions seem to move
pretty quickly and the one distributed by Cannonical in Ubuntu Bionic is
already too old to run some applications. In my case I was trying to install
<code>firebase-tools</code>, the CLI for Google Firebase, and it wouldn't run with Apt's
version of the package.</p>
<h2 id="what-about-nvm">What about NVM?</h2>
<p>NVM, or Node Version Manager, is a nice little application for installing
various versions of Node.js. It kind of reminds me of python virtualenv.
The problem with NVM is that it relies on setting environment variables and
defining the <code>nvm</code> function using the <code>.bashrc</code> file. Ansible doesn't read
that file when it runs.</p>
<p>In theory, the Ansible <code>npm</code> module supports accepting an executable path,
specifically to support NVM, but I had to do some ugly shell commands to
collect that path and even afterwards it wouldn't actually work. Specifically
running the <code>npm</code> binary deployed by NVM without scoping the <code>.bashrc</code> file
throws stack traces instead of installing your app.</p>
<h2 id="chris-leanodejs">chris-lea/node.js</h2>
<p>A lot of the sites I found were adding the Apt PPA <code>'ppa:chris-lea/node.js'</code>.
I don't know what that is, but it doesn't sit well with me. Just seems like
something that wont age well, might be insecure, and is probably fragile.</p>
<p>To make matters worse, when I tried it out it didn't seem to support Bionic.</p>
<h2 id="curl-x--bash">curl <em>x</em> | bash</h2>
<p>Some examples I saw mentioned just curling various URLs and piping them to bash
to run a script.
Sometimes I'm OK with doing this, but that sort of installation method doesn't
really seem appropriate for an Ansible playbook. If nothing else it'd be hard
to make those tasks idempotent. Plus, we can do better.</p>
<hr>
<h1 id="the-solution">The Solution</h1>
<p>If you look at the <a href="https://github.com/nodesource/distributions">nodesource/distributions GitHub page</a>,
they have some really nice documentation in their <code>README.md</code>. Under the
<strong>Manual Installation</strong> section you can see that they have a nice, stable
looking Apt repository URL. This worked great for me, and is trivial to
set up Ansible to work with.</p>
<hr>
<h1 id="ansible-example">Ansible Example</h1>
<h2 id="add-nodesource-repository-to-apts-sourceslist">Add nodesource repository to Apt's sources.list</h2>
<pre><code class="language-yaml">- name: "Add nodejs apt key"
  apt_key:
    url: https://deb.nodesource.com/gpgkey/nodesource.gpg.key
    state: present

- name: "Add nodejs 13.x ppa for apt repo"
  apt_repository:
    repo: deb https://deb.nodesource.com/node_13.x bionic main
    update_cache: yes
</code></pre>
<h2 id="install-nodejs">Install Node.js</h2>
<pre><code class="language-yaml">- name: "Install nodejs"
  apt:
    update_cache: yes
    name: nodejs
    state: present
</code></pre>
<h2 id="use-npm-normally">Use NPM normally</h2>
<p>With this version of Node.js, I was able to install and run the application.</p>
<pre><code class="language-yaml">- name: "Install NPM-distributed command-line tools"
  npm:
    global: yes
    name: "{{ item }}"
  with_items:
    - firebase-tools
</code></pre>


<p><strong>This post is linked to from the <a href="/blog-website.html">Blog Website Project</a></strong></p>
<hr>
<p>This static website is hosted using Google Cloud Storage, which is great except
that it doesn't support HTTPS. There's no really good excuse to not support
HTTPS on a website today.</p>
<p>To close that gap, CloudFlare can handle the HTTPS negotiation for you and it
takes no time at all to set up.</p>
<h1 id="configure-https-in-cloudflare">Configure HTTPS in CloudFlare</h1>
<p>This guide assumes CloudFlare is already acting as a CDN for your site.</p>
<ol>
<li>Go to the CloudFlare DNS page and find the A record or CNAME for your FQDN</li>
<li>Ensure that the cloud icon is yellow and not grey, indicating that the CDN
is in use and not bypassing CloudFlare.</li>
<li>Click on the padlock labelled SSL/TLS</li>
<li>Go to the Overview tab</li>
<li>Select Flexible.... Basically that's it, you're good.
HTTPS is now running on your site. CloudFlare is a really great product!
There's more that can be done, though.</li>
<li>Go to the Edge Certificates tab</li>
<li>Set "Always Use HTTPS" to On</li>
</ol>
<p>In my testing the "Always Use HTTPS" redirect change didn't take effect
immediately. After about 15 minutes 301's were coming back as expected.</p>
<pre><code class="language-text">vagrant@dev-env:~/$ curl -i http://kyle.pericak.com
HTTP/1.1 301 Moved Permanently
Date: Tue, 03 Dec 2019 00:52:25 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Tue, 03 Dec 2019 01:52:25 GMT
Location: https://kyle.pericak.com/
Server: cloudflare
CF-RAY: 53f18ecdec10b677-YWG
</code></pre>
<p><a href="https://support.cloudflare.com/hc/en-us/articles/204144518">You can find more details and official documentation here.</a></p>


<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<p>In this project I'll be trying out each of the Google Cloud Platform's
services. I'll be updating it pretty regularly.</p>
<hr>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes, or will include, the following posts.
If any aren't finished, check back later! I'll also certainly be adding more
as time goes on.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left"></td>
<td align="left"><strong>Continuous Integration</strong></td>
</tr>
<tr>
<td align="left">2019-08-06</td>
<td align="left"><a href="/google-cloud-build.html">Google Cloud Build: Basics</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Database</strong></td>
</tr>
<tr>
<td align="left">2019-10-14</td>
<td align="left"><a href="/gcp-firestore-python.html">Google Cloud Firestore Basics - Python</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Containers</strong></td>
</tr>
<tr>
<td align="left">2019-08-05</td>
<td align="left"><a href="/google-container-registry.html">Google Container Registry: Basics</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Cloud Storage</strong></td>
</tr>
<tr>
<td align="left">2019-08-09</td>
<td align="left"><a href="/google-cloud-storage-website.html">Hosting websites on Cloud Storage</a></td>
</tr>
</tbody>
</table>


<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<p>In this project I'll be trying many of AWS's services.</p>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes the following posts.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left"></td>
<td align="left"><strong>Running Servers on AWS</strong></td>
</tr>
<tr>
<td align="left">2020-01-09</td>
<td align="left"><a href="/aws-ec2-python.html">Create &#x26; Terminate EC2 Instances from Python</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Running Containers on AWS</strong></td>
</tr>
<tr>
<td align="left">2019-08-26</td>
<td align="left"><a href="/aws-ecr.html">Save a Docker image to Elastic Container Registry (ECR)</a></td>
</tr>
<tr>
<td align="left">2019-09-10</td>
<td align="left"><a href="/aws-fargate.html">Run a Docker container Using AWS Fargate</a></td>
</tr>
<tr>
<td align="left">2020-01-15</td>
<td align="left"><a href="/aws-codepipeline-ecr.html">CodePipeline: Continuous Delivery to AWS ECR</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Cloud Storage</strong></td>
</tr>
<tr>
<td align="left">2019-09-26</td>
<td align="left"><a href="/aws-s3-python.html">Using AWS S3 from Python</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Serverless Applications</strong></td>
</tr>
<tr>
<td align="left">2020-01-08</td>
<td align="left"><a href="/aws-lambda.html">Lambda basics</a></td>
</tr>
<tr>
<td align="left">2020-01-09</td>
<td align="left"><a href="/aws-lambda-iam.html">IAM for Lambda</a></td>
</tr>
<tr>
<td align="left">2020-01-21</td>
<td align="left"><a href="/aws-api-gateway-iam.html">IAM Authentication using API Gateway</a></td>
</tr>
</tbody>
</table>


<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<h4 id="table-of-contents">Table of contents</h4>
<ul>
<li><a href="#what-is-openstack">What is OpenStack?</a></li>
<li><a href="#project-posts--progress">Project Posts &#x26; Progress</a></li>
<li><a href="#why-use-an-openstack-private-cloud">Why Use an OpenStack Private Cloud?</a></li>
<li><a href="#complexity-and-openstack-ad">Complexity and OpenStack (Ad)</a></li>
<li><a href="#installing-openstack">Installing OpenStack</a>
<ul>
<li><a href="#non-production-installation-options">Non-Production Installation Options</a>
<ul>
<li><a href="#devstack">DevStack</a></li>
<li><a href="#small-physical-server--kolla-ansible">Small Physical Server + Kolla-Ansible</a></li>
<li><a href="#virtual-server--kolla-ansible">Virtual Server + Kolla-Ansible</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#containerized-openstack">Containerized OpenStack</a></li>
</ul>
<h1 id="what-is-openstack">What is OpenStack?</h1>
<p><a href="https://www.openstack.org/">OpenStack</a> is a collection of open-source projects
that work together to form a private cloud solution. It's by far the
biggest open source private cloud option available. The <a href="https://www.openstack.org/foundation/">OpenStack Foundation</a>
has over 105,000 community members from 187 countries around the world.</p>
<p>At a high level, you can think of it as an abstraction layer that sits on top
of other popular cloud technologies such as storage clusters, hypervisors, and
network solutions. For example, KVM (the popular Linux hypervisor) is not part
of OpenStack, but it's used by <strong>Nova</strong>, a component of OpenStack which can
orchestrate against KVM, VMWare, HyperV, and Xen.</p>
<hr>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes the following posts.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">2019-08-13</td>
<td align="left"><a href="/openstack-aio-ka-metal.html">Install OpenStack on Metal - Intel NUC</a></td>
</tr>
<tr>
<td align="left">2019-08-11</td>
<td align="left"><a href="/openstack-aio-ka-vm.html">Install OpenStack inside a VM</a></td>
</tr>
<tr>
<td align="left">2019-08-26</td>
<td align="left"><a href="/openstack-kolla-custom-plugin.html">Modifying OpenStack Kolla Docker Images</a></td>
</tr>
<tr>
<td align="left">2019-12-03</td>
<td align="left"><a href="/openstack-ansible.html">Operating OpenStack from Ansible</a></td>
</tr>
<tr>
<td align="left">2020-01-25</td>
<td align="left"><a href="/openstack-cloudinit-powershell.html">OpenStack Cloud-Init Powershell Example</a></td>
</tr>
<tr>
<td align="left">2019-12-17</td>
<td align="left"><a href="windows-kvm-drivers">Injecting KVM Drivers into Windows</a></td>
</tr>
</tbody>
</table>
<hr>
<h1 id="why-use-an-openstack-private-cloud">Why Use an OpenStack Private Cloud?</h1>
<ol>
<li><strong>Price at scale:</strong> At small scales, private clouds lose on price
against the public clouds. As you start to ramp up though, the public clouds
start to really nickle and dime you. Private clouds have higher up-front
costs but they win out in the long-run.</li>
<li><strong>Predictable Cost:</strong> Spin up a few services on AWS or Azure. How much will
they cost you each month? As it stands, you're guess is going to be wrong.
After the free trial periods and special offers stop muddying the water,
you'll usually discover you're spending more than you thought.</li>
<li><strong>Hardware Re-Use:</strong> Many IT organizations already have hardware sitting
around running expensive hypervisor software. OpenStack is free and can use
that existing hardware, giving you more for less.</li>
<li><strong>Data Sovereignty:</strong> For legal and security reasons, many organizations
must know exactly where their data is sitting at rest. With private clouds,
there's no ambiguity.</li>
<li><strong>Behind Your Firewall:</strong> So you spent a ton of money on some next
generation firewall technologies with intrusion detection/prevention, and
trained the staff up to use it. You run an arguably well secured
data-center. Running OpenStack keeps the cloud behind your existing security
measures.</li>
<li><strong>Latency:</strong> Particularly for edge clouds and VDI solutions, having the
cloud located right on-premises can hugely improve user experience.</li>
<li><strong>Shadow IT:</strong> Developers are going to develop, even if your organization
can't get them the tools they need in time. By offering your own in-house
cloud, you can help prevent
<a href="https://en.wikipedia.org/wiki/Shadow_IT">shadow IT</a> from emerging.</li>
<li><strong>Control:</strong> This one is a double-edged sword. Having more control over your
cloud can be its own benefit, but there's also value in letting the pros
handle it.</li>
<li><strong>Business as usual, without VMWare:</strong> Don't get me wrong, VMWare make a
great product... I've never heard anyone say they were happy with their
VMWare bill. It's insanely expensive when there's an open source alternative
that can check all the same boxes.</li>
</ol>
<hr>
<h1 id="complexity-and-openstack-ad">Complexity and OpenStack (Ad)</h1>
<p>OpenStack is significantly more difficult to get started with than the public
clouds. You can't really install OpenStack without first understanding what
each of the main components does, and even then each one has its own quirks for
their installation. If you want to install in production, you also need to
figure out how to get it all highly available.</p>
<p>Unless you've got a huge team that's eager to tackle this complexity, I'm going
to use this opportunity to plug the software &#x26; solution I've helped build for
<a href="https://breqwatr.com">Breqwatr</a>. They'll stand up your cloud in a couple days
flat, teach your team how to use it, and keep the lights on - all while
charging significantly less than VMWare would cost for the same hardware.</p>
<p>Breqwatr, and companies like it, can be your trusted cloud partner and abstract
away that complexity by handling the installation, upgrades, monitoring, and
day-to-day operation of the cloud.</p>
<hr>
<h1 id="installing-openstack">Installing OpenStack</h1>
<h2 id="non-production-installation-options">Non-Production Installation Options</h2>
<p>If you're not looking for a production deployment and instead want to try
OpenStack out to learn it, there are a few good options.</p>
<h3 id="devstack">DevStack</h3>
<p>The standard developer option is probably <a href="https://docs.openstack.org/devstack/latest/">DevStack</a>.</p>
<p>In truth I haven't used this since I've always needed my OpenStack installs to
mirror a production-ready environment. If I try it out, I'll replace the above
link with one to a guide I write myself.</p>
<h3 id="small-physical-server--kolla-ansible">Small Physical Server + Kolla-Ansible</h3>
<p>Another option is to install OpenStack on a physical server. I've documented
my experience doing this using images from the <a href="https://github.com/openstack/kolla">Kolla project</a>,
and Ansible automation from the <a href="https://github.com/openstack/kolla-ansible">Kolla-Ansible project</a>.</p>
<p>The nice thing about this approach is you can use a server at home, and once
you're done it can be a VM host as your private house cloud.</p>
<p><strong><a href="/openstack-aio-ka-metal.html">Installing OpenStack on a physical server</a></strong></p>
<h3 id="virtual-server--kolla-ansible">Virtual Server + Kolla-Ansible</h3>
<p>This option is great for development use if you already have a cloud available
or want to use a public cloud for testing OpenStack. The VMs you launch on it
will perform terribly though, and there are some networking complications.</p>
<p>Don't bother trying to launch Windows VMs at all in a nested virtualisation
scenario, you'll be lucky if they even boot.</p>
<p><strong><a href="/openstack-aio-ka-vm.html">Installing OpenStack on a Virtual Server</a></strong></p>
<hr>
<h1 id="containerized-openstack">Containerized OpenStack</h1>
<p>Managing OpenStack clusters is a pain. It has so many services, each with their
own dependencies and operating instructions, there's a lot that can go wrong.
One thing that's really helped with that is to containerize each service and
deploy them all using Docker.</p>
<p>The <a href="https://github.com/openstack/kolla">Kolla project</a> does such a great job
of that, that there's no need to do it yourself. That is, until you need to
change an image, for instance to install a Cinder plugin.</p>
<p><strong><a href="/openstack-kolla-custom-plugin.html">Modifying OpenStack Kolla Docker Images</a></strong></p>


<p>As of Vim7, spell-checking is a built-in feature.</p>
<p>The spell checker will highlight incorrectly spelled words in red.
Light blue highlights indicate words which are only found in another dictionary
(such as "color" from <code>en_us</code>).</p>
<hr>
<h1 id="commands">Commands</h1>
<h2 id="enabledisable">Enable/Disable</h2>
<p>Enable spellcheck</p>
<ul>
<li><code>set</code> for all buffers, <code>setlocal</code> for the current one</li>
<li><code>en_ca</code> will use Canadian spelling.</li>
</ul>
<pre><code class="language-text">:set spell spelllang=en_ca
:setlocal spell spelllang=en_ca
</code></pre>
<p>Or if you're not picky, the <code>spelllang</code> argument can be omitted. I'm not sure
what the default is, but I figure it's probable <code>en</code> which uses all the English
dictionaries together.</p>
<pre><code>:set spell
</code></pre>
<p>Disable spellcheck:</p>
<pre><code class="language-text">:set nospell
</code></pre>
<h2 id="using-spell-check">Using Spell-Check</h2>
<p>Move to the next misspelled word</p>
<pre><code>]s
</code></pre>
<p>previous misspelled word</p>
<pre><code>[s
</code></pre>
<p>Highlight a misspelled word. Show possible replacements.
Press return with no input to go back.</p>
<pre><code>z=
</code></pre>
<p>Accept the spelling of a word:</p>
<pre><code class="language-text">zg
</code></pre>
<p>Mark a word's spelling as wrong</p>
<pre><code class="language-text">zw
</code></pre>


<p><strong>This post is linked to from the <a href="/gcp.html">GCP: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to interact with Google's Cloud Firestore, using examples
written in Python. The concepts should apply to any language.</p>
<p>Firestore is the newer version of Datastore. You can find the differences
<a href="https://cloud.google.com/firestore/docs/firestore-or-datastore">here</a>.
In this post, the  database is configured in Native mode.
Other than some extra language support, Native mode's coolest new feature seems
to be the ability to listen to a post. From what I can tell, this could replace
the need for pub/sub in some scenarios.</p>
<p>Throughout the post I'll use a project named "example". Be sure to replace this
with your own project name.</p>
<hr>
<h1 id="pricing">Pricing</h1>
<p>Official Firestore pricing data is <a href="https://cloud.google.com/firestore/pricing">here</a>.</p>
<p><strong>Note that Firestore mode is almost twice the price of Datastore mode.</strong>
Datastore pricing can be found <a href="https://cloud.google.com/datastore/pricing">here</a>.</p>
<p>Every day a number of free operations are allowed, which for a small
application makes Datastore very affordable. Even after the free tier though,
the pricing is pretty good. Firestore and Datastore have the same free tiers.</p>
<h2 id="summary-of-pricing-datastore-vs-firestore">Summary of pricing: Datastore vs Firestore</h2>
<ul>
<li>Storage: $/GB/Month
<ul>
<li>Datastore: 0.108</li>
<li>Firestore: 0.180</li>
</ul>
</li>
<li>Reads: $/100,000
<ul>
<li>Datastore: 0.036</li>
<li>Firestore: 0.060</li>
</ul>
</li>
<li>Writes: $/100,000
<ul>
<li>Datastore: $0.108</li>
<li>Firestore: $0.180</li>
</ul>
</li>
<li>Deletes: $/100,000
<ul>
<li>Datastore: 0.012</li>
<li>Firestore: 0.020</li>
</ul>
</li>
</ul>
<p>Max entity size and document size are both 1MB, so you could realistically just
use JSON strings in entities instead to save money if you don't need any of
Firestore's other features.</p>
<hr>
<h1 id="datastorefirestore-concepts">Datastore/Firestore Concepts</h1>
<p>Both options use a non-relational/NoSQL database. NoSQL databases in general
allow for the direct serialization of a programs objects/dictionaries.</p>
<h2 id="firestore">Firestore</h2>
<p><a href="https://cloud.google.com/firestore/docs/data-model">Official Firestore data model docs</a></p>
<p>Firestore uses a document database organized into documents and collections.
This is a change from the older Datastore model of Entities organized by Kinds.</p>
<p>Documents support nested data structures ("Maps"). Think of them like JSON
objects, but with more types supported. Documents are limited to 1MB each.</p>
<p>Documents are stored in Collections.
Collections are containers for documents, like a filesystem's directories.
Documents can also contain their own nested collections, which store other
documents. Nesting has a depth limit of 100.</p>
<p>Documents are identified by References. In python, a reference looks like
this:</p>
<pre><code class="language-python">collection('collection1').document('document1')
</code></pre>
<h2 id="datastore">Datastore</h2>
<p>In Datastore, objects are called Entities. The key-value data pairs stored by
entities are called properties.</p>
<p>Entities are uniquely identified by keys.
Entity Keys consists of the following components:</p>
<ul>
<li><em>Namespace</em>: Basically a tenant identifier, like a user ID. By default, the
namespace is an empty string. Any entity created as a "child" will inherit
its parent's namespace.</li>
<li><em>Kind</em>: A string describing the category or purpose of the entity.</li>
<li><em>Identifier</em>: Unlike namespace and kind which are shared between like
entities, the identifier is unique to each entity. This is either a string
called a "key name", or an integer numeric ID.</li>
<li><em>Ancestor Path</em>: An optional component of the key, identifying its parents to
form a database hierarchy.</li>
</ul>
<hr>
<h1 id="create-iam-service-account">Create IAM Service Account</h1>
<p>Before your code can access Datastore/Firestore, access needs to be configured
in the <a href="https://console.cloud.google.com/iam-admin">Identity Access Management</a>
tool.</p>
<p>First, create a <a href="https://console.cloud.google.com/iam-admin/serviceaccounts">Service Account</a>:</p>
<ol>
<li>Open the Service Account page</li>
<li>Select your project</li>
<li>Click "Create Service Account"</li>
<li>Service Account Details:
<ol>
<li>Service account name: <code>PythonDatastoreAdmin</code></li>
<li>Service account ID: <code>pythondatastoreadmin</code></li>
<li>Service account description: SA with Full Datastore Access for Python</li>
</ol>
</li>
<li>Grant this service account access to project (optional)
<ol>
<li>Select a role: Datastore > Cloud Datastore Owner</li>
<li>Continue</li>
</ol>
</li>
<li>Grant users access to this service account (optional)
<ol>
<li>Download a key file for Python to use. Use JSON as the key type.</li>
<li>Done</li>
</ol>
</li>
</ol>
<p>I downloaded my key to <code>$HOME/gcp_keys/datastore_admin.json</code></p>
<hr>
<h1 id="python-environment-setup">Python Environment Setup</h1>
<p>Build a virtualenv. If you're in a git repo, don't forget to ignore this
directory.</p>
<pre><code class="language-bash">python3 -m venv env3
source env3/bin/activate
</code></pre>
<p>Write the requirement file</p>
<p><code>vi requirements.txt</code></p>
<pre><code class="language-text">google-cloud-firestore
</code></pre>
<p>Install the requirements</p>
<pre><code class="language-bash">pip install -r rqeuirements.txt
</code></pre>
<hr>
<h1 id="python-code-examples">Python Code Examples</h1>
<p>A fake project name of "example" will be used. Be sure to use a valid one.</p>
<p>Some notes about the Firestore python API:</p>
<ul>
<li><code>firestore.client().collection</code> returns a <a href="https://googleapis.dev/python/firestore/latest/collection.html">CollectionReference</a>,
which allows you to operate on the named collection.</li>
<li><code>firestore.client().collection().list_documents</code> returns a <a href="google.api_core.page_iterator.GRPCIterator">Page Iterator</a>.
Casting a page iterator to a list will trigger a read operation, returning a
list of <code>DocumentReference</code> objects.</li>
<li><code>firestore.client().collection().document</code> returns
a <a href="https://googleapis.dev/python/firestore/latest/document.html">DocumentReferece</a>,
which interacts with the actual Firestore documents. The <code>DocumentReference</code>
object can operate (get/set/create) on the actual Document.</li>
<li><code>firestore.client().collection().document().get()</code> will return
a <a href="https://developers.google.com/android/reference/com/google/firebase/firestore/DocumentSnapshot">DocumentSnapshot</a>
of the document. The document snapshot is how you read data from a document.
Document snapshots are a representation of the document state taken at the
instant of their creation.</li>
<li><code>firestore.client().collection().document().create()</code> makes a new document.
If the object already  existed, a <code>google.api_core.exceptions.AlreadyExists</code>
exception is thrown.</li>
<li>Write operations (create &#x26; set) return a <a href="https://cloud.google.com/firestore/docs/reference/rest/v1/WriteResult">WriteResult</a>.</li>
</ul>
<pre><code class="language-python">from os import environ
from google.cloud import firestore

# The key file is an exported JSON file from the IAM tool
key_file_path = '{}/gcp_keys/datastore_admin.json'.format(environ['HOME'])

# When running this on a local system, the cloud.google package from GCP
# expects the GOOGLE_APPLICATION_CREDENTIALS environment variable to be set
# to the exported IAM key file's path
environ['GOOGLE_APPLICATION_CREDENTIALS'] = key_file_path

# google.cloud.datastore.Client expects the project name as an argument
project = 'example'

# Instantiate the datastore client. This step authenticates to GCP.
# See the Credential Error note below if authentication fails
client = firestore.Client(project)

# Get a reference to the test collection. This returns a
# google.cloud.firestore_v1.collection.CollectionReference object
ref_col_test = client.collection('test')

# Get a reference to a document which may or may not exist. This returns a
# google.cloud.firestore_v1.document.DocumentReference object
ref_doc_document1 = ref_col_test.document('document1')

# Perform the actual database query to get the document. get() returns a
# google.cloud.firestore_v1.document.DocumentSnapshot object.
snapshot_document1 = ref_doc_document1.get()

# Note that there is no data
assert document1.to_dict() is None

# You can use document.exists to check if the document already exists.
# Here's an example of doing a safe insert of some sample data:
sample_data = {
    'nums': {'num1': 100, 'num2': 200},
    'strs': {'animal1': 'dog', 'animal2': 'cat'}}
if snapshot_document1.exists is False:
    createe_write_result = ref_doc_document1.create(sample_data)

document1_data = ref_doc_document1.get().to_dict()

# Finally, update the data
sample_data['strs']['animal3'] = 'bird'
set_write_result = ref_doc_document1.set(sample_data)

# List the documents in the collection. The iterator returns a paginated list
# of document references. You can then operate on those references.
doc_list_iterator = ref_col_test.list_documents()
doc_ref_list = list(doc_list_iterator)
sample_document_snapshot = doc_ref_list[0].get()

# Show that the changes were applied to the data. This will return 'bird'
animal3 = sample_document_snapshot.to_dict()['strs']['animal3']
</code></pre>
<h2 id="troubleshooting">Troubleshooting</h2>
<h3 id="credentials-error">Credentials Error</h3>
<p>If your local google cloud authentication isn't set up correctly, you'll get an
error saying:</p>
<pre><code class="language-text">google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
</code></pre>
<p>If that happens, review the IAM Service account steps.</p>
<h3 id="iterator-has-already-started">Iterator has already started</h3>
<p>If you try and cast a <code>GRPCIterator</code> to list twice, you get a <code>ValueError</code>.</p>
<pre><code class="language-text">ValueError: ('Iterator has already started', &#x3C;google.api_core.page_iterator.GRPCIterator object)
</code></pre>
<p>The <code>ValueError</code> is thrown because <code>._started</code> is <code>True</code> in the iterator. It has
already done its thing and won't work again. I didn't see any good way to
restart the iterator without discarding it.
You can see the details in the <a href="https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/page_iterator.py">page iterator source code</a>.</p>
<p>Best course is just to get a new iterator from the reference object.
See in the following example how the ID changes. The <code>doc_list_iterator</code> var is
being pointed at a new iterator. The old iterator gets gc'd by Python as usual.</p>
<pre><code>>>> id(doc_list_iterator)
4422123408
>>> doc_list_iterator = ref_col_test.list_documents()
>>> id(doc_list_iterator)
4422123472
</code></pre>
<p>It will work as expected.</p>


<p>This post covers getting started with a very simple Google Cloud Functions API,
a great serverless solution with affordable pricing. This API will pull the
IP address from the HTTP request, and return <code>Hello &#x3C;source IP></code>.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#pricing-details">Pricing Details</a></li>
<li><a href="#deploy-hello-world-function">Deploy Hello World Function</a>
<ul>
<li><a href="#mainpy">main.py</a></li>
<li><a href="#requirementspy">requirements.py</a></li>
</ul>
</li>
<li><a href="#accessing-the-function">Accessing the Function</a>
<ul>
<li><a href="#from-the-console-web-ui">From the Console Web UI</a></li>
<li><a href="#from-the-request-url">From the Request URL</a></li>
</ul>
</li>
</ul>
<h1 id="pricing-details">Pricing Details</h1>
<p>Cloud Functions pricing is pretty affordable for most use cases, assuming you
write your functions in such a way that they don't get invoked a few too many
millions of times.</p>
<p>Check the official <a href="https://cloud.google.com/functions/pricing">Google docs</a>
for accurate and up-to-date pricing, but here's what it looked like at the time
of my writing this.</p>
<p>Pricing is broken into a few fees:</p>
<ul>
<li><em>Invocations</em>: A flat fee per function execution. $0.0000004 per invocation,
excluding the first 2 million.</li>
<li><em>Compute Time</em>: At the lowest clock speed (200MHz x 128MB RAM), compute costs
$0.000000231/100ms, ceil 100.</li>
<li><em>Networking</em>: $0.12/GB egress for data, while ingress is free.</li>
</ul>
<hr>
<h1 id="deploy-hello-world-function">Deploy Hello World Function</h1>
<p>Open <a href="https://console.cloud.google.com/functions/">Google Cloud Functions</a> in
your browser.</p>
<p>Create a function.</p>
<ul>
<li>Name: HelloWorld</li>
<li>Memory Allocated: 128 MB</li>
<li>Trigger: HTTP</li>
<li>Authentication: Allow unauthenticated invocations</li>
<li>Source code: Inline Editor</li>
<li>Runtime: Python 3.7</li>
<li>Other settings: Leave them as defaults</li>
</ul>
<h2 id="mainpy">main.py</h2>
<p>The API function defined by <code>Function to execute</code> accepts a
<a href="https://flask.palletsprojects.com/en/1.1.x/api/#incoming-request-data">flask.Request</a>
argument.</p>
<pre><code class="language-python">def hello_world(request):
    """" Return a Hello message to sender of the API call """
		return 'Hello {}!'.format(request.remote_addr)
</code></pre>
<h2 id="requirementspy">requirements.py</h2>
<p>You can leave this blank. It's a list of libraries that will be used.</p>
<hr>
<h1 id="accessing-the-function">Accessing the Function</h1>
<h2 id="from-the-console-web-ui">From the Console Web UI</h2>
<ol>
<li>Navigate to the <a href="https://console.cloud.google.com/functions">Functions page</a>.</li>
<li>Click on the 3 dots next to the function name, and choose "Test function".</li>
<li>Populate and JSON for the GET request in "Triggering event". The above
example doesn't accept any input data so <code>{}</code> is all you need.</li>
<li>Click "Test the function"</li>
</ol>
<h2 id="from-the-request-url">From the Request URL</h2>
<ol>
<li>Navigate to the <a href="https://console.cloud.google.com/functions">Functions page</a>.</li>
<li>Click the function's name to navigate to the "Function details" page</li>
<li>Go to the 'Trigger" tab</li>
<li>Find the URL. Copy it.</li>
</ol>
<p>You can now test the URL with curl, your browser, httpie, python, etc.</p>
<pre><code class="language-bash">https://&#x3C;region>-&#x3C;project>.cloudfunctions.net/HelloWorld
</code></pre>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to upload and download files from Amazon's S3 using
Python. Setup is done from the web UI.</p>
<h1 id="create-a-bucket">Create a Bucket</h1>
<p>Go to the <a href="https://s3.console.aws.amazon.com">AWS S3 page</a>.</p>
<p>Click on "+ Create Bucket" and fill out the form. For access, choose
Block all public access. For the rest of the options, the defaults are fine.</p>
<hr>
<h1 id="create-iam-credentials">Create IAM Credentials</h1>
<p>IAM stands for Identity and Access Management.</p>
<h2 id="create-a-policy">Create a Policy</h2>
<p>From the left nav-bar, go to policies. The default policies don't grant
programmatic access, so you need to make a new one.</p>
<p>For admin rights, I just used the visual policy builder and granted it
access to everything. I'll come back and update this guide with how to lock
that down later.</p>
<hr>
<h1 id="add-the-policy-to-a-group">Add the policy to a group</h1>
<p>Go to the groups on the left. Either make a new one or edit an existing one.
Under Permissions, click Attach Policy. If you want, you can also add
<code>AmazonS3FullAccess</code> to see the difference between the two.</p>
<h2 id="createedit-a-user">Create/Edit a User</h2>
<p>On the left nav-bar, go to Users > Add User. Or edit a user that exists.
Fill in the wizard and assign the group.</p>
<hr>
<h1 id="use-python-to-interact-with-s3">Use Python to interact with S3</h1>
<h2 id="install-boto3">Install Boto3</h2>
<p>Python needs boto3 installed to interact with AWS.</p>
<pre><code class="language-bash">pip install boto3
</code></pre>
<h2 id="uploading-and-downloading-files">Uploading and Downloading Files</h2>
<pre><code class="language-python">import boto3

# Fill in the key and key id, or read them from stdin/file/whatever
key_id = ''
key = ''

# Create the authenticated s3 client
session = boto3.Session(aws_access_key_id=key_id, aws_secret_access_key=key)
client = session.client('s3')

# Define the file/object
filename = '/example/file/path'
bucket_name = 'my_bucket'
key = 'my_key'

# Upload Example
client.upload_file(filename, bucket_name, key)

# Download Example
client.download_file(bucket_name, object_name, path)
</code></pre>


<p>You've deployed a server in a customer environment and you don't have remote
network access configured. The customer is OK with you being able to remote
into their network, but its not practical to set you up with a VPN. Enter the
"Dial-Home Device".</p>
<h1 id="what-is-the-dial-home-device">What is the Dial-Home Device</h1>
<p>It's just an Ubuntu server that repeatedly tries to dial an internet connected
jump box in order to open a reverse session. As soon as the networking is set
up, it will dial home and open itself up for remote administration.</p>
<p>Also, <a href="https://stargate.fandom.com/wiki/Dial_Home_Device">this.</a></p>
<hr>
<h1 id="deploy-the-pivot-server">Deploy the Pivot Server</h1>
<h2 id="whats-a-pivot-server">What's a Pivot Server?</h2>
<p>This is an internet-connected server such as a VPS that the DHD will SSH to.</p>
<p>You'll SSH to it, then from there you can SSH through the DHD's reverse tunnel.</p>
<h2 id="steps-to-deploy">Steps to Deploy</h2>
<h3 id="launch-a-vps">Launch a VPS</h3>
<p>Using AWS, GCP, Azure, Linode, Digital Ocean, so on, launch an
internet-connected Ubuntu server. It needs to have a static, internet routed IP
address. Its not a bad idea to make a DNS entry for it too, so you can change
the IP down the road.</p>
<p>SSH to the server to confirm that it works.</p>
<h3 id="configure-remote-user">Configure Remote User</h3>
<p>These steps will create a user named <code>remoteuser</code></p>
<pre><code class="language-bash"># Create the user
useradd remoteuser

# Set a password for the user
passwd remoteuser

# Create the home &#x26; ssh directories for the user
mkdir -p /home/remoteuser/.ssh
chown -R remoteuser:remoteuser /home/remoteuser

# Disable shell access to the user, so it can't run commands on the pivot
usermod -s /bin/false remoteuser
</code></pre>
<hr>
<h1 id="deploy-the-dial-home-device">Deploy the Dial-Home Device</h1>
<p>Install Ubuntu Server on a physical server or VM at the remote location. This
server should have network connectivity to whatever resource you want to
manage, and outbound internet access (it can ping google).</p>
<h2 id="create-an-ssh-key-on-the-dhd">Create an SSH key on the DHD</h2>
<p>On the DHD server, you may already have SSH keys. Look in <code>~/.ssh</code>. If not,
run the following to create one. Don't use a passphrase, just hit enter when
prompted.</p>
<pre><code class="language-bash">ssh-keygen
</code></pre>
<p>Collect the public key for the next step:</p>
<pre><code class="language-bash">cat ~/.ssh/id_rsa.pub
</code></pre>
<p>For convenience, I like to copy the key to the root user too. Optional.</p>
<h2 id="authorize-the-ssh-public-key-on-the-pivot-serer">Authorize the SSH Public Key on the Pivot Serer</h2>
<p>Back on the Pivot server, edit the remote user's <code>authorized_keys</code> file to
grant this SSH key-pair access. Paste the SSH key at the end of the file</p>
<pre><code class="language-bash">vi /home/remoteuser/.ssh/authorized_keys
</code></pre>
<h2 id="test-a-remote-connection">Test a remote connection</h2>
<h3 id="initiate-the-connection">Initiate the connection</h3>
<p>From the DHD server, open the reverse tunnel manually to make sure it works.</p>
<ul>
<li>Replace the username as needed</li>
<li>The <code>$pivot</code> variable should be the IP or FQDN of the pivot server</li>
<li>Pick a random port in the range 1024-49151 for the listen port.</li>
</ul>
<pre><code class="language-bash">user="remoteuser"
pivot="support.example.com"
port=22222
ssh -fN -R 0.0.0.0:$port:127.0.0.1:22 $user@$pivot
</code></pre>
<h3 id="connect-through-the-pivot">Connect through the pivot</h3>
<p>From the pivot server, SSH to the DHD. This example assumes your username
is <code>myuser</code>.</p>
<pre><code class="language-bash">ssh -p 22222 myuser@127.0.0.1
</code></pre>
<p>Consider adding your public key to the <code>~/.ssh/authorized_keys</code> file now so
passwords aren't needed.</p>
<p>Now you can jump through the pivot into the DHD.</p>
<hr>
<h1 id="persist-the-connection">Persist the Connection</h1>
<h2 id="build-the-dial-home-command">Build the dial-home command</h2>
<p>From any directory, create the script:</p>
<p><code>vi dial-home</code></p>
<pre><code class="language-bash">#!/usr/bin/env bash

# Check if the reverse tunnel  is already open
port=22222
if [[ $(ps aux | grep ssh | grep $port) ]]; then
  # reverse tunnel is open
  exit 0
fi

# tunnel is not open, open it
user="remoteuser"
pivot="support.example.com"
ssh -fN -R 0.0.0.0:$port:127.0.0.1:22 $user@$pivot
</code></pre>
<p>Make the script executable</p>
<pre><code class="language-bash">chmod +x dial-home
</code></pre>
<p>Move the script to somewhere in the <code>$PATH</code></p>
<pre><code class="language-bash">sudo mv dial-home /usr/local/bin/
</code></pre>
<p>Now the <code>dial-home</code> command will open the tunnel.</p>
<h2 id="make-cron-dial-home-every-minute">Make cron dial-home every minute</h2>
<p>The connection will die for lots of reasons. Make it bring itself back online
automatically using cron.</p>
<p>From the DHD, make sure cron is running (by default, it is)</p>
<pre><code class="language-bash">systemctl status cron
</code></pre>
<p>Edit the crontab</p>
<p><code>sudo vi /etc/crontab</code></p>
<p>and insert this line, where <code>myuser</code> is your username on the DHD.
If you use root, you have to also give root the ssh private key.</p>
<pre><code class="language-text">* * * * *   myuser dial-home
</code></pre>
<p>Reload cron</p>
<pre><code class="language-bash">service cron reload
</code></pre>
<p>Then kill the session (using <code>ps aux</code> and <code>kill</code>) then watch it come back.</p>
<p>Done. Now the server is a dial-home device. It will constantly dial your pivot
server and re-open the session whenever its closed. To lock the server out,
just remove its public key from <code>remoteuser</code>'s <code>authorized_keys</code> file.</p>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page lists some of the errors that I've personally encountered,
and how I fixed them. I might also include errors I never fixed in case anyone
wants to talk about it.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#docker-login-fails">Docker Login Fails</a></li>
<li><a href="#runaway-memory-usage-in-ceph-osd-process">Runaway memory usage in Ceph OSD process</a></li>
<li><a href="#chrome-blocks-https127001-with-neterr_cert_revoked">Chrome blocks https://127.0.0.1 with NET::ERR_CERT_REVOKED</a></li>
<li><a href="#failed-command-aws-ecr-get-login">Failed command: aws ecr get-login</a></li>
<li><a href="#pure-storage-iscsi-with-kolla-openstack---fdisk-freezes">Pure Storage iSCSI with Kolla OpenStack - fdisk freezes</a></li>
<li><a href="#weird-chars-when-pasting-on-mac-os">Weird chars when pasting on Mac OS</a></li>
</ul>
<h1 id="docker-login-fails">Docker Login Fails</h1>
<pre><code>Error saving credentials: error storing credentials - err: exit status 1, out: `Cannot autolaunch D-Bus without X11 $DISPLAY`
</code></pre>
<p>I had this happen after running both Ceph-Ansible then Kolla-Ansible on an Ubuntu server, so it could be either of those that set up the conditions for this to happen. <a href="https://github.com/docker/cli/issues/1136">this solution</a> worked for me:</p>
<pre><code class="language-bash">apt-get install -y gnupg2 pass
</code></pre>
<hr>
<h1 id="runaway-memory-usage-in-ceph-osd-process">Runaway memory usage in Ceph OSD process</h1>
<p>It looked like a memory leak. Each OSD process was using around 30GB of RAM.</p>
<p>It turns out that Ceph-Ansible has the following logic when generating its
<code>ceph.conf</code> file:</p>
<pre><code># this math
roles/ceph-config/templates/ceph.conf.j2:{% set _osd_memory_target = (ansible_memtotal_mb * 1048576 * hci_safety_factor / _num_osds) | int %}
# sets osd memory target
roles/ceph-config/templates/ceph.conf.j2:osd memory target = {{ _osd_memory_target | default(osd_memory_target) }}
</code></pre>
<p>In my case, this set the value WAY too high. In one of my deployments, it was
set to <code>37853425827</code>, which is like 35GB (PER OSD SERVICE!).</p>
<p>The 4GB default is much more sane, <code>4294967296</code>.</p>
<pre><code># ceph.conf
[osd]
osd memory target = 4294967296
</code></pre>
<hr>
<h1 id="chrome-blocks-https127001-with-neterr_cert_revoked">Chrome blocks <a href="https://127.0.0.1">https://127.0.0.1</a> with NET::ERR_CERT_REVOKED</h1>
<p>This comes up when I reverse tunnel to a remove web server, forwarding it to my workstation. Chrome has decided to not let you do that. They give this errror:</p>
<pre><code class="language-text">Your connection is not private
Attackers might be trying to steal your information from 127.0.0.1 (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_REVOKED
</code></pre>
<p>To bypass that, open <code>chrome://flags/#allow-insecure-localhost</code> and set allow-insecure-localhost to enabled. Click relaunch. This will nuke your current browser…</p>
<p>Once it relaunches you'll be able to open the site.</p>
<hr>
<h1 id="failed-command-aws-ecr-get-login">Failed command: aws ecr get-login</h1>
<p>This error was caused by a bad NTP config. Renew the NTP lease to fix it.</p>
<pre><code>An error occurred (InvalidSignatureException) when calling the GetAuthorizationToken operation: Signature expired: 20190911T174538Z is now earlier than 20190912T004835Z (20190912T010335Z - 15 min.)
</code></pre>
<hr>
<h1 id="pure-storage-iscsi-with-kolla-openstack---fdisk-freezes">Pure Storage iSCSI with Kolla OpenStack - fdisk freezes</h1>
<p>This was a weird one. When using the Kolla containers project to make a
Cinder-volume container that has the <code>purecinder</code> plugin, I was able to get
OpenStack to create the volumes in Pure but it couldn't mount them to the hosts
for things like creating a volume with the <code>--image</code> argument. That made the
volumes I could create basically useless.</p>
<p>These errors were in the logs:</p>
<pre><code class="language-text"># Lots of these in cinder's log
Trying to connect to iSCSI portal

# also from cinder's log, also recurring:
iscsiadm stderr output when getting sessions: iscsiadm: No active sessions

# in DMESG
FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
</code></pre>
<p>The weirdest part was that the <code>fdisk -l</code> command would totally lock up and
the process running it couldn't be killed even with <code>kill -9</code>. The iSCSI volume
never got properly mounted but Pure support said they could see some data
coming into it.</p>
<p><strong>Root cause</strong>: MTU mismatch. Of course. I had set MTU 9000 on the iSCSI ports,
my host ports, and the switchports heading to the Pure Storage iSCSI
interfaces. I had forgotten to set it on the switch heading to the Dell servers
being used for OpenStack.</p>
<hr>
<h1 id="weird-chars-when-pasting-on-mac-os">Weird chars when pasting on Mac OS</h1>
<p>When you paste sometimes you get <code>^[[200~</code> at the start and <code>^[[201~</code> at the
end of the pasted content. Or maybe <code>0~</code> and <code>1~</code>.</p>
<p>To resolve, run:</p>
<pre><code class="language-bash">printf "\e[?2004l"^C
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page contains operations-related mini-guides and minor posts.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</a></li>
<li><a href="#convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</a></li>
<li><a href="#journalctl-commands">Journalctl commands</a></li>
<li><a href="#get-real-ram-use-of-processes">Get real RAM use of processes</a>
<ul>
<li><a href="#of-ceph">of ceph</a></li>
<li><a href="#of-vms">of VMs</a></li>
</ul>
</li>
<li><a href="#bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</a></li>
<li><a href="#create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</a>
<ul>
<li><a href="#use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</a></li>
</ul>
</li>
<li><a href="#test-mtu-from-ubuntu">Test MTU from Ubuntu</a></li>
<li><a href="#find-big-files-on-mac-os">Find big files on Mac OS</a></li>
<li><a href="#renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</a></li>
<li><a href="#create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</a></li>
<li><a href="#passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</a></li>
<li><a href="#vim-tasks">VIM Tasks</a>
<ul>
<li><a href="#find--replace">Find &#x26; Replace</a></li>
<li><a href="#spell-check-commands">Spell-Check commands</a></li>
</ul>
</li>
<li><a href="#use-an-insecure-docker-registry">Use an Insecure Docker Registry</a></li>
<li><a href="#ansible-ignore-errors">Ansible ignore errors</a></li>
<li><a href="#how-to-tail--f-on-dmesg">How to tail -f on dmesg</a></li>
<li><a href="#ubuntu-downgrade-a-package">Ubuntu Downgrade a package</a></li>
<li><a href="#make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</a></li>
<li><a href="#bash-print-all-but-first-line">Bash: Print all but first line</a></li>
<li><a href="#bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</a></li>
</ul>
<h1 id="find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</h1>
<pre><code>cat /proc/net/bonding/bond0
</code></pre>
<hr>
<h1 id="convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</h1>
<pre><code class="language-bash">openssl pkcs7 -print_certs -in certnew.p7b -out cert.pem
</code></pre>
<hr>
<h1 id="journalctl-commands">Journalctl commands</h1>
<p>Watch the logs as they come in:</p>
<pre><code class="language-bash">journalctl -f
</code></pre>
<p>When journalctl logs too much and you need to pair it down:</p>
<pre><code class="language-bash"># to 30 days
journalctl --vacuum-time=30d

# to 500MB
journalctl --vacuum-size=500M
</code></pre>
<p>Show for a specific service</p>
<pre><code class="language-bash">journalctl ceph*
</code></pre>
<hr>
<h1 id="get-real-ram-use-of-processes">Get real RAM use of processes</h1>
<p>You can always use top or htop or whatever. I use this to specifically get the
info about those processes.</p>
<h2 id="of-ceph">of ceph</h2>
<pre><code>ps aux | grep ceph | grep -v -e grep -e qemu -e "\[" | while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); id=$(echo $line | awk '{print $16}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tid: $id,    \tRAM: ($size)"; done
</code></pre>
<h2 id="of-vms">of VMs</h2>
<p>for KVM</p>
<pre><code>ps aux | grep qemu | grep -v -e grep| while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); guest=$(echo $line | awk '{print $13}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tguest: $guest,   \t\tRAM: ($size)" ; done
</code></pre>
<h1 id="bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</h1>
<p>As of Mac Catalina, developing a site using self-signed certs is a bigger pain
than before. Yeah you can go get the key and add it, but that's a nuisance.
It looks like the Chrome team built something they call an
"interstitial bypass keyword" to deal with that. That's one cool feature name.</p>
<p><strong>Obvious disclaimer</strong> If you're not a developer doing this to access your own
site, don't do this.</p>
<p>To bypass the warning, type</p>
<pre><code>thisisunsafe
</code></pre>
<p>Apparently it used to be <code>badidea</code> but they rotated it since it got too well
known. They'll probably do it again someday. I suspect people are using this
on sites they "trust" without really knowing any better and getting hurt by
it.</p>
<hr>
<h1 id="create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</h1>
<p>Useful for simulating a real drive when building a test environment. I use this to test OpenStack in a VM as a Ceph or LVM back-end.</p>
<pre><code class="language-bash"># create empty file of all zeros
sudo dd if=/dev/zero of=/root/virtual-disk bs=1M count=512
# show list of loopback devices. Note its not there
losetup --list
# create the loopback device
losetup /dev/loop0 /root/virtual-disk
# confirm it worked
losetup --list
</code></pre>
<p>You can now treat it like a normal volume. Mount it, put a filesystem on it, whatever. Example:</p>
<pre><code class="language-bash">pvcreate /dev/loop0
vgcreate cinder-volumes /dev/loop0
</code></pre>
<h2 id="use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</h2>
<p>I found <a href="https://unix.stackexchange.com/questions/418322/persistent-lvm-device-with-loopback-devices-by-fstab">this useful systemd unit example</a> that didn't quite work for me, but was close. Here's mine:</p>
<p><code>vi /etc/systemd/system/loops-setup.service</code></p>
<pre><code class="language-text">[Unit]
Description=Setup loopback devices

DefaultDependencies=no
Conflicts=umount.target

Requires=lvm2-lvmetad.service mnt-host.mount
Before=local-fs.target umount.target
After=lvm2-lvmetad.service mnt-host.mount

[Service]
ExecStart=/sbin/losetup /dev/loop0 /root/virtual-disk
ExecStop=/sbin/losetup -d /dev/loop0

RemainAfterExit=yes
Type=oneshot

[Install]
WantedBy=local-fs-pre.target
</code></pre>
<p>Then enable it</p>
<pre><code class="language-bash">systemctl enable loops-setup
</code></pre>
<p>Reboot, then check <code>vgdisplay</code> (if that's your use case). It'll be there.</p>
<hr>
<h1 id="test-mtu-from-ubuntu">Test MTU from Ubuntu</h1>
<p>You would not believe the strange problems that an MTU issue can be the root cause of.</p>
<pre><code class="language-bash">ping -M do -s 1472 &#x3C;remote server>
ping -M do -s 8900 &#x3C;remote server>
</code></pre>
<hr>
<h1 id="find-big-files-on-mac-os">Find big files on Mac OS</h1>
<p>My SSD is always almost full. Here are useful commands for cleaning up:</p>
<pre><code class="language-bash"># Find all the files over 1G
sudo find / -type f -size +1G -exec ls -lh {} \; | awk '{ print $9 ": " $5 }' 2>/dev/null

# List files in Downloads directory by size
ls -lhS ~/Downloads
</code></pre>
<h1 id="renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</h1>
<pre><code class="language-bash">service ntp stop
ntpd -gq
service ntp start
</code></pre>
<hr>
<h1 id="create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</h1>
<pre><code class="language-bash">aws ecr describe-repositories --repository-names $repo_name \
    || aws ecr create-repository --repository-name $repo_name
</code></pre>
<hr>
<h1 id="passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</h1>
<p>Run <code>visudo</code> as root.</p>
<pre><code class="language-bash"># as root
export EDITOR=vim
visudo
</code></pre>
<p>Add the user (replace <code>exampleUser</code>) at the end.</p>
<pre><code class="language-text">exampleUser ALL=(ALL) NOPASSWD: ALL
</code></pre>
<p>For those who don't use vim, a quick reminder:
<code>i</code> enter to insert mode , <code>[esc]</code> to exit insert mode, <code>:wq</code> to save and quit.</p>
<hr>
<h1 id="vim-tasks">VIM Tasks</h1>
<h2 id="find--replace">Find &#x26; Replace</h2>
<pre><code class="language-bash"># replace findThis with ReplaceWithThis
:%s/findThis/ReplaceWithThis/g
</code></pre>
<h2 id="spell-check-commands">Spell-Check commands</h2>
<pre><code class="language-text"># enable/disable
:set spell spelllang=en_ca
:setlocal spell spelllang=en_ca
:set nospell

# next/last
]s
[s

# suggest change
z=

# add/remove word from dictionary
zg
zw`
</code></pre>
<hr>
<h1 id="use-an-insecure-docker-registry">Use an Insecure Docker Registry</h1>
<p>To use a registry with no HTTP cert:</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
    "insecure-registries" : ["myregistrydomain.com:5000"]
}
</code></pre>
<hr>
<h1 id="ansible-ignore-errors">Ansible ignore errors</h1>
<p>The command is <code>ignore_errors: yes</code></p>
<p>Just throw this into the task that's failing. It will still fail, but the play
won't stop.</p>
<p>Example:</p>
<pre><code class="language-yml">- name: Ensure the docker service is running
  script: ../tools/validate-docker-execute.sh
    ignore_errors: yes
</code></pre>
<hr>
<h1 id="how-to-tail--f-on-dmesg">How to tail -f on dmesg</h1>
<pre><code class="language-bash">dmesg -wH
</code></pre>
<hr>
<h1 id="ubuntu-downgrade-a-package">Ubuntu Downgrade a package</h1>
<pre><code class="language-bash"># Find the available versions under Versions:
apt-cache showpkg &#x3C;package>

# Install one of those
apt-get install &#x3C;package>=&#x3C;version>
</code></pre>
<hr>
<h1 id="make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</h1>
<p>Useful for OS that ship with python3 but not python2.
Edit the inventory, find a group of the groups and apply this var.</p>
<pre><code class="language-ini">[baremetal:vars]
ansible_python_interpreter=/usr/bin/python3
</code></pre>
<hr>
<h1 id="bash-print-all-but-first-line">Bash: Print all but first line</h1>
<p>Use <code>tail -n +2</code>.
This was unintuitive to me because +1 seems like it should do it.</p>
<pre><code class="language-bash"># Example
docker images | awk '{print $3}' | tail -n +2
</code></pre>
<hr>
<h1 id="bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</h1>
<p>Use trap.</p>
<ul>
<li><a href="http://redsymbol.net/articles/bash-exit-traps/">This is a nice post covering lots of examples</a>.</li>
<li><a href="https://github.com/GoogleCloudPlatform/endpoints-quickstart/blob/master/scripts/deploy_api.sh">Here's where I first saw it used</a></li>
</ul>
<pre><code class="language-bash">temp_file=$(mktemp)

cleanup() {
  rm "$temp_file"
}

trap cleanup EXIT
</code></pre>


Kyle Pericak

"It works in my environment"

Categories

Tags