<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<p>In this project I'll be trying out each of the Google Cloud Platform's
services. I'll be updating it pretty regularly.</p>
<hr>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes, or will include, the following posts.
If any aren't finished, check back later! I'll also certainly be adding more
as time goes on.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left"></td>
<td align="left"><strong>Continuous Integration</strong></td>
</tr>
<tr>
<td align="left">2019-08-06</td>
<td align="left"><a href="/google-cloud-build.html">Google Cloud Build: Basics</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Database</strong></td>
</tr>
<tr>
<td align="left">2019-10-14</td>
<td align="left"><a href="/gcp-firestore-python.html">Google Cloud Firestore Basics - Python</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Containers</strong></td>
</tr>
<tr>
<td align="left">2019-08-05</td>
<td align="left"><a href="/google-container-registry.html">Google Container Registry: Basics</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Cloud Storage</strong></td>
</tr>
<tr>
<td align="left">2019-08-09</td>
<td align="left"><a href="/google-cloud-storage-website.html">Hosting websites on Cloud Storage</a></td>
</tr>
</tbody>
</table>


<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<p>In this project I'll be trying many of AWS's services.</p>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes the following posts.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left"></td>
<td align="left"><strong>Running Servers on AWS</strong></td>
</tr>
<tr>
<td align="left">2020-01-09</td>
<td align="left"><a href="/aws-ec2-python.html">Create &#x26; Terminate EC2 Instances from Python</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Running Containers on AWS</strong></td>
</tr>
<tr>
<td align="left">2019-08-26</td>
<td align="left"><a href="/aws-ecr.html">Save a Docker image to Elastic Container Registry (ECR)</a></td>
</tr>
<tr>
<td align="left">2019-09-10</td>
<td align="left"><a href="/aws-fargate.html">Run a Docker container Using AWS Fargate</a></td>
</tr>
<tr>
<td align="left">2020-01-15</td>
<td align="left"><a href="/aws-codepipeline-ecr.html">CodePipeline: Continuous Delivery to AWS ECR</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Cloud Storage</strong></td>
</tr>
<tr>
<td align="left">2019-09-26</td>
<td align="left"><a href="/aws-s3-python.html">Using AWS S3 from Python</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Serverless Applications</strong></td>
</tr>
<tr>
<td align="left">2020-01-08</td>
<td align="left"><a href="/aws-lambda.html">Lambda basics</a></td>
</tr>
<tr>
<td align="left">2020-01-09</td>
<td align="left"><a href="/aws-lambda-iam.html">IAM for Lambda</a></td>
</tr>
<tr>
<td align="left">2020-01-21</td>
<td align="left"><a href="/aws-api-gateway-iam.html">IAM Authentication using API Gateway</a></td>
</tr>
</tbody>
</table>


<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<h4 id="table-of-contents">Table of contents</h4>
<ul>
<li><a href="#what-is-openstack">What is OpenStack?</a></li>
<li><a href="#project-posts--progress">Project Posts &#x26; Progress</a></li>
<li><a href="#why-use-an-openstack-private-cloud">Why Use an OpenStack Private Cloud?</a></li>
<li><a href="#complexity-and-openstack-ad">Complexity and OpenStack (Ad)</a></li>
<li><a href="#installing-openstack">Installing OpenStack</a>
<ul>
<li><a href="#non-production-installation-options">Non-Production Installation Options</a>
<ul>
<li><a href="#devstack">DevStack</a></li>
<li><a href="#small-physical-server--kolla-ansible">Small Physical Server + Kolla-Ansible</a></li>
<li><a href="#virtual-server--kolla-ansible">Virtual Server + Kolla-Ansible</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#containerized-openstack">Containerized OpenStack</a></li>
</ul>
<h1 id="what-is-openstack">What is OpenStack?</h1>
<p><a href="https://www.openstack.org/">OpenStack</a> is a collection of open-source projects
that work together to form a private cloud solution. It's by far the
biggest open source private cloud option available. The <a href="https://www.openstack.org/foundation/">OpenStack Foundation</a>
has over 105,000 community members from 187 countries around the world.</p>
<p>At a high level, you can think of it as an abstraction layer that sits on top
of other popular cloud technologies such as storage clusters, hypervisors, and
network solutions. For example, KVM (the popular Linux hypervisor) is not part
of OpenStack, but it's used by <strong>Nova</strong>, a component of OpenStack which can
orchestrate against KVM, VMWare, HyperV, and Xen.</p>
<hr>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes the following posts.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">2019-08-13</td>
<td align="left"><a href="/openstack-aio-ka-metal.html">Install OpenStack on Metal - Intel NUC</a></td>
</tr>
<tr>
<td align="left">2019-08-11</td>
<td align="left"><a href="/openstack-aio-ka-vm.html">Install OpenStack inside a VM</a></td>
</tr>
<tr>
<td align="left">2019-08-26</td>
<td align="left"><a href="/openstack-kolla-custom-plugin.html">Modifying OpenStack Kolla Docker Images</a></td>
</tr>
<tr>
<td align="left">2019-12-03</td>
<td align="left"><a href="/openstack-ansible.html">Operating OpenStack from Ansible</a></td>
</tr>
<tr>
<td align="left">2020-01-25</td>
<td align="left"><a href="/openstack-cloudinit-powershell.html">OpenStack Cloud-Init Powershell Example</a></td>
</tr>
<tr>
<td align="left">2019-12-17</td>
<td align="left"><a href="windows-kvm-drivers">Injecting KVM Drivers into Windows</a></td>
</tr>
</tbody>
</table>
<hr>
<h1 id="why-use-an-openstack-private-cloud">Why Use an OpenStack Private Cloud?</h1>
<ol>
<li><strong>Price at scale:</strong> At small scales, private clouds lose on price
against the public clouds. As you start to ramp up though, the public clouds
start to really nickle and dime you. Private clouds have higher up-front
costs but they win out in the long-run.</li>
<li><strong>Predictable Cost:</strong> Spin up a few services on AWS or Azure. How much will
they cost you each month? As it stands, you're guess is going to be wrong.
After the free trial periods and special offers stop muddying the water,
you'll usually discover you're spending more than you thought.</li>
<li><strong>Hardware Re-Use:</strong> Many IT organizations already have hardware sitting
around running expensive hypervisor software. OpenStack is free and can use
that existing hardware, giving you more for less.</li>
<li><strong>Data Sovereignty:</strong> For legal and security reasons, many organizations
must know exactly where their data is sitting at rest. With private clouds,
there's no ambiguity.</li>
<li><strong>Behind Your Firewall:</strong> So you spent a ton of money on some next
generation firewall technologies with intrusion detection/prevention, and
trained the staff up to use it. You run an arguably well secured
data-center. Running OpenStack keeps the cloud behind your existing security
measures.</li>
<li><strong>Latency:</strong> Particularly for edge clouds and VDI solutions, having the
cloud located right on-premises can hugely improve user experience.</li>
<li><strong>Shadow IT:</strong> Developers are going to develop, even if your organization
can't get them the tools they need in time. By offering your own in-house
cloud, you can help prevent
<a href="https://en.wikipedia.org/wiki/Shadow_IT">shadow IT</a> from emerging.</li>
<li><strong>Control:</strong> This one is a double-edged sword. Having more control over your
cloud can be its own benefit, but there's also value in letting the pros
handle it.</li>
<li><strong>Business as usual, without VMWare:</strong> Don't get me wrong, VMWare make a
great product... I've never heard anyone say they were happy with their
VMWare bill. It's insanely expensive when there's an open source alternative
that can check all the same boxes.</li>
</ol>
<hr>
<h1 id="complexity-and-openstack-ad">Complexity and OpenStack (Ad)</h1>
<p>OpenStack is significantly more difficult to get started with than the public
clouds. You can't really install OpenStack without first understanding what
each of the main components does, and even then each one has its own quirks for
their installation. If you want to install in production, you also need to
figure out how to get it all highly available.</p>
<p>Unless you've got a huge team that's eager to tackle this complexity, I'm going
to use this opportunity to plug the software &#x26; solution I've helped build for
<a href="https://breqwatr.com">Breqwatr</a>. They'll stand up your cloud in a couple days
flat, teach your team how to use it, and keep the lights on - all while
charging significantly less than VMWare would cost for the same hardware.</p>
<p>Breqwatr, and companies like it, can be your trusted cloud partner and abstract
away that complexity by handling the installation, upgrades, monitoring, and
day-to-day operation of the cloud.</p>
<hr>
<h1 id="installing-openstack">Installing OpenStack</h1>
<h2 id="non-production-installation-options">Non-Production Installation Options</h2>
<p>If you're not looking for a production deployment and instead want to try
OpenStack out to learn it, there are a few good options.</p>
<h3 id="devstack">DevStack</h3>
<p>The standard developer option is probably <a href="https://docs.openstack.org/devstack/latest/">DevStack</a>.</p>
<p>In truth I haven't used this since I've always needed my OpenStack installs to
mirror a production-ready environment. If I try it out, I'll replace the above
link with one to a guide I write myself.</p>
<h3 id="small-physical-server--kolla-ansible">Small Physical Server + Kolla-Ansible</h3>
<p>Another option is to install OpenStack on a physical server. I've documented
my experience doing this using images from the <a href="https://github.com/openstack/kolla">Kolla project</a>,
and Ansible automation from the <a href="https://github.com/openstack/kolla-ansible">Kolla-Ansible project</a>.</p>
<p>The nice thing about this approach is you can use a server at home, and once
you're done it can be a VM host as your private house cloud.</p>
<p><strong><a href="/openstack-aio-ka-metal.html">Installing OpenStack on a physical server</a></strong></p>
<h3 id="virtual-server--kolla-ansible">Virtual Server + Kolla-Ansible</h3>
<p>This option is great for development use if you already have a cloud available
or want to use a public cloud for testing OpenStack. The VMs you launch on it
will perform terribly though, and there are some networking complications.</p>
<p>Don't bother trying to launch Windows VMs at all in a nested virtualisation
scenario, you'll be lucky if they even boot.</p>
<p><strong><a href="/openstack-aio-ka-vm.html">Installing OpenStack on a Virtual Server</a></strong></p>
<hr>
<h1 id="containerized-openstack">Containerized OpenStack</h1>
<p>Managing OpenStack clusters is a pain. It has so many services, each with their
own dependencies and operating instructions, there's a lot that can go wrong.
One thing that's really helped with that is to containerize each service and
deploy them all using Docker.</p>
<p>The <a href="https://github.com/openstack/kolla">Kolla project</a> does such a great job
of that, that there's no need to do it yourself. That is, until you need to
change an image, for instance to install a Cinder plugin.</p>
<p><strong><a href="/openstack-kolla-custom-plugin.html">Modifying OpenStack Kolla Docker Images</a></strong></p>


<p>I've found that it's pretty common that I need to send emails from my software.
At first it seems easy, there are well documented libraries out there. Pretty
soon though, your super critical outage notification email has wound up in the
SPAM filter.</p>
<p>In this project, I explore the tools available to send email alerts from your
automated systems and how to ensure those emails are trusted by their receiving
mail servers.</p>
<p>This will be an ongoing project that I intend to circle back to as time
permits.</p>
<hr>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes, or will include, the following posts.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">2019-08-15</td>
<td align="left"><a href="/emails-postfix-ubuntu.html">Postfix Send-Only Mail Service: Basic emails from a dedicated VM</a></td>
</tr>
<tr>
<td align="left">2019-08-18</td>
<td align="left"><a href="/emails-bash-cron.html">Sending cron-scheduled ping failure alerts using a bash script</a></td>
</tr>
<tr>
<td align="left">2019-08-20</td>
<td align="left"><a href="/email-spf.html">Using SPF to ensure emails are trusted</a></td>
</tr>
</tbody>
</table>


<p>The objective of this project is to build a technology blog website,
satisfying the following requirements:</p>
<ul>
<li><strong>Affordable</strong>: The website should cost as little as possible to maintain</li>
<li><strong>Maintainable</strong>: Authoring and updating content on the website should be
simple and fast.</li>
<li><strong>Extensible</strong>: This is a software developer's website. It needs to enable
custom code so fun features can be built into it.</li>
<li><strong>Custom Domain</strong>: Some free/cheap blog options require a custom FQDN suffix,
such as <code>x.wordpress.com</code> sites. This blog needs to work with any domain
name.</li>
<li><strong>Fun</strong>: The blog needs to use technologies that are current and interesting.</li>
</ul>
<hr>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes, or will include, the following posts.
If any aren't finished, check back later!</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">2019-08-01</td>
<td align="left"><a href="/docker-pelican-image.html">Building a Pelican Docker image</a></td>
</tr>
<tr>
<td align="left">2019-08-03</td>
<td align="left"><a href="/writing-pelican-content.html">Writing blog content for Pelican</a></td>
</tr>
<tr>
<td align="left">2019-08-04</td>
<td align="left"><a href="/pelican-dev-environment.html">A simple Pelican development environment</a></td>
</tr>
<tr>
<td align="left">2019-08-05</td>
<td align="left"><a href="/google-container-registry.html">Saving Docker images in Google's Container Registry</a></td>
</tr>
<tr>
<td align="left">2019-08-06</td>
<td align="left"><a href="/google-cloud-build.html">Google Cloud Build with GitHub Trigger</a></td>
</tr>
<tr>
<td align="left">2019-08-09</td>
<td align="left"><a href="/google-cloud-storage-website.html">Hosting a website using Google Cloud Storage</a></td>
</tr>
<tr>
<td align="left">2019-12-02</td>
<td align="left"><a href="/cloudflare-https.html">Using Cloudflare to provide HTTPS</a></td>
</tr>
</tbody>
</table>
<hr>
<h1 id="design--workflow">Design &#x26; Workflow</h1>
<p>At a high level, with this setup you just write your content and push it to
git. The rest takes care of itself.</p>
<p>Here's how it works:</p>
<ol>
<li>Blog content is written as text in markdown files. I use Vim.</li>
<li>When the content is ready, code is pushed to a GitHub repository.</li>
<li>A Google Cloud Build trigger watches for commits. When one happens, it loads
the Git master branch to its temporary workspace.</li>
<li>Cloud Build loads a pre-created Pelican Docker image is from Google's
Container Registry, and executes it against the markdown files to generate
a static website's .html and .css files.</li>
<li>Cloud Build uses an rsync image to transfer the files to a Google Storage
bucket.</li>
<li>Google Storage hosts the static site files on a custom domain as a website.</li>
</ol>
<p><img src="/images/blog_workflow.png" alt="Blog architecture diagram"></p>
<hr>
<h1 id="setup-configuring-pelican-and-gcp">Setup: Configuring Pelican and GCP</h1>
<h2 id="pelican-theme">Pelican Theme</h2>
<p>Before Pelican can render the markdown files into a site worth looking at,
it needs a theme to work with.</p>
<p>The easiest way to go is to <a href="http://www.pelicanthemes.com/">select a pre-made theme.</a></p>
<p>Personally, I forked a theme then heavily modified it.
Forking the theme prevents it from changing, and lets you change things as
needed. I'd suggest creating a fork of the theme even if you won't change it.</p>
<h2 id="pelican-docker-image">Pelican Docker Image</h2>
<h3 id="containerize-pelican">Containerize Pelican</h3>
<p>Next, the pelican software needs to be containerized for Cloud Build to use it.
You can either grab an existing image, such as the one I've shared on Docker
Hub, or build your own.</p>
<p><strong><a href="/docker-pelican-image.html">How to build a Docker image for Pelican</a></strong></p>
<h3 id="push-pelican-image-to-gcr">Push Pelican Image to GCR</h3>
<p>The Pelican image needs to be in the Google Container Registry for Cloud Build
to use it. GCR is like Docker Hub, but private and not free. It's pretty
affordable.</p>
<p><strong><a href="/google-container-registry.html">Using Google Cloud Registry</a></strong></p>
<h2 id="development-environment">Development Environment</h2>
<p>Google Storage's website hosting feature is great, but it has a cache that I
haven't yet figured out how to bypass. This means that if you push a mistake to
the storage, then push a fix, the fix may not display for like an hour.</p>
<p>To get past this, using a local service for development is a nice solution.
Also, this way you're not testing your code in production.</p>
<p>You can see the utility script I made <a href="https://github.com/kylep/kyle.pericak.com/blob/master/bin/pelican-dev.sh">here</a>,
but basically you just run a local Pelican image, then run <code>pelican</code> against
your content files with <code>--listen</code> and <code>--autoreload</code>.</p>
<p><strong><a href="/pelican-dev-environment.html">Pelican Website: Local Development Environment</a></strong></p>
<h2 id="cloud-build">Cloud Build</h2>
<p>Google Cloud Build is the automation framework used to watch for changes
committed to github, then transform the content files into a website and save
them to Google Cloud Storage.</p>
<p>Configure Cloud Build to watch for commits to the master branch of the blog
content repository, then run the pelican and rsync jobs against it.</p>
<p><strong><a href="/google-cloud-build.html">Google Cloud Build: Automatically Deploy Changes from GitHub</a></strong></p>
<h2 id="google-storage-web-hosting">Google Storage Web Hosting</h2>
<p>At this point, you've written your content, converted it into web files, and
synchronized those files with Google Cloud Storage. To make the website
accessible, you need to configure Google Cloud Storage to host the content as
a website. This step is also where you point your custom domain name at the new
site.</p>
<p><strong><a href="/google-cloud-storage-website.html">Google Cloud Storage: Static Website Hosting</a></strong></p>
<h1 id="secure-the-site-with-https">Secure the site with HTTPS</h1>
<p>These days there's really no good excuse to not have HTTPS on your site. On
a self-hosted site I usually use <a href="https://letsencrypt.org/">LetsEncrypt</a>, but
for this static site instead I've used the built-in HTTPS feature of
CloudFlare's free CDN solution.</p>
<p><strong><a href="/cloudflare-https.html">Using Cloudflare to provide HTTPS to an HTTP static site</a></strong></p>


Kyle Pericak

"It works in my environment"

Categories

Tags