<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<p>In this project I'll be trying out each of the Google Cloud Platform's
services. I'll be updating it pretty regularly.</p>
<hr>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes, or will include, the following posts.
If any aren't finished, check back later! I'll also certainly be adding more
as time goes on.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left"></td>
<td align="left"><strong>Continuous Integration</strong></td>
</tr>
<tr>
<td align="left">2019-08-06</td>
<td align="left"><a href="/google-cloud-build.html">Google Cloud Build: Basics</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Database</strong></td>
</tr>
<tr>
<td align="left">2019-10-14</td>
<td align="left"><a href="/gcp-firestore-python.html">Google Cloud Firestore Basics - Python</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Containers</strong></td>
</tr>
<tr>
<td align="left">2019-08-05</td>
<td align="left"><a href="/google-container-registry.html">Google Container Registry: Basics</a></td>
</tr>
<tr>
<td align="left"></td>
<td align="left"><strong>Cloud Storage</strong></td>
</tr>
<tr>
<td align="left">2019-08-09</td>
<td align="left"><a href="/google-cloud-storage-website.html">Hosting websites on Cloud Storage</a></td>
</tr>
</tbody>
</table>


<p><strong>This post is linked to from the <a href="/gcp.html">GCP: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to interact with Google's Cloud Firestore, using examples
written in Python. The concepts should apply to any language.</p>
<p>Firestore is the newer version of Datastore. You can find the differences
<a href="https://cloud.google.com/firestore/docs/firestore-or-datastore">here</a>.
In this post, the  database is configured in Native mode.
Other than some extra language support, Native mode's coolest new feature seems
to be the ability to listen to a post. From what I can tell, this could replace
the need for pub/sub in some scenarios.</p>
<p>Throughout the post I'll use a project named "example". Be sure to replace this
with your own project name.</p>
<hr>
<h1 id="pricing">Pricing</h1>
<p>Official Firestore pricing data is <a href="https://cloud.google.com/firestore/pricing">here</a>.</p>
<p><strong>Note that Firestore mode is almost twice the price of Datastore mode.</strong>
Datastore pricing can be found <a href="https://cloud.google.com/datastore/pricing">here</a>.</p>
<p>Every day a number of free operations are allowed, which for a small
application makes Datastore very affordable. Even after the free tier though,
the pricing is pretty good. Firestore and Datastore have the same free tiers.</p>
<h2 id="summary-of-pricing-datastore-vs-firestore">Summary of pricing: Datastore vs Firestore</h2>
<ul>
<li>Storage: $/GB/Month
<ul>
<li>Datastore: 0.108</li>
<li>Firestore: 0.180</li>
</ul>
</li>
<li>Reads: $/100,000
<ul>
<li>Datastore: 0.036</li>
<li>Firestore: 0.060</li>
</ul>
</li>
<li>Writes: $/100,000
<ul>
<li>Datastore: $0.108</li>
<li>Firestore: $0.180</li>
</ul>
</li>
<li>Deletes: $/100,000
<ul>
<li>Datastore: 0.012</li>
<li>Firestore: 0.020</li>
</ul>
</li>
</ul>
<p>Max entity size and document size are both 1MB, so you could realistically just
use JSON strings in entities instead to save money if you don't need any of
Firestore's other features.</p>
<hr>
<h1 id="datastorefirestore-concepts">Datastore/Firestore Concepts</h1>
<p>Both options use a non-relational/NoSQL database. NoSQL databases in general
allow for the direct serialization of a programs objects/dictionaries.</p>
<h2 id="firestore">Firestore</h2>
<p><a href="https://cloud.google.com/firestore/docs/data-model">Official Firestore data model docs</a></p>
<p>Firestore uses a document database organized into documents and collections.
This is a change from the older Datastore model of Entities organized by Kinds.</p>
<p>Documents support nested data structures ("Maps"). Think of them like JSON
objects, but with more types supported. Documents are limited to 1MB each.</p>
<p>Documents are stored in Collections.
Collections are containers for documents, like a filesystem's directories.
Documents can also contain their own nested collections, which store other
documents. Nesting has a depth limit of 100.</p>
<p>Documents are identified by References. In python, a reference looks like
this:</p>
<pre><code class="language-python">collection('collection1').document('document1')
</code></pre>
<h2 id="datastore">Datastore</h2>
<p>In Datastore, objects are called Entities. The key-value data pairs stored by
entities are called properties.</p>
<p>Entities are uniquely identified by keys.
Entity Keys consists of the following components:</p>
<ul>
<li><em>Namespace</em>: Basically a tenant identifier, like a user ID. By default, the
namespace is an empty string. Any entity created as a "child" will inherit
its parent's namespace.</li>
<li><em>Kind</em>: A string describing the category or purpose of the entity.</li>
<li><em>Identifier</em>: Unlike namespace and kind which are shared between like
entities, the identifier is unique to each entity. This is either a string
called a "key name", or an integer numeric ID.</li>
<li><em>Ancestor Path</em>: An optional component of the key, identifying its parents to
form a database hierarchy.</li>
</ul>
<hr>
<h1 id="create-iam-service-account">Create IAM Service Account</h1>
<p>Before your code can access Datastore/Firestore, access needs to be configured
in the <a href="https://console.cloud.google.com/iam-admin">Identity Access Management</a>
tool.</p>
<p>First, create a <a href="https://console.cloud.google.com/iam-admin/serviceaccounts">Service Account</a>:</p>
<ol>
<li>Open the Service Account page</li>
<li>Select your project</li>
<li>Click "Create Service Account"</li>
<li>Service Account Details:
<ol>
<li>Service account name: <code>PythonDatastoreAdmin</code></li>
<li>Service account ID: <code>pythondatastoreadmin</code></li>
<li>Service account description: SA with Full Datastore Access for Python</li>
</ol>
</li>
<li>Grant this service account access to project (optional)
<ol>
<li>Select a role: Datastore > Cloud Datastore Owner</li>
<li>Continue</li>
</ol>
</li>
<li>Grant users access to this service account (optional)
<ol>
<li>Download a key file for Python to use. Use JSON as the key type.</li>
<li>Done</li>
</ol>
</li>
</ol>
<p>I downloaded my key to <code>$HOME/gcp_keys/datastore_admin.json</code></p>
<hr>
<h1 id="python-environment-setup">Python Environment Setup</h1>
<p>Build a virtualenv. If you're in a git repo, don't forget to ignore this
directory.</p>
<pre><code class="language-bash">python3 -m venv env3
source env3/bin/activate
</code></pre>
<p>Write the requirement file</p>
<p><code>vi requirements.txt</code></p>
<pre><code class="language-text">google-cloud-firestore
</code></pre>
<p>Install the requirements</p>
<pre><code class="language-bash">pip install -r rqeuirements.txt
</code></pre>
<hr>
<h1 id="python-code-examples">Python Code Examples</h1>
<p>A fake project name of "example" will be used. Be sure to use a valid one.</p>
<p>Some notes about the Firestore python API:</p>
<ul>
<li><code>firestore.client().collection</code> returns a <a href="https://googleapis.dev/python/firestore/latest/collection.html">CollectionReference</a>,
which allows you to operate on the named collection.</li>
<li><code>firestore.client().collection().list_documents</code> returns a <a href="google.api_core.page_iterator.GRPCIterator">Page Iterator</a>.
Casting a page iterator to a list will trigger a read operation, returning a
list of <code>DocumentReference</code> objects.</li>
<li><code>firestore.client().collection().document</code> returns
a <a href="https://googleapis.dev/python/firestore/latest/document.html">DocumentReferece</a>,
which interacts with the actual Firestore documents. The <code>DocumentReference</code>
object can operate (get/set/create) on the actual Document.</li>
<li><code>firestore.client().collection().document().get()</code> will return
a <a href="https://developers.google.com/android/reference/com/google/firebase/firestore/DocumentSnapshot">DocumentSnapshot</a>
of the document. The document snapshot is how you read data from a document.
Document snapshots are a representation of the document state taken at the
instant of their creation.</li>
<li><code>firestore.client().collection().document().create()</code> makes a new document.
If the object already  existed, a <code>google.api_core.exceptions.AlreadyExists</code>
exception is thrown.</li>
<li>Write operations (create &#x26; set) return a <a href="https://cloud.google.com/firestore/docs/reference/rest/v1/WriteResult">WriteResult</a>.</li>
</ul>
<pre><code class="language-python">from os import environ
from google.cloud import firestore

# The key file is an exported JSON file from the IAM tool
key_file_path = '{}/gcp_keys/datastore_admin.json'.format(environ['HOME'])

# When running this on a local system, the cloud.google package from GCP
# expects the GOOGLE_APPLICATION_CREDENTIALS environment variable to be set
# to the exported IAM key file's path
environ['GOOGLE_APPLICATION_CREDENTIALS'] = key_file_path

# google.cloud.datastore.Client expects the project name as an argument
project = 'example'

# Instantiate the datastore client. This step authenticates to GCP.
# See the Credential Error note below if authentication fails
client = firestore.Client(project)

# Get a reference to the test collection. This returns a
# google.cloud.firestore_v1.collection.CollectionReference object
ref_col_test = client.collection('test')

# Get a reference to a document which may or may not exist. This returns a
# google.cloud.firestore_v1.document.DocumentReference object
ref_doc_document1 = ref_col_test.document('document1')

# Perform the actual database query to get the document. get() returns a
# google.cloud.firestore_v1.document.DocumentSnapshot object.
snapshot_document1 = ref_doc_document1.get()

# Note that there is no data
assert document1.to_dict() is None

# You can use document.exists to check if the document already exists.
# Here's an example of doing a safe insert of some sample data:
sample_data = {
    'nums': {'num1': 100, 'num2': 200},
    'strs': {'animal1': 'dog', 'animal2': 'cat'}}
if snapshot_document1.exists is False:
    createe_write_result = ref_doc_document1.create(sample_data)

document1_data = ref_doc_document1.get().to_dict()

# Finally, update the data
sample_data['strs']['animal3'] = 'bird'
set_write_result = ref_doc_document1.set(sample_data)

# List the documents in the collection. The iterator returns a paginated list
# of document references. You can then operate on those references.
doc_list_iterator = ref_col_test.list_documents()
doc_ref_list = list(doc_list_iterator)
sample_document_snapshot = doc_ref_list[0].get()

# Show that the changes were applied to the data. This will return 'bird'
animal3 = sample_document_snapshot.to_dict()['strs']['animal3']
</code></pre>
<h2 id="troubleshooting">Troubleshooting</h2>
<h3 id="credentials-error">Credentials Error</h3>
<p>If your local google cloud authentication isn't set up correctly, you'll get an
error saying:</p>
<pre><code class="language-text">google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
</code></pre>
<p>If that happens, review the IAM Service account steps.</p>
<h3 id="iterator-has-already-started">Iterator has already started</h3>
<p>If you try and cast a <code>GRPCIterator</code> to list twice, you get a <code>ValueError</code>.</p>
<pre><code class="language-text">ValueError: ('Iterator has already started', &#x3C;google.api_core.page_iterator.GRPCIterator object)
</code></pre>
<p>The <code>ValueError</code> is thrown because <code>._started</code> is <code>True</code> in the iterator. It has
already done its thing and won't work again. I didn't see any good way to
restart the iterator without discarding it.
You can see the details in the <a href="https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/page_iterator.py">page iterator source code</a>.</p>
<p>Best course is just to get a new iterator from the reference object.
See in the following example how the ID changes. The <code>doc_list_iterator</code> var is
being pointed at a new iterator. The old iterator gets gc'd by Python as usual.</p>
<pre><code>>>> id(doc_list_iterator)
4422123408
>>> doc_list_iterator = ref_col_test.list_documents()
>>> id(doc_list_iterator)
4422123472
</code></pre>
<p>It will work as expected.</p>


<p>This post covers getting started with a very simple Google Cloud Functions API,
a great serverless solution with affordable pricing. This API will pull the
IP address from the HTTP request, and return <code>Hello &#x3C;source IP></code>.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#pricing-details">Pricing Details</a></li>
<li><a href="#deploy-hello-world-function">Deploy Hello World Function</a>
<ul>
<li><a href="#mainpy">main.py</a></li>
<li><a href="#requirementspy">requirements.py</a></li>
</ul>
</li>
<li><a href="#accessing-the-function">Accessing the Function</a>
<ul>
<li><a href="#from-the-console-web-ui">From the Console Web UI</a></li>
<li><a href="#from-the-request-url">From the Request URL</a></li>
</ul>
</li>
</ul>
<h1 id="pricing-details">Pricing Details</h1>
<p>Cloud Functions pricing is pretty affordable for most use cases, assuming you
write your functions in such a way that they don't get invoked a few too many
millions of times.</p>
<p>Check the official <a href="https://cloud.google.com/functions/pricing">Google docs</a>
for accurate and up-to-date pricing, but here's what it looked like at the time
of my writing this.</p>
<p>Pricing is broken into a few fees:</p>
<ul>
<li><em>Invocations</em>: A flat fee per function execution. $0.0000004 per invocation,
excluding the first 2 million.</li>
<li><em>Compute Time</em>: At the lowest clock speed (200MHz x 128MB RAM), compute costs
$0.000000231/100ms, ceil 100.</li>
<li><em>Networking</em>: $0.12/GB egress for data, while ingress is free.</li>
</ul>
<hr>
<h1 id="deploy-hello-world-function">Deploy Hello World Function</h1>
<p>Open <a href="https://console.cloud.google.com/functions/">Google Cloud Functions</a> in
your browser.</p>
<p>Create a function.</p>
<ul>
<li>Name: HelloWorld</li>
<li>Memory Allocated: 128 MB</li>
<li>Trigger: HTTP</li>
<li>Authentication: Allow unauthenticated invocations</li>
<li>Source code: Inline Editor</li>
<li>Runtime: Python 3.7</li>
<li>Other settings: Leave them as defaults</li>
</ul>
<h2 id="mainpy">main.py</h2>
<p>The API function defined by <code>Function to execute</code> accepts a
<a href="https://flask.palletsprojects.com/en/1.1.x/api/#incoming-request-data">flask.Request</a>
argument.</p>
<pre><code class="language-python">def hello_world(request):
    """" Return a Hello message to sender of the API call """
		return 'Hello {}!'.format(request.remote_addr)
</code></pre>
<h2 id="requirementspy">requirements.py</h2>
<p>You can leave this blank. It's a list of libraries that will be used.</p>
<hr>
<h1 id="accessing-the-function">Accessing the Function</h1>
<h2 id="from-the-console-web-ui">From the Console Web UI</h2>
<ol>
<li>Navigate to the <a href="https://console.cloud.google.com/functions">Functions page</a>.</li>
<li>Click on the 3 dots next to the function name, and choose "Test function".</li>
<li>Populate and JSON for the GET request in "Triggering event". The above
example doesn't accept any input data so <code>{}</code> is all you need.</li>
<li>Click "Test the function"</li>
</ol>
<h2 id="from-the-request-url">From the Request URL</h2>
<ol>
<li>Navigate to the <a href="https://console.cloud.google.com/functions">Functions page</a>.</li>
<li>Click the function's name to navigate to the "Function details" page</li>
<li>Go to the 'Trigger" tab</li>
<li>Find the URL. Copy it.</li>
</ol>
<p>You can now test the URL with curl, your browser, httpie, python, etc.</p>
<pre><code class="language-bash">https://&#x3C;region>-&#x3C;project>.cloudfunctions.net/HelloWorld
</code></pre>


<p>Google Cloud Storage has a feature where storage buckets can be used as a cost
effective web server. It can only host static sites, but that works just fine
for anything with all the logic on the client side.</p>
<p>Hosting a static website using Cloud Storage is <strong>extremely affordable</strong>.</p>
<p>This site is using this approach for hosting.</p>
<p><a href="https://cloud.google.com/storage/docs/hosting-static-website">The official documentation can be found here</a>.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#pricing">Pricing</a></li>
<li><a href="#add-the-domain-to-google-search-console">Add the Domain to Google Search Console</a></li>
<li><a href="#link-the-domain-name-to-your-storage-bucket">Link the Domain Name to Your Storage Bucket</a>
<ul>
<li><a href="#update-dns-cname-record">Update DNS CNAME Record</a></li>
<li><a href="#create-a-storage-bucket-with-matching-name">Create a Storage Bucket with Matching Name</a></li>
<li><a href="#configure-storage-bucket-for-access-control--index-file">Configure Storage Bucket for Access Control &#x26; Index File</a>
<ul>
<li><a href="#update-cloud-storage-bucket-permissions">Update Cloud Storage Bucket Permissions</a></li>
<li><a href="#configure-indexhtml-as-the-mainpagesuffix">Configure index.html as the MainPageSuffix</a></li>
<li><a href="#set-the-default-acl-for-the-bucket">Set the default ACL for the bucket</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#push-site-content-to-cloud-storage">Push Site Content to Cloud Storage</a></li>
</ul>
<h1 id="pricing">Pricing</h1>
<p><a href="https://cloud.google.com/storage/pricing">Find the latest pricing data from Google here</a>.</p>
<p>Google doesn't seem to actually charge for the static website hosting feature
of Cloud Storage. Instead, it just bills you for the storage usage. Notable is
that the operation usage cost will be far higher, and the data storage cost far
lower than in a typical Cloud Storage use case.</p>
<p>These prices are rough and there are lots of edge cases:</p>
<ul>
<li><strong>Data Storage</strong>: Free for the first 5GB, which will cover basically any
website. After that it's around  $0.026 US / GB / Month.</li>
<li><strong>Network Egress</strong>: Around $0.12/ GB</li>
<li><strong>Operations</strong>: This one will be higher. Every file served is an operation.
This includes the .html files, each image, the .css files, and so on. A CDN
such as <a href="https://dash.cloudflare.com/">Cloudflare</a> can help reduce the number
of read operations. After the first 50,000 free reads, Cloud Storage will
charge $0.004 / 10,000 operations.</li>
</ul>
<p>As you can see, for anything but hugely popular sites this will usually work
out to <em>almost free</em>.</p>
<hr>
<h1 id="add-the-domain-to-google-search-console">Add the Domain to Google Search Console</h1>
<p>A DNS TXT record containing a special string needs to be assigned to the domain
name that will be applied to the storage bucket. This is used to confirm
ownership of the domain.</p>
<p><strong>To add the domain to the search console:</strong></p>
<ol>
<li>Open the <a href="https://search.google.com/search-console/welcome">Google Search Console</a></li>
<li>Enter your domain name</li>
<li>Copy the TXT record they gave you and apply it to your domain. In Cloudflare
you go to your site, click DNS, Add Record, TXT. For kyle.pericak.com I used
<code>kyle</code> as the "Name" and the string from Google as the "Content". TTL Auto.</li>
<li>Go back to the search console page and verify</li>
<li>Click the link to go to your domain</li>
</ol>
<p>The domain name can now be bound to a Cloud Storage bucket.</p>
<p><strong>Note:</strong> I had a really hard time doing this with GoDaddy. The TXT change
would not propagate even after over 24 hours. I transferred my domain to
<a href="https://cloudflare.com/">Cloudflare</a>. Their updates apply basically right
away!</p>
<p>See also: <a href="/dns-xfer-godaddy-cloudflare.html">How I transferred the domain from GoDaddy to Cloudflare</a>.</p>
<hr>
<h1 id="link-the-domain-name-to-your-storage-bucket">Link the Domain Name to Your Storage Bucket</h1>
<h2 id="update-dns-cname-record">Update DNS CNAME Record</h2>
<p>Update the subdomain CNAME record to point to Google's API URL.
In Cloudflare I made a CNAME entry for <code>kyle.pericak.com</code> that looked like
this:</p>
<ul>
<li>Name: <code>kyle</code></li>
<li>Content: <code>c.storage.googleapis.com</code></li>
</ul>
<h2 id="create-a-storage-bucket-with-matching-name">Create a Storage Bucket with Matching Name</h2>
<p>Go to the <a href="https://console.cloud.google.com/storage/browser">Google Cloud Storage Browser</a>.
Click Create Bucket up at the top.</p>
<ol>
<li>Enter your FQDN in the bucket name. For example, I used <code>kyle.pericak.com</code></li>
<li>Location Type: Region</li>
<li>Location: northamerica-northeast1</li>
<li>Storage Class: standard</li>
<li>Access Control: Per-object (not bucket-only)</li>
<li>Encryption: Google managed keys</li>
<li>Create</li>
</ol>
<p>The storage bucket now exists and is ready to host the static content.</p>
<h2 id="configure-storage-bucket-for-access-control--index-file">Configure Storage Bucket for Access Control &#x26; Index File</h2>
<h3 id="update-cloud-storage-bucket-permissions">Update Cloud Storage Bucket Permissions</h3>
<p>Permit everyone to read the website files from Cloud Storage:</p>
<ol>
<li>From the sidebar go Storage</li>
<li>Click the 3 dots to the right of this storage bucket</li>
<li>Edit Bucket Permissions</li>
<li>New Members: <code>allUsers</code></li>
<li>Add the role <code>Storage Object Viewer</code></li>
</ol>
<h3 id="configure-indexhtml-as-the-mainpagesuffix">Configure index.html as the MainPageSuffix</h3>
<p>Cloud Storage doesn't know where the page root is by default. Define it:</p>
<ol>
<li>Click on the 3 dots to the right of the bucket name</li>
<li>Click Edit Website Configuration</li>
<li>Main page: index.html</li>
</ol>
<h3 id="set-the-default-acl-for-the-bucket">Set the default ACL for the bucket</h3>
<p>If this isn't done then the website will show XML with an error stating
<code>Anonymous caller does not have storage.objects.list access</code>.</p>
<p><strong>Option 1: Using gsutil:</strong></p>
<pre><code class="language-bash">gsutil defacl ch -u AllUsers:R gs://kyle.pericak.com
</code></pre>
<p><strong>Option 2: From the web UI:</strong></p>
<ol>
<li>Open <a href="https://console.cloud.google.com/storage/">console.cloud.google.com/storage/</a>.</li>
<li>Go to the bucket named after the domain</li>
<li>Permissions</li>
<li>The permissions can be modified here.</li>
</ol>
<hr>
<h1 id="push-site-content-to-cloud-storage">Push Site Content to Cloud Storage</h1>
<p>While its possible to manually upload each file using the web UI, it would be
super tedious.</p>
<p>Run the <code>gsutil rsync</code> module to upload files at all once.
Note that <code>$dst_url</code> is pointing at my domain-associated cloud storage bucket.</p>
<pre><code class="language-bash"># -r    recurse - sync directories
# -c    compare checksums instead of mtime
# -d    Delete extra files under dst_url not found under src_url
src_url="./output"
dst_url="gs://kyle.pericak.com"
gsutil -m rsync -r -c -d $src_url $dst_url
</code></pre>


<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#pricing">Pricing</a></li>
<li><a href="#write-the-cloud-build-file">Write the Cloud Build file</a></li>
<li><a href="#iam-roles">IAM Roles</a>
<ul>
<li><a href="#docker-image-to-cloud-registry">Docker Image to Cloud Registry</a></li>
<li><a href="#static-website-to-google-cloud-storage">Static Website to Google Cloud Storage</a></li>
</ul>
</li>
<li><a href="#define-cloud-build-triggers">Define Cloud Build Triggers</a></li>
</ul>
<h1 id="pricing">Pricing</h1>
<p>Current pricing data can be found <a href="https://cloud.google.com/cloud-build/pricing">here</a>.</p>
<ul>
<li>For the first 120 builds in a day, cloud build is <strong>free</strong>.</li>
<li>each build after the free quota costs $0.003 US / build-minute when using the
single core build server with no additional SSD storage</li>
</ul>
<h1 id="write-the-cloud-build-file">Write the Cloud Build file</h1>
<p>Google Cloud Build uses a file, <code>cloudbuild.yml</code>, to define the tasks that will
be executed when a build is triggered.</p>
<p>The online references for the syntax are pretty good:</p>
<ul>
<li><a href="https://cloud.google.com/cloud-build/docs/build-config">Google's Cloud-Build Documentation</a></li>
<li><a href="https://cloud.google.com/cloud-build/docs/configuring-builds/substitute-variable-values">Variables Available in cloudbuild.yaml</a></li>
</ul>
<p>Below are some example configuration files used for this site.</p>
<h1 id="iam-roles">IAM Roles</h1>
<p><a href="https://cloud.google.com/cloud-build/docs/securing-builds/configure-access-control">Official Documentation</a></p>
<ul>
<li>Cloud Build Editor</li>
<li>Cloud Build Viewer</li>
</ul>
<h2 id="docker-image-to-cloud-registry">Docker Image to Cloud Registry</h2>
<p>Cloud Build is great for keeping your Docker Registry images aligned with your
version control code.</p>
<p>In this example I use Cloud Build to build and push a <a href="/docker-pelican-image.html">pelican image</a>
to the <a href="/google-container-registry.html">Google Container Registry</a>.</p>
<p>Be sure to keep the Dockerfile code in version control. Aside from being a good
idea in general, Cloud Build can also use Git commits as a build trigger.
<a href="https://github.com/kylep/pelican">Here's my Pelican Docker image repository</a>.</p>
<pre><code class="language-yaml">steps:
- name: 'gcr.io/cloud-builders/docker'
  args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/pelican', '.' ]
images:
- 'gcr.io/$PROJECT_ID/pelican'
</code></pre>
<h2 id="static-website-to-google-cloud-storage">Static Website to Google Cloud Storage</h2>
<p>Another use for Cloud Builder is to generate static website files and then sync
them up with Cloud Storage every time the source control files or template are
changed.</p>
<p>Here's an example that triggers when I commit code to my
<a href="https://github.com/kylep/kyle.pericak.com">blog content git repository</a>. It
copies the content into <code>/workspace</code> and uses the above pelican image to render
the content and output it to the <code>output/</code> directory.</p>
<p>Once the site content is generated, the next step uses the <code>gcloud</code> image's
<code>gsutil -m rsync</code> command to upload the files to the Google Storage Bucket.</p>
<hr>
<h1 id="define-cloud-build-triggers">Define Cloud Build Triggers</h1>
<p>Here's the procedure I used to link a Cloud Build trigger to my Pelican image.</p>
<ol>
<li>Go to <a href="https://console.cloud.google.com/cloud-build">Google Cloud-Build</a>.</li>
<li>On the left side-bar, navigate to Triggers</li>
<li>Add Trigger</li>
<li>Select source: GitHub</li>
<li>Authenticate: Log in if needed</li>
<li>Select Repository: This is the repo that will be watched for changes</li>
<li>Name: This is the name of the trigger</li>
<li>Description: Something like <code>Push to master branch</code></li>
<li>Trigger Type: <code>Branch</code></li>
<li>Branch: <code>master</code></li>
<li>Build Configuration: <code>cloudbuild.yaml</code></li>
<li><strong>Create Trigger</strong></li>
</ol>
<p>With this trigger in place, all pushes to the pelican project will trigger the
rebuilding of its image, followed by updating the image in GCR with the latest
changes.</p>


<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#why-use-google-container-registry">Why use Google Container Registry</a></li>
<li><a href="#pricing">Pricing</a></li>
<li><a href="#command-line">Command Line</a></li>
<li><a href="#authentication">Authentication</a>
<ul>
<li><a href="#create-a-service-account">Create a service account</a></li>
<li><a href="#authenticate-your-local-docker-client">Authenticate your local Docker client</a></li>
</ul>
</li>
<li><a href="#using-the-google-container-registry">Using the Google Container Registry</a>
<ul>
<li><a href="#pull-image">Pull Image</a></li>
<li><a href="#push-image">Push Image</a>
<ul>
<li><a href="#access-denied">Access Denied</a></li>
</ul>
</li>
<li><a href="#delete-an-image">Delete an Image</a></li>
</ul>
</li>
</ul>
<h1 id="why-use-google-container-registry">Why use Google Container Registry</h1>
<p>The Google Container Registry is a SaaS version of the public <a href="https://hub.docker.com/_/registry">Registry</a>
docker image. It solves a number of problems that the public Docker image has.
Namely:</p>
<ul>
<li><strong>Authentication</strong>: GCR has built-in access controls, where the Registry image
does not. With the Docker Registry you need to put something like NGINX in
front of it to provide access control.</li>
<li><strong>HTTPS</strong>: Like with the authentication problem, Registry needs a reverse proxy
in front of it to serve a certificate. Also, you need to worry about
obtaining and maintaining the valid cert files. Without a valid cert, each
Docker client needs to list the registry as an insecure registry, which is
both insecure and inconvenient.</li>
<li><strong>Serverless</strong>: To host a traditional Docker registry, you need a public
server running Docker. With Google's Container Registry, there's no server to
maintain.</li>
<li><strong>Vulnerability Scanning</strong>: GCR can <a href="https://cloud.google.com/container-registry/docs/get-image-vulnerabilities">scan your images for security issues</a>.</li>
</ul>
<hr>
<h1 id="pricing">Pricing</h1>
<p>The pricing is well documented <a href="https://cloud.google.com/container-registry/pricing">here</a>,
and it has a lot of variables.</p>
<p>Basically though, it boils down to the following approximate costs:</p>
<ul>
<li><strong>Storage</strong>: $0.026 USD / GB / month</li>
<li><strong>Network</strong>: ~ $0.12 USD / GB</li>
<li><strong>Vulnerability Scanning</strong>: $0.26 per scanned image</li>
</ul>
<hr>
<h1 id="command-line">Command Line</h1>
<p>GCR can be managed from the web UI or the command line. I try and keep my steps
limited to the web UI, but the command line is really easy to use. You can find
its documentation <a href="https://cloud.google.com/sdk/gcloud/reference/container/images">here</a>.</p>
<hr>
<h1 id="authentication">Authentication</h1>
<p>Before you start, be sure to install:</p>
<ul>
<li><a href="https://cloud.google.com/sdk/">Google Cloud SDK</a></li>
<li><a href="https://docs.docker.com/install/">Docker</a></li>
</ul>
<p>To interact with Google Container Registry you need to first authenticate your
local Docker client. Usually this is done with the <code>docker login &#x3C;registry></code>
command, but that's not how it's done with Google container registry. Instead
we'll use the <code>gcloud</code> command line.</p>
<h2 id="create-a-service-account">Create a service account</h2>
<p>While this isn't strictly necessary, it's good practice.</p>
<p>The container registry uses GCS buckets to store the images. It doesn't have
dedicated roles, and instead uses the GCS roles. You can find a detailed list
of the roles <a href="https://cloud.google.com/container-registry/docs/access-control">here</a>.
The two important roles are <code>roles/storage.admin</code> for write and
<code>roles/storage.objectViewer</code> for read.</p>
<p>To create the SA from the Google Cloud Web UI:</p>
<ol>
<li>Open the <a href="https://console.cloud.google.com/iam-admin/serviceaccounts">IAM Service Accounts page</a></li>
<li>If needed, select or create your project</li>
<li>Click <code>Create Service Account</code></li>
<li>Create a member, such as <code>gcr-readonly</code> or <code>gcr-admin</code>.
When possible, its nice to have the account name and account ID match.</li>
<li>Assign a useful description to the service-account.</li>
<li>Assign a role. <code>Storage Object Viewer</code> grants read access and
<code>Storage Admin</code>.</li>
<li>Create and download a key. Name it something like <code>gcr-readonly.json</code>.</li>
</ol>
<h2 id="authenticate-your-local-docker-client">Authenticate your local Docker client</h2>
<p>Use the <code>gcloud</code> tool to deploy some <code>credHelpers</code> to <code>~/.docker/config.json</code>.</p>
<pre><code class="language-bash">gcloud auth configure-docker --project &#x3C;project name>
</code></pre>
<p>Next up, it's time to authenticate.</p>
<p>You could run <code>gcloud auth login</code>, but that doesn't use the service account we
just created and also requires a browser to open a generated link.</p>
<p>Instead, use the service account's JSON file generated in the above step with
the <code>--key-file</code> argument as follows:</p>
<pre><code class="language-bash">gcloud auth activate-service-account --key-file /vagrant/auth/gcr-readonly.json
</code></pre>
<p>If successful, you'll get a confirmation message:</p>
<pre><code class="language-bash">Activated service account credentials for: [gcr-readonly@*.gserviceaccount.com]
</code></pre>
<hr>
<h1 id="using-the-google-container-registry">Using the Google Container Registry</h1>
<p>The GCR works just like any other registry once you've ran the <code>gcloud auth</code>
command. You don't need to run any <code>docker login</code> command.</p>
<h2 id="pull-image">Pull Image</h2>
<p>Here's how to pull a Pelican image from GCR:</p>
<pre><code class="language-bash">docker pull gcr.io/myProject/pelican
</code></pre>
<h2 id="push-image">Push Image</h2>
<p>It's pretty straightforward. Replace <code>myProject</code> with your project name.</p>
<p>In this example I add a tag to my pelican image, calling it <code>pelican2</code>, and
push it up to the registry as a new image. You'd normally make your own Docker
image.</p>
<pre><code class="language-bash">sudo docker tag gcr.io/myProject/pelican gcr.io/myProject/pelican2
sudo docker push gcr.io/myProject/pelican2
</code></pre>
<p>If you didn't get an error, then the image is now pushed.</p>
<h3 id="access-denied">Access Denied</h3>
<p>If you receive the following message, your user account lacks write access:</p>
<pre><code class="language-text">denied: Token exchange failed for project 'myProject'. Caller does not have permission 'storage.buckets.get'.
</code></pre>
<p>To resolve this, create a service account  with the role <code>roles/storage.admin</code>.
If there's a finer grained access level to permit this, I'd be interested to
hear about it.</p>
<pre><code class="language-bash">gcloud auth activate-service-account --key-file /vagrant/auth/gcr-admin.json
sudo docker push gcr.io/kylepericak/pelican2
</code></pre>
<h2 id="delete-an-image">Delete an Image</h2>
<p>At the time of writing this, there's a warning in the cloud UI saying that you
can't delete images unless you use the CLI.</p>
<p>Here's how to do that. Be sure to replace <code>myProject</code> with your project.</p>
<pre><code class="language-bash"># List current images
gcloud config set project myProject

# Ignore this warning if you're using a service account:
# WARNING: You do not appear to have access to project [myProject] or it does not exist.

# Delete the image
cloud container images delete gcr.io/myProject/pelican2
</code></pre>


<p>The objective of this project is to build a technology blog website,
satisfying the following requirements:</p>
<ul>
<li><strong>Affordable</strong>: The website should cost as little as possible to maintain</li>
<li><strong>Maintainable</strong>: Authoring and updating content on the website should be
simple and fast.</li>
<li><strong>Extensible</strong>: This is a software developer's website. It needs to enable
custom code so fun features can be built into it.</li>
<li><strong>Custom Domain</strong>: Some free/cheap blog options require a custom FQDN suffix,
such as <code>x.wordpress.com</code> sites. This blog needs to work with any domain
name.</li>
<li><strong>Fun</strong>: The blog needs to use technologies that are current and interesting.</li>
</ul>
<hr>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes, or will include, the following posts.
If any aren't finished, check back later!</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">2019-08-01</td>
<td align="left"><a href="/docker-pelican-image.html">Building a Pelican Docker image</a></td>
</tr>
<tr>
<td align="left">2019-08-03</td>
<td align="left"><a href="/writing-pelican-content.html">Writing blog content for Pelican</a></td>
</tr>
<tr>
<td align="left">2019-08-04</td>
<td align="left"><a href="/pelican-dev-environment.html">A simple Pelican development environment</a></td>
</tr>
<tr>
<td align="left">2019-08-05</td>
<td align="left"><a href="/google-container-registry.html">Saving Docker images in Google's Container Registry</a></td>
</tr>
<tr>
<td align="left">2019-08-06</td>
<td align="left"><a href="/google-cloud-build.html">Google Cloud Build with GitHub Trigger</a></td>
</tr>
<tr>
<td align="left">2019-08-09</td>
<td align="left"><a href="/google-cloud-storage-website.html">Hosting a website using Google Cloud Storage</a></td>
</tr>
<tr>
<td align="left">2019-12-02</td>
<td align="left"><a href="/cloudflare-https.html">Using Cloudflare to provide HTTPS</a></td>
</tr>
</tbody>
</table>
<hr>
<h1 id="design--workflow">Design &#x26; Workflow</h1>
<p>At a high level, with this setup you just write your content and push it to
git. The rest takes care of itself.</p>
<p>Here's how it works:</p>
<ol>
<li>Blog content is written as text in markdown files. I use Vim.</li>
<li>When the content is ready, code is pushed to a GitHub repository.</li>
<li>A Google Cloud Build trigger watches for commits. When one happens, it loads
the Git master branch to its temporary workspace.</li>
<li>Cloud Build loads a pre-created Pelican Docker image is from Google's
Container Registry, and executes it against the markdown files to generate
a static website's .html and .css files.</li>
<li>Cloud Build uses an rsync image to transfer the files to a Google Storage
bucket.</li>
<li>Google Storage hosts the static site files on a custom domain as a website.</li>
</ol>
<p><img src="/images/blog_workflow.png" alt="Blog architecture diagram"></p>
<hr>
<h1 id="setup-configuring-pelican-and-gcp">Setup: Configuring Pelican and GCP</h1>
<h2 id="pelican-theme">Pelican Theme</h2>
<p>Before Pelican can render the markdown files into a site worth looking at,
it needs a theme to work with.</p>
<p>The easiest way to go is to <a href="http://www.pelicanthemes.com/">select a pre-made theme.</a></p>
<p>Personally, I forked a theme then heavily modified it.
Forking the theme prevents it from changing, and lets you change things as
needed. I'd suggest creating a fork of the theme even if you won't change it.</p>
<h2 id="pelican-docker-image">Pelican Docker Image</h2>
<h3 id="containerize-pelican">Containerize Pelican</h3>
<p>Next, the pelican software needs to be containerized for Cloud Build to use it.
You can either grab an existing image, such as the one I've shared on Docker
Hub, or build your own.</p>
<p><strong><a href="/docker-pelican-image.html">How to build a Docker image for Pelican</a></strong></p>
<h3 id="push-pelican-image-to-gcr">Push Pelican Image to GCR</h3>
<p>The Pelican image needs to be in the Google Container Registry for Cloud Build
to use it. GCR is like Docker Hub, but private and not free. It's pretty
affordable.</p>
<p><strong><a href="/google-container-registry.html">Using Google Cloud Registry</a></strong></p>
<h2 id="development-environment">Development Environment</h2>
<p>Google Storage's website hosting feature is great, but it has a cache that I
haven't yet figured out how to bypass. This means that if you push a mistake to
the storage, then push a fix, the fix may not display for like an hour.</p>
<p>To get past this, using a local service for development is a nice solution.
Also, this way you're not testing your code in production.</p>
<p>You can see the utility script I made <a href="https://github.com/kylep/kyle.pericak.com/blob/master/bin/pelican-dev.sh">here</a>,
but basically you just run a local Pelican image, then run <code>pelican</code> against
your content files with <code>--listen</code> and <code>--autoreload</code>.</p>
<p><strong><a href="/pelican-dev-environment.html">Pelican Website: Local Development Environment</a></strong></p>
<h2 id="cloud-build">Cloud Build</h2>
<p>Google Cloud Build is the automation framework used to watch for changes
committed to github, then transform the content files into a website and save
them to Google Cloud Storage.</p>
<p>Configure Cloud Build to watch for commits to the master branch of the blog
content repository, then run the pelican and rsync jobs against it.</p>
<p><strong><a href="/google-cloud-build.html">Google Cloud Build: Automatically Deploy Changes from GitHub</a></strong></p>
<h2 id="google-storage-web-hosting">Google Storage Web Hosting</h2>
<p>At this point, you've written your content, converted it into web files, and
synchronized those files with Google Cloud Storage. To make the website
accessible, you need to configure Google Cloud Storage to host the content as
a website. This step is also where you point your custom domain name at the new
site.</p>
<p><strong><a href="/google-cloud-storage-website.html">Google Cloud Storage: Static Website Hosting</a></strong></p>
<h1 id="secure-the-site-with-https">Secure the site with HTTPS</h1>
<p>These days there's really no good excuse to not have HTTPS on your site. On
a self-hosted site I usually use <a href="https://letsencrypt.org/">LetsEncrypt</a>, but
for this static site instead I've used the built-in HTTPS feature of
CloudFlare's free CDN solution.</p>
<p><strong><a href="/cloudflare-https.html">Using Cloudflare to provide HTTPS to an HTTP static site</a></strong></p>


Kyle Pericak

"It works in my environment"

Categories

Tags