<p>As of Vim7, spell-checking is a built-in feature.</p>
<p>The spell checker will highlight incorrectly spelled words in red.
Light blue highlights indicate words which are only found in another dictionary
(such as "color" from <code>en_us</code>).</p>
<hr>
<h1>Commands</h1>
<h2>Enable/Disable</h2>
<p>Enable spellcheck</p>
<ul>
<li><code>set</code> for all buffers, <code>setlocal</code> for the current one</li>
<li><code>en_ca</code> will use Canadian spelling.</li>
</ul>
<pre><code class="language-text">:set spell spelllang=en_ca
:setlocal spell spelllang=en_ca
</code></pre>
<p>Or if you're not picky, the <code>spelllang</code> argument can be omitted. I'm not sure
what the default is, but I figure it's probable <code>en</code> which uses all the English
dictionaries together.</p>
<pre><code>:set spell
</code></pre>
<p>Disable spellcheck:</p>
<pre><code class="language-text">:set nospell
</code></pre>
<h2>Using Spell-Check</h2>
<p>Move to the next misspelled word</p>
<pre><code>]s
</code></pre>
<p>previous misspelled word</p>
<pre><code>[s
</code></pre>
<p>Highlight a misspelled word. Show possible replacements.
Press return with no input to go back.</p>
<pre><code>z=
</code></pre>
<p>Accept the spelling of a word:</p>
<pre><code class="language-text">zg
</code></pre>
<p>Mark a word's spelling as wrong</p>
<pre><code class="language-text">zw
</code></pre>


<p><strong>This post is linked to from the <a href="/gcp.html">GCP: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to interact with Google's Cloud Firestore, using examples
written in Python. The concepts should apply to any language.</p>
<p>Firestore is the newer version of Datastore. You can find the differences
<a href="https://cloud.google.com/firestore/docs/firestore-or-datastore">here</a>.
In this post, the  database is configured in Native mode.
Other than some extra language support, Native mode's coolest new feature seems
to be the ability to listen to a post. From what I can tell, this could replace
the need for pub/sub in some scenarios.</p>
<p>Throughout the post I'll use a project named "example". Be sure to replace this
with your own project name.</p>
<hr>
<h1>Pricing</h1>
<p>Official Firestore pricing data is <a href="https://cloud.google.com/firestore/pricing">here</a>.</p>
<p><strong>Note that Firestore mode is almost twice the price of Datastore mode.</strong>
Datastore pricing can be found <a href="https://cloud.google.com/datastore/pricing">here</a>.</p>
<p>Every day a number of free operations are allowed, which for a small
application makes Datastore very affordable. Even after the free tier though,
the pricing is pretty good. Firestore and Datastore have the same free tiers.</p>
<h2>Summary of pricing: Datastore vs Firestore</h2>
<ul>
<li>Storage: $/GB/Month
<ul>
<li>Datastore: 0.108</li>
<li>Firestore: 0.180</li>
</ul>
</li>
<li>Reads: $/100,000
<ul>
<li>Datastore: 0.036</li>
<li>Firestore: 0.060</li>
</ul>
</li>
<li>Writes: $/100,000
<ul>
<li>Datastore: $0.108</li>
<li>Firestore: $0.180</li>
</ul>
</li>
<li>Deletes: $/100,000
<ul>
<li>Datastore: 0.012</li>
<li>Firestore: 0.020</li>
</ul>
</li>
</ul>
<p>Max entity size and document size are both 1MB, so you could realistically just
use JSON strings in entities instead to save money if you don't need any of
Firestore's other features.</p>
<hr>
<h1>Datastore/Firestore Concepts</h1>
<p>Both options use a non-relational/NoSQL database. NoSQL databases in general
allow for the direct serialization of a programs objects/dictionaries.</p>
<h2>Firestore</h2>
<p><a href="https://cloud.google.com/firestore/docs/data-model">Official Firestore data model docs</a></p>
<p>Firestore uses a document database organized into documents and collections.
This is a change from the older Datastore model of Entities organized by Kinds.</p>
<p>Documents support nested data structures ("Maps"). Think of them like JSON
objects, but with more types supported. Documents are limited to 1MB each.</p>
<p>Documents are stored in Collections.
Collections are containers for documents, like a filesystem's directories.
Documents can also contain their own nested collections, which store other
documents. Nesting has a depth limit of 100.</p>
<p>Documents are identified by References. In python, a reference looks like
this:</p>
<pre><code class="language-python">collection('collection1').document('document1')
</code></pre>
<h2>Datastore</h2>
<p>In Datastore, objects are called Entities. The key-value data pairs stored by
entities are called properties.</p>
<p>Entities are uniquely identified by keys.
Entity Keys consists of the following components:</p>
<ul>
<li><em>Namespace</em>: Basically a tenant identifier, like a user ID. By default, the
namespace is an empty string. Any entity created as a "child" will inherit
its parent's namespace.</li>
<li><em>Kind</em>: A string describing the category or purpose of the entity.</li>
<li><em>Identifier</em>: Unlike namespace and kind which are shared between like
entities, the identifier is unique to each entity. This is either a string
called a "key name", or an integer numeric ID.</li>
<li><em>Ancestor Path</em>: An optional component of the key, identifying its parents to
form a database hierarchy.</li>
</ul>
<hr>
<h1>Create IAM Service Account</h1>
<p>Before your code can access Datastore/Firestore, access needs to be configured
in the <a href="https://console.cloud.google.com/iam-admin">Identity Access Management</a>
tool.</p>
<p>First, create a <a href="https://console.cloud.google.com/iam-admin/serviceaccounts">Service Account</a>:</p>
<ol>
<li>Open the Service Account page</li>
<li>Select your project</li>
<li>Click "Create Service Account"</li>
<li>Service Account Details:
<ol>
<li>Service account name: <code>PythonDatastoreAdmin</code></li>
<li>Service account ID: <code>pythondatastoreadmin</code></li>
<li>Service account description: SA with Full Datastore Access for Python</li>
</ol>
</li>
<li>Grant this service account access to project (optional)
<ol>
<li>Select a role: Datastore > Cloud Datastore Owner</li>
<li>Continue</li>
</ol>
</li>
<li>Grant users access to this service account (optional)
<ol>
<li>Download a key file for Python to use. Use JSON as the key type.</li>
<li>Done</li>
</ol>
</li>
</ol>
<p>I downloaded my key to <code>$HOME/gcp_keys/datastore_admin.json</code></p>
<hr>
<h1>Python Environment Setup</h1>
<p>Build a virtualenv. If you're in a git repo, don't forget to ignore this
directory.</p>
<pre><code class="language-bash">python3 -m venv env3
source env3/bin/activate
</code></pre>
<p>Write the requirement file</p>
<p><code>vi requirements.txt</code></p>
<pre><code class="language-text">google-cloud-firestore
</code></pre>
<p>Install the requirements</p>
<pre><code class="language-bash">pip install -r rqeuirements.txt
</code></pre>
<hr>
<h1>Python Code Examples</h1>
<p>A fake project name of "example" will be used. Be sure to use a valid one.</p>
<p>Some notes about the Firestore python API:</p>
<ul>
<li><code>firestore.client().collection</code> returns a <a href="https://googleapis.dev/python/firestore/latest/collection.html">CollectionReference</a>,
which allows you to operate on the named collection.</li>
<li><code>firestore.client().collection().list_documents</code> returns a <a href="google.api_core.page_iterator.GRPCIterator">Page Iterator</a>.
Casting a page iterator to a list will trigger a read operation, returning a
list of <code>DocumentReference</code> objects.</li>
<li><code>firestore.client().collection().document</code> returns
a <a href="https://googleapis.dev/python/firestore/latest/document.html">DocumentReferece</a>,
which interacts with the actual Firestore documents. The <code>DocumentReference</code>
object can operate (get/set/create) on the actual Document.</li>
<li><code>firestore.client().collection().document().get()</code> will return
a <a href="https://developers.google.com/android/reference/com/google/firebase/firestore/DocumentSnapshot">DocumentSnapshot</a>
of the document. The document snapshot is how you read data from a document.
Document snapshots are a representation of the document state taken at the
instant of their creation.</li>
<li><code>firestore.client().collection().document().create()</code> makes a new document.
If the object already  existed, a <code>google.api_core.exceptions.AlreadyExists</code>
exception is thrown.</li>
<li>Write operations (create &#x26; set) return a <a href="https://cloud.google.com/firestore/docs/reference/rest/v1/WriteResult">WriteResult</a>.</li>
</ul>
<pre><code class="language-python">from os import environ
from google.cloud import firestore

# The key file is an exported JSON file from the IAM tool
key_file_path = '{}/gcp_keys/datastore_admin.json'.format(environ['HOME'])

# When running this on a local system, the cloud.google package from GCP
# expects the GOOGLE_APPLICATION_CREDENTIALS environment variable to be set
# to the exported IAM key file's path
environ['GOOGLE_APPLICATION_CREDENTIALS'] = key_file_path

# google.cloud.datastore.Client expects the project name as an argument
project = 'example'

# Instantiate the datastore client. This step authenticates to GCP.
# See the Credential Error note below if authentication fails
client = firestore.Client(project)

# Get a reference to the test collection. This returns a
# google.cloud.firestore_v1.collection.CollectionReference object
ref_col_test = client.collection('test')

# Get a reference to a document which may or may not exist. This returns a
# google.cloud.firestore_v1.document.DocumentReference object
ref_doc_document1 = ref_col_test.document('document1')

# Perform the actual database query to get the document. get() returns a
# google.cloud.firestore_v1.document.DocumentSnapshot object.
snapshot_document1 = ref_doc_document1.get()

# Note that there is no data
assert document1.to_dict() is None

# You can use document.exists to check if the document already exists.
# Here's an example of doing a safe insert of some sample data:
sample_data = {
    'nums': {'num1': 100, 'num2': 200},
    'strs': {'animal1': 'dog', 'animal2': 'cat'}}
if snapshot_document1.exists is False:
    createe_write_result = ref_doc_document1.create(sample_data)

document1_data = ref_doc_document1.get().to_dict()

# Finally, update the data
sample_data['strs']['animal3'] = 'bird'
set_write_result = ref_doc_document1.set(sample_data)

# List the documents in the collection. The iterator returns a paginated list
# of document references. You can then operate on those references.
doc_list_iterator = ref_col_test.list_documents()
doc_ref_list = list(doc_list_iterator)
sample_document_snapshot = doc_ref_list[0].get()

# Show that the changes were applied to the data. This will return 'bird'
animal3 = sample_document_snapshot.to_dict()['strs']['animal3']
</code></pre>
<h2>Troubleshooting</h2>
<h3>Credentials Error</h3>
<p>If your local google cloud authentication isn't set up correctly, you'll get an
error saying:</p>
<pre><code class="language-text">google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
</code></pre>
<p>If that happens, review the IAM Service account steps.</p>
<h3>Iterator has already started</h3>
<p>If you try and cast a <code>GRPCIterator</code> to list twice, you get a <code>ValueError</code>.</p>
<pre><code class="language-text">ValueError: ('Iterator has already started', &#x3C;google.api_core.page_iterator.GRPCIterator object)
</code></pre>
<p>The <code>ValueError</code> is thrown because <code>._started</code> is <code>True</code> in the iterator. It has
already done its thing and won't work again. I didn't see any good way to
restart the iterator without discarding it.
You can see the details in the <a href="https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/page_iterator.py">page iterator source code</a>.</p>
<p>Best course is just to get a new iterator from the reference object.
See in the following example how the ID changes. The <code>doc_list_iterator</code> var is
being pointed at a new iterator. The old iterator gets gc'd by Python as usual.</p>
<pre><code>>>> id(doc_list_iterator)
4422123408
>>> doc_list_iterator = ref_col_test.list_documents()
>>> id(doc_list_iterator)
4422123472
</code></pre>
<p>It will work as expected.</p>


<p>This post covers getting started with a very simple Google Cloud Functions API,
a great serverless solution with affordable pricing. This API will pull the
IP address from the HTTP request, and return <code>Hello &#x3C;source IP></code>.</p>
<hr>
<h3>Table of Contents</h3>
<ul>
<li><a href="#pricing-details">Pricing Details</a></li>
<li><a href="#deploy-hello-world-function">Deploy Hello World Function</a>
<ul>
<li><a href="#mainpy">main.py</a></li>
<li><a href="#requirementspy">requirements.py</a></li>
</ul>
</li>
<li><a href="#accessing-the-function">Accessing the Function</a>
<ul>
<li><a href="#from-the-console-web-ui">From the Console Web UI</a></li>
<li><a href="#from-the-request-url">From the Request URL</a></li>
</ul>
</li>
</ul>
<h1>Pricing Details</h1>
<p>Cloud Functions pricing is pretty affordable for most use cases, assuming you
write your functions in such a way that they don't get invoked a few too many
millions of times.</p>
<p>Check the official <a href="https://cloud.google.com/functions/pricing">Google docs</a>
for accurate and up-to-date pricing, but here's what it looked like at the time
of my writing this.</p>
<p>Pricing is broken into a few fees:</p>
<ul>
<li><em>Invocations</em>: A flat fee per function execution. $0.0000004 per invocation,
excluding the first 2 million.</li>
<li><em>Compute Time</em>: At the lowest clock speed (200MHz x 128MB RAM), compute costs
$0.000000231/100ms, ceil 100.</li>
<li><em>Networking</em>: $0.12/GB egress for data, while ingress is free.</li>
</ul>
<hr>
<h1>Deploy Hello World Function</h1>
<p>Open <a href="https://console.cloud.google.com/functions/">Google Cloud Functions</a> in
your browser.</p>
<p>Create a function.</p>
<ul>
<li>Name: HelloWorld</li>
<li>Memory Allocated: 128 MB</li>
<li>Trigger: HTTP</li>
<li>Authentication: Allow unauthenticated invocations</li>
<li>Source code: Inline Editor</li>
<li>Runtime: Python 3.7</li>
<li>Other settings: Leave them as defaults</li>
</ul>
<h2>main.py</h2>
<p>The API function defined by <code>Function to execute</code> accepts a
<a href="https://flask.palletsprojects.com/en/1.1.x/api/#incoming-request-data">flask.Request</a>
argument.</p>
<pre><code class="language-python">def hello_world(request):
    """" Return a Hello message to sender of the API call """
		return 'Hello {}!'.format(request.remote_addr)
</code></pre>
<h2>requirements.py</h2>
<p>You can leave this blank. It's a list of libraries that will be used.</p>
<hr>
<h1>Accessing the Function</h1>
<h2>From the Console Web UI</h2>
<ol>
<li>Navigate to the <a href="https://console.cloud.google.com/functions">Functions page</a>.</li>
<li>Click on the 3 dots next to the function name, and choose "Test function".</li>
<li>Populate and JSON for the GET request in "Triggering event". The above
example doesn't accept any input data so <code>{}</code> is all you need.</li>
<li>Click "Test the function"</li>
</ol>
<h2>From the Request URL</h2>
<ol>
<li>Navigate to the <a href="https://console.cloud.google.com/functions">Functions page</a>.</li>
<li>Click the function's name to navigate to the "Function details" page</li>
<li>Go to the 'Trigger" tab</li>
<li>Find the URL. Copy it.</li>
</ol>
<p>You can now test the URL with curl, your browser, httpie, python, etc.</p>
<pre><code class="language-bash">https://&#x3C;region>-&#x3C;project>.cloudfunctions.net/HelloWorld
</code></pre>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to upload and download files from Amazon's S3 using
Python. Setup is done from the web UI.</p>
<h1>Create a Bucket</h1>
<p>Go to the <a href="https://s3.console.aws.amazon.com">AWS S3 page</a>.</p>
<p>Click on "+ Create Bucket" and fill out the form. For access, choose
Block all public access. For the rest of the options, the defaults are fine.</p>
<hr>
<h1>Create IAM Credentials</h1>
<p>IAM stands for Identity and Access Management.</p>
<h2>Create a Policy</h2>
<p>From the left nav-bar, go to policies. The default policies don't grant
programmatic access, so you need to make a new one.</p>
<p>For admin rights, I just used the visual policy builder and granted it
access to everything. I'll come back and update this guide with how to lock
that down later.</p>
<hr>
<h1>Add the policy to a group</h1>
<p>Go to the groups on the left. Either make a new one or edit an existing one.
Under Permissions, click Attach Policy. If you want, you can also add
<code>AmazonS3FullAccess</code> to see the difference between the two.</p>
<h2>Create/Edit a User</h2>
<p>On the left nav-bar, go to Users > Add User. Or edit a user that exists.
Fill in the wizard and assign the group.</p>
<hr>
<h1>Use Python to interact with S3</h1>
<h2>Install Boto3</h2>
<p>Python needs boto3 installed to interact with AWS.</p>
<pre><code class="language-bash">pip install boto3
</code></pre>
<h2>Uploading and Downloading Files</h2>
<pre><code class="language-python">import boto3

# Fill in the key and key id, or read them from stdin/file/whatever
key_id = ''
key = ''

# Create the authenticated s3 client
session = boto3.Session(aws_access_key_id=key_id, aws_secret_access_key=key)
client = session.client('s3')

# Define the file/object
filename = '/example/file/path'
bucket_name = 'my_bucket'
key = 'my_key'

# Upload Example
client.upload_file(filename, bucket_name, key)

# Download Example
client.download_file(bucket_name, object_name, path)
</code></pre>


<p>You've deployed a server in a customer environment and you don't have remote
network access configured. The customer is OK with you being able to remote
into their network, but its not practical to set you up with a VPN. Enter the
"Dial-Home Device".</p>
<h1>What is the Dial-Home Device</h1>
<p>It's just an Ubuntu server that repeatedly tries to dial an internet connected
jump box in order to open a reverse session. As soon as the networking is set
up, it will dial home and open itself up for remote administration.</p>
<p>Also, <a href="https://stargate.fandom.com/wiki/Dial_Home_Device">this.</a></p>
<hr>
<h1>Deploy the Pivot Server</h1>
<h2>What's a Pivot Server?</h2>
<p>This is an internet-connected server such as a VPS that the DHD will SSH to.</p>
<p>You'll SSH to it, then from there you can SSH through the DHD's reverse tunnel.</p>
<h2>Steps to Deploy</h2>
<h3>Launch a VPS</h3>
<p>Using AWS, GCP, Azure, Linode, Digital Ocean, so on, launch an
internet-connected Ubuntu server. It needs to have a static, internet routed IP
address. Its not a bad idea to make a DNS entry for it too, so you can change
the IP down the road.</p>
<p>SSH to the server to confirm that it works.</p>
<h3>Configure Remote User</h3>
<p>These steps will create a user named <code>remoteuser</code></p>
<pre><code class="language-bash"># Create the user
useradd remoteuser

# Set a password for the user
passwd remoteuser

# Create the home &#x26; ssh directories for the user
mkdir -p /home/remoteuser/.ssh
chown -R remoteuser:remoteuser /home/remoteuser

# Disable shell access to the user, so it can't run commands on the pivot
usermod -s /bin/false remoteuser
</code></pre>
<hr>
<h1>Deploy the Dial-Home Device</h1>
<p>Install Ubuntu Server on a physical server or VM at the remote location. This
server should have network connectivity to whatever resource you want to
manage, and outbound internet access (it can ping google).</p>
<h2>Create an SSH key on the DHD</h2>
<p>On the DHD server, you may already have SSH keys. Look in <code>~/.ssh</code>. If not,
run the following to create one. Don't use a passphrase, just hit enter when
prompted.</p>
<pre><code class="language-bash">ssh-keygen
</code></pre>
<p>Collect the public key for the next step:</p>
<pre><code class="language-bash">cat ~/.ssh/id_rsa.pub
</code></pre>
<p>For convenience, I like to copy the key to the root user too. Optional.</p>
<h2>Authorize the SSH Public Key on the Pivot Serer</h2>
<p>Back on the Pivot server, edit the remote user's <code>authorized_keys</code> file to
grant this SSH key-pair access. Paste the SSH key at the end of the file</p>
<pre><code class="language-bash">vi /home/remoteuser/.ssh/authorized_keys
</code></pre>
<h2>Test a remote connection</h2>
<h3>Initiate the connection</h3>
<p>From the DHD server, open the reverse tunnel manually to make sure it works.</p>
<ul>
<li>Replace the username as needed</li>
<li>The <code>$pivot</code> variable should be the IP or FQDN of the pivot server</li>
<li>Pick a random port in the range 1024-49151 for the listen port.</li>
</ul>
<pre><code class="language-bash">user="remoteuser"
pivot="support.example.com"
port=22222
ssh -fN -R 0.0.0.0:$port:127.0.0.1:22 $user@$pivot
</code></pre>
<h3>Connect through the pivot</h3>
<p>From the pivot server, SSH to the DHD. This example assumes your username
is <code>myuser</code>.</p>
<pre><code class="language-bash">ssh -p 22222 myuser@127.0.0.1
</code></pre>
<p>Consider adding your public key to the <code>~/.ssh/authorized_keys</code> file now so
passwords aren't needed.</p>
<p>Now you can jump through the pivot into the DHD.</p>
<hr>
<h1>Persist the Connection</h1>
<h2>Build the dial-home command</h2>
<p>From any directory, create the script:</p>
<p><code>vi dial-home</code></p>
<pre><code class="language-bash">#!/usr/bin/env bash

# Check if the reverse tunnel  is already open
port=22222
if [[ $(ps aux | grep ssh | grep $port) ]]; then
  # reverse tunnel is open
  exit 0
fi

# tunnel is not open, open it
user="remoteuser"
pivot="support.example.com"
ssh -fN -R 0.0.0.0:$port:127.0.0.1:22 $user@$pivot
</code></pre>
<p>Make the script executable</p>
<pre><code class="language-bash">chmod +x dial-home
</code></pre>
<p>Move the script to somewhere in the <code>$PATH</code></p>
<pre><code class="language-bash">sudo mv dial-home /usr/local/bin/
</code></pre>
<p>Now the <code>dial-home</code> command will open the tunnel.</p>
<h2>Make cron dial-home every minute</h2>
<p>The connection will die for lots of reasons. Make it bring itself back online
automatically using cron.</p>
<p>From the DHD, make sure cron is running (by default, it is)</p>
<pre><code class="language-bash">systemctl status cron
</code></pre>
<p>Edit the crontab</p>
<p><code>sudo vi /etc/crontab</code></p>
<p>and insert this line, where <code>myuser</code> is your username on the DHD.
If you use root, you have to also give root the ssh private key.</p>
<pre><code class="language-text">* * * * *   myuser dial-home
</code></pre>
<p>Reload cron</p>
<pre><code class="language-bash">service cron reload
</code></pre>
<p>Then kill the session (using <code>ps aux</code> and <code>kill</code>) then watch it come back.</p>
<p>Done. Now the server is a dial-home device. It will constantly dial your pivot
server and re-open the session whenever its closed. To lock the server out,
just remove its public key from <code>remoteuser</code>'s <code>authorized_keys</code> file.</p>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page lists some of the errors that I've personally encountered,
and how I fixed them. I might also include errors I never fixed in case anyone
wants to talk about it.</p>
<hr>
<h3>Table of Contents</h3>
<ul>
<li><a href="#docker-login-fails">Docker Login Fails</a></li>
<li><a href="#runaway-memory-usage-in-ceph-osd-process">Runaway memory usage in Ceph OSD process</a></li>
<li><a href="#chrome-blocks-https127001-with-neterr_cert_revoked">Chrome blocks https://127.0.0.1 with NET::ERR_CERT_REVOKED</a></li>
<li><a href="#failed-command-aws-ecr-get-login">Failed command: aws ecr get-login</a></li>
<li><a href="#pure-storage-iscsi-with-kolla-openstack---fdisk-freezes">Pure Storage iSCSI with Kolla OpenStack - fdisk freezes</a></li>
<li><a href="#weird-chars-when-pasting-on-mac-os">Weird chars when pasting on Mac OS</a></li>
</ul>
<h1>Docker Login Fails</h1>
<pre><code>Error saving credentials: error storing credentials - err: exit status 1, out: `Cannot autolaunch D-Bus without X11 $DISPLAY`
</code></pre>
<p>I had this happen after running both Ceph-Ansible then Kolla-Ansible on an Ubuntu server, so it could be either of those that set up the conditions for this to happen. <a href="https://github.com/docker/cli/issues/1136">this solution</a> worked for me:</p>
<pre><code class="language-bash">apt-get install -y gnupg2 pass
</code></pre>
<hr>
<h1>Runaway memory usage in Ceph OSD process</h1>
<p>It looked like a memory leak. Each OSD process was using around 30GB of RAM.</p>
<p>It turns out that Ceph-Ansible has the following logic when generating its
<code>ceph.conf</code> file:</p>
<pre><code># this math
roles/ceph-config/templates/ceph.conf.j2:{% set _osd_memory_target = (ansible_memtotal_mb * 1048576 * hci_safety_factor / _num_osds) | int %}
# sets osd memory target
roles/ceph-config/templates/ceph.conf.j2:osd memory target = {{ _osd_memory_target | default(osd_memory_target) }}
</code></pre>
<p>In my case, this set the value WAY too high. In one of my deployments, it was
set to <code>37853425827</code>, which is like 35GB (PER OSD SERVICE!).</p>
<p>The 4GB default is much more sane, <code>4294967296</code>.</p>
<pre><code># ceph.conf
[osd]
osd memory target = 4294967296
</code></pre>
<hr>
<h1>Chrome blocks <a href="https://127.0.0.1">https://127.0.0.1</a> with NET::ERR_CERT_REVOKED</h1>
<p>This comes up when I reverse tunnel to a remove web server, forwarding it to my workstation. Chrome has decided to not let you do that. They give this errror:</p>
<pre><code class="language-text">Your connection is not private
Attackers might be trying to steal your information from 127.0.0.1 (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_REVOKED
</code></pre>
<p>To bypass that, open <code>chrome://flags/#allow-insecure-localhost</code> and set allow-insecure-localhost to enabled. Click relaunch. This will nuke your current browser…</p>
<p>Once it relaunches you'll be able to open the site.</p>
<hr>
<h1>Failed command: aws ecr get-login</h1>
<p>This error was caused by a bad NTP config. Renew the NTP lease to fix it.</p>
<pre><code>An error occurred (InvalidSignatureException) when calling the GetAuthorizationToken operation: Signature expired: 20190911T174538Z is now earlier than 20190912T004835Z (20190912T010335Z - 15 min.)
</code></pre>
<hr>
<h1>Pure Storage iSCSI with Kolla OpenStack - fdisk freezes</h1>
<p>This was a weird one. When using the Kolla containers project to make a
Cinder-volume container that has the <code>purecinder</code> plugin, I was able to get
OpenStack to create the volumes in Pure but it couldn't mount them to the hosts
for things like creating a volume with the <code>--image</code> argument. That made the
volumes I could create basically useless.</p>
<p>These errors were in the logs:</p>
<pre><code class="language-text"># Lots of these in cinder's log
Trying to connect to iSCSI portal

# also from cinder's log, also recurring:
iscsiadm stderr output when getting sessions: iscsiadm: No active sessions

# in DMESG
FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
</code></pre>
<p>The weirdest part was that the <code>fdisk -l</code> command would totally lock up and
the process running it couldn't be killed even with <code>kill -9</code>. The iSCSI volume
never got properly mounted but Pure support said they could see some data
coming into it.</p>
<p><strong>Root cause</strong>: MTU mismatch. Of course. I had set MTU 9000 on the iSCSI ports,
my host ports, and the switchports heading to the Pure Storage iSCSI
interfaces. I had forgotten to set it on the switch heading to the Dell servers
being used for OpenStack.</p>
<hr>
<h1>Weird chars when pasting on Mac OS</h1>
<p>When you paste sometimes you get <code>^[[200~</code> at the start and <code>^[[201~</code> at the
end of the pasted content. Or maybe <code>0~</code> and <code>1~</code>.</p>
<p>To resolve, run:</p>
<pre><code class="language-bash">printf "\e[?2004l"^C
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page contains operations-related mini-guides and minor posts.</p>
<hr>
<h3>Table of Contents</h3>
<ul>
<li><a href="#find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</a></li>
<li><a href="#convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</a></li>
<li><a href="#journalctl-commands">Journalctl commands</a></li>
<li><a href="#get-real-ram-use-of-processes">Get real RAM use of processes</a>
<ul>
<li><a href="#of-ceph">of ceph</a></li>
<li><a href="#of-vms">of VMs</a></li>
</ul>
</li>
<li><a href="#bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</a></li>
<li><a href="#create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</a>
<ul>
<li><a href="#use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</a></li>
</ul>
</li>
<li><a href="#test-mtu-from-ubuntu">Test MTU from Ubuntu</a></li>
<li><a href="#find-big-files-on-mac-os">Find big files on Mac OS</a></li>
<li><a href="#renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</a></li>
<li><a href="#create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</a></li>
<li><a href="#passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</a></li>
<li><a href="#vim-tasks">VIM Tasks</a>
<ul>
<li><a href="#find--replace">Find &#x26; Replace</a></li>
<li><a href="#spell-check-commands">Spell-Check commands</a></li>
</ul>
</li>
<li><a href="#use-an-insecure-docker-registry">Use an Insecure Docker Registry</a></li>
<li><a href="#ansible-ignore-errors">Ansible ignore errors</a></li>
<li><a href="#how-to-tail--f-on-dmesg">How to tail -f on dmesg</a></li>
<li><a href="#ubuntu-downgrade-a-package">Ubuntu Downgrade a package</a></li>
<li><a href="#make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</a></li>
<li><a href="#bash-print-all-but-first-line">Bash: Print all but first line</a></li>
<li><a href="#bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</a></li>
</ul>
<h1>Find which active-passive bond member is active</h1>
<pre><code>cat /proc/net/bonding/bond0
</code></pre>
<hr>
<h1>Convert p7b certificate to pem format</h1>
<pre><code class="language-bash">openssl pkcs7 -print_certs -in certnew.p7b -out cert.pem
</code></pre>
<hr>
<h1>Journalctl commands</h1>
<p>Watch the logs as they come in:</p>
<pre><code class="language-bash">journalctl -f
</code></pre>
<p>When journalctl logs too much and you need to pair it down:</p>
<pre><code class="language-bash"># to 30 days
journalctl --vacuum-time=30d

# to 500MB
journalctl --vacuum-size=500M
</code></pre>
<p>Show for a specific service</p>
<pre><code class="language-bash">journalctl ceph*
</code></pre>
<hr>
<h1>Get real RAM use of processes</h1>
<p>You can always use top or htop or whatever. I use this to specifically get the
info about those processes.</p>
<h2>of ceph</h2>
<pre><code>ps aux | grep ceph | grep -v -e grep -e qemu -e "\[" | while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); id=$(echo $line | awk '{print $16}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tid: $id,    \tRAM: ($size)"; done
</code></pre>
<h2>of VMs</h2>
<p>for KVM</p>
<pre><code>ps aux | grep qemu | grep -v -e grep| while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); guest=$(echo $line | awk '{print $13}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tguest: $guest,   \t\tRAM: ($size)" ; done
</code></pre>
<h1>Bypass Chrome's self-signed cert warning</h1>
<p>As of Mac Catalina, developing a site using self-signed certs is a bigger pain
than before. Yeah you can go get the key and add it, but that's a nuisance.
It looks like the Chrome team built something they call an
"interstitial bypass keyword" to deal with that. That's one cool feature name.</p>
<p><strong>Obvious disclaimer</strong> If you're not a developer doing this to access your own
site, don't do this.</p>
<p>To bypass the warning, type</p>
<pre><code>thisisunsafe
</code></pre>
<p>Apparently it used to be <code>badidea</code> but they rotated it since it got too well
known. They'll probably do it again someday. I suspect people are using this
on sites they "trust" without really knowing any better and getting hurt by
it.</p>
<hr>
<h1>Create a virtual loopback volume on Ubuntu</h1>
<p>Useful for simulating a real drive when building a test environment. I use this to test OpenStack in a VM as a Ceph or LVM back-end.</p>
<pre><code class="language-bash"># create empty file of all zeros
sudo dd if=/dev/zero of=/root/virtual-disk bs=1M count=512
# show list of loopback devices. Note its not there
losetup --list
# create the loopback device
losetup /dev/loop0 /root/virtual-disk
# confirm it worked
losetup --list
</code></pre>
<p>You can now treat it like a normal volume. Mount it, put a filesystem on it, whatever. Example:</p>
<pre><code class="language-bash">pvcreate /dev/loop0
vgcreate cinder-volumes /dev/loop0
</code></pre>
<h2>Use Systemd to re-map loopback on reboot</h2>
<p>I found <a href="https://unix.stackexchange.com/questions/418322/persistent-lvm-device-with-loopback-devices-by-fstab">this useful systemd unit example</a> that didn't quite work for me, but was close. Here's mine:</p>
<p><code>vi /etc/systemd/system/loops-setup.service</code></p>
<pre><code class="language-text">[Unit]
Description=Setup loopback devices

DefaultDependencies=no
Conflicts=umount.target

Requires=lvm2-lvmetad.service mnt-host.mount
Before=local-fs.target umount.target
After=lvm2-lvmetad.service mnt-host.mount

[Service]
ExecStart=/sbin/losetup /dev/loop0 /root/virtual-disk
ExecStop=/sbin/losetup -d /dev/loop0

RemainAfterExit=yes
Type=oneshot

[Install]
WantedBy=local-fs-pre.target
</code></pre>
<p>Then enable it</p>
<pre><code class="language-bash">systemctl enable loops-setup
</code></pre>
<p>Reboot, then check <code>vgdisplay</code> (if that's your use case). It'll be there.</p>
<hr>
<h1>Test MTU from Ubuntu</h1>
<p>You would not believe the strange problems that an MTU issue can be the root cause of.</p>
<pre><code class="language-bash">ping -M do -s 1472 &#x3C;remote server>
ping -M do -s 8900 &#x3C;remote server>
</code></pre>
<hr>
<h1>Find big files on Mac OS</h1>
<p>My SSD is always almost full. Here are useful commands for cleaning up:</p>
<pre><code class="language-bash"># Find all the files over 1G
sudo find / -type f -size +1G -exec ls -lh {} \; | awk '{ print $9 ": " $5 }' 2>/dev/null

# List files in Downloads directory by size
ls -lhS ~/Downloads
</code></pre>
<h1>Renew NTP Lease Ubuntu</h1>
<pre><code class="language-bash">service ntp stop
ntpd -gq
service ntp start
</code></pre>
<hr>
<h1>Create AWS ECR Registry If Not Found</h1>
<pre><code class="language-bash">aws ecr describe-repositories --repository-names $repo_name \
    || aws ecr create-repository --repository-name $repo_name
</code></pre>
<hr>
<h1>Passwordless sudo on Ubuntu Server</h1>
<p>Run <code>visudo</code> as root.</p>
<pre><code class="language-bash"># as root
export EDITOR=vim
visudo
</code></pre>
<p>Add the user (replace <code>exampleUser</code>) at the end.</p>
<pre><code class="language-text">exampleUser ALL=(ALL) NOPASSWD: ALL
</code></pre>
<p>For those who don't use vim, a quick reminder:
<code>i</code> enter to insert mode , <code>[esc]</code> to exit insert mode, <code>:wq</code> to save and quit.</p>
<hr>
<h1>VIM Tasks</h1>
<h2>Find &#x26; Replace</h2>
<pre><code class="language-bash"># replace findThis with ReplaceWithThis
:%s/findThis/ReplaceWithThis/g
</code></pre>
<h2>Spell-Check commands</h2>
<pre><code class="language-text"># enable/disable
:set spell spelllang=en_ca
:setlocal spell spelllang=en_ca
:set nospell

# next/last
]s
[s

# suggest change
z=

# add/remove word from dictionary
zg
zw`
</code></pre>
<hr>
<h1>Use an Insecure Docker Registry</h1>
<p>To use a registry with no HTTP cert:</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
    "insecure-registries" : ["myregistrydomain.com:5000"]
}
</code></pre>
<hr>
<h1>Ansible ignore errors</h1>
<p>The command is <code>ignore_errors: yes</code></p>
<p>Just throw this into the task that's failing. It will still fail, but the play
won't stop.</p>
<p>Example:</p>
<pre><code class="language-yml">- name: Ensure the docker service is running
  script: ../tools/validate-docker-execute.sh
    ignore_errors: yes
</code></pre>
<hr>
<h1>How to tail -f on dmesg</h1>
<pre><code class="language-bash">dmesg -wH
</code></pre>
<hr>
<h1>Ubuntu Downgrade a package</h1>
<pre><code class="language-bash"># Find the available versions under Versions:
apt-cache showpkg &#x3C;package>

# Install one of those
apt-get install &#x3C;package>=&#x3C;version>
</code></pre>
<hr>
<h1>Make Ansible run Python 3 on target nodes</h1>
<p>Useful for OS that ship with python3 but not python2.
Edit the inventory, find a group of the groups and apply this var.</p>
<pre><code class="language-ini">[baremetal:vars]
ansible_python_interpreter=/usr/bin/python3
</code></pre>
<hr>
<h1>Bash: Print all but first line</h1>
<p>Use <code>tail -n +2</code>.
This was unintuitive to me because +1 seems like it should do it.</p>
<pre><code class="language-bash"># Example
docker images | awk '{print $3}' | tail -n +2
</code></pre>
<hr>
<h1>Bash: Run code whenever a script exits</h1>
<p>Use trap.</p>
<ul>
<li><a href="http://redsymbol.net/articles/bash-exit-traps/">This is a nice post covering lots of examples</a>.</li>
<li><a href="https://github.com/GoogleCloudPlatform/endpoints-quickstart/blob/master/scripts/deploy_api.sh">Here's where I first saw it used</a></li>
</ul>
<pre><code class="language-bash">temp_file=$(mktemp)

cleanup() {
  rm "$temp_file"
}

trap cleanup EXIT
</code></pre>


<p>How to run a docker container on AWS ECS (Elastic Compute Cloud)  using the AWS
web console. A container from the Elastic Container Registry will be launched.
It will run on Amazon's <a href="https://aws.amazon.com/fargate/">Fargate Container-as-a-Service</a>.</p>
<hr>
<h3>Table of Contents</h3>
<ul>
<li><a href="#launch-a-container-on-fargate">Launch a container on Fargate</a></li>
<li><a href="#publish-containers-network-to-the-internet">Publish container's network to the internet</a></li>
<li><a href="#using-a-custom-domain-name">Using a custom domain name</a></li>
</ul>
<h1>Launch a container on Fargate</h1>
<ol>
<li>Upload an image to AWS ECR. <a href="/aws-ecr.html">See my guide on using AWS ECR</a>.</li>
<li>Open <a href="https://ca-central-1.console.aws.amazon.com/ecs/">AWS ECS</a></li>
<li>Under Container Definition, select Configure</li>
<li>Fill in the form
<ol>
<li>For the image URL, use the URL of the image uploaded to
ECR. I was testing a reverse proxy image, so it looked like this:
<code>000000000000.dkr.ecr.ca-central-1.amazonaws.com/test/revproxy:latest</code></li>
<li>I have my nginx reverse proxy a 256MB soft limit for RAM</li>
<li>For my reverse proxy I made port mappings for port 80 and 443</li>
<li>No need to fill out any of the advanced fields</li>
</ol>
</li>
<li>Fill out the Task Definition section, click Next</li>
<li>Define your service
<ol>
<li>Without a load-balancer, the only way to access your container will be by
using <a href="https://aws.amazon.com/blogs/compute/access-private-applications-on-aws-fargate-using-amazon-api-gateway-privatelink/">PrivateLink</a>.</li>
<li>During the initial cluster create wizard you can choose an Application
Load Balancer. That seems to work well.</li>
</ol>
</li>
<li>Name the Fargate cluster, next</li>
<li>Click create</li>
</ol>
<p>It will take a little while. Once its finished you can click on View Service.</p>
<hr>
<h1>Publish container's network to the internet</h1>
<p>Navigate to the newly created service and go to the Details tab. From there you
can see a Load Balancing section. It should have a link to the new load
balancer.</p>
<p>You can also access this page from the left nav-bar by going to Load Balancing,
then choosing the new load balancer.</p>
<p>In the Description tab here, you can find the DNS name that AWS has assigned
to this load balancer. Unfortunately you can't depend on the IP address behind
this FQDN to stay the same.</p>
<hr>
<h1>Using a custom domain name</h1>
<p>I'm working on that. Its not intuitive. If you know how, I'd like to hear about
it..</p>


<p>Docker supports shared memory. The default is only 64MB though which is TINY.</p>
<p>The small size became a problem during a containerized OpenStack upgrade.</p>
<p>This will be a really short post because I haven't done much playing with the
shared memory yet, I just think its really neat.</p>
<hr>
<h1>Change the /dev/shm size on the system</h1>
<p>This lets you change the size of shared memory on a host.</p>
<pre><code class="language-bash">mount -o remount,size=1G /dev/shm
</code></pre>
<hr>
<h1>Update Docker Daemon's Default Shared Memory Size</h1>
<p>This will update the default across the whole Docker service.</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
  "default-shm-size": "512M"
}
</code></pre>
<hr>
<h1>Set a Containers Shared Memory Size</h1>
<p>Here's how to change it for a specific container at runtime.</p>
<pre><code class="language-bash">docker run -it --shm-size=512M
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This page covers how to do some things in Git.</p>
<hr>
<h3>Table of Contents</h3>
<ul>
<li><a href="#use-vim-for-commits">Use Vim for commits</a></li>
<li><a href="#rename-branch">Rename Branch</a></li>
<li><a href="#roll-back-merge">Roll Back Merge</a>
<ul>
<li><a href="#option-1-abort-the-merge">Option 1: Abort the merge</a></li>
<li><a href="#option-2-reset-to-the-last-commit">Option 2: Reset to the last commit</a></li>
</ul>
</li>
<li><a href="#delete-a-file-from-git-history">Delete a File From Git History</a>
<ul>
<li><a href="#if-you-dont-push-with---force">If you don't push with --force</a></li>
</ul>
</li>
</ul>
<h1>Use Vim for commits</h1>
<p>Ugh, Nano. Kidding, but seriously here's how to use Vim instead.</p>
<pre><code class="language-bash">git config --global core.editor "vim"
</code></pre>
<hr>
<h1>Rename Branch</h1>
<pre><code class="language-bash"># Check out the branch to rename
git checkout development
# Rename the branch
git branch -m master
# Push the change
git push origin :development master
</code></pre>
<p>If you're using BitBucket and it fails to delete the branch, it might be set
as your MAIN branch. You need to change that before it can be deleted. You can
change it in the settings page.</p>
<hr>
<h1>Roll Back Merge</h1>
<p>If you do a git merge and its really ugly, you might want to undo that.</p>
<p>First you did the commit</p>
<pre><code class="language-bash">git checkout master
git merge --squash development
</code></pre>
<p>Then you got a TON of <code>CONFLICT</code> messages and a really dirty state.</p>
<h2>Option 1: Abort the merge</h2>
<p>This one's my preference. There might be a few <code>~</code> files left over to <code>rm</code>
manually.</p>
<pre><code class="language-bash">git merge --abort
</code></pre>
<h2>Option 2: Reset to the last commit</h2>
<p>First, get the commit right before this one from git log</p>
<pre><code class="language-bash">git log -n 1
</code></pre>
<p>Copy the commit hash</p>
<pre><code class="language-bash">git reset --hard &#x3C;commit hash>
</code></pre>
<p>You may need to clean up some files after that.</p>
<p><strong>extra</strong>
Usually, the next thing you wanna do is try the merge again with
<code>--stattheirs</code>, or</p>
<pre><code class="language-bash">git merge --squash --strategy-option=theirs development
</code></pre>
<hr>
<h1>Delete a File From Git History</h1>
<p>It's important to note, as GitHub highlights <a href="https://help.github.com/en/articles/removing-sensitive-data-from-a-repository">in their documentation</a>,
if you've uploaded a secret password or something, you still need to change it.</p>
<p>The file I removed was called <code>sqldata.json</code>, and it was in my project root.
From the project root, I ran:</p>
<pre><code class="language-bash">git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch sqldata.json" \
  --prune-empty --tag-name-filter cat -- --all

git push origin --force --all
</code></pre>
<h2>If you don't push with --force</h2>
<p>When I tried to push, though, I got an error. I'd forgotten to use <code>--force</code>.</p>
<pre><code class="language-bash">To https://github.com/kylep/jsc2f.git
 ! [rejected]        master -> master (non-fast-forward)
 error: failed to push some refs to 'https://github.com/kylep/jsc2f.git'
 hint: Updates were rejected because the tip of your current branch is behind
 hint: its remote counterpart. Integrate the remote changes (e.g.
 hint: 'git pull ...') before pushing again.
 hint: See the 'Note about fast-forwards' in 'git push --help' for details.
</code></pre>
<p>Then, when I tried to pull, I got</p>
<pre><code class="language-bash">From https://github.com/kylep/jsc2f
 + ca7d7cc...710d37b master     -> origin/master  (forced update)
 fatal: refusing to merge unrelated histories
</code></pre>
<p>If you're getting errors like that, maybe you need to run push with <code>--force</code>.</p>
<pre><code class="language-bash">git push origin --force --all
</code></pre>


<p><strong>This post is linked to from the <a href="/open-source-cli-project.html">FOSS Python App Project</a></strong></p>
<hr>
<p>At a high level, the steps to upload a package to PyPi are as follows:</p>
<ol>
<li>Write the project code</li>
<li>Create a release on GitHub and put it in the <code>download_url</code> of setup.py</li>
<li>Write a setup.py file to build and describe the project</li>
<li>Write a setup.cfg file pointing to the readme</li>
<li>create a source distribution</li>
</ol>
<h1>Write the project code</h1>
<p>I'm not going to cover how to build a python project in this post, but here's
the very small and simple project I used when learning this procedure:
<a href="https://github.com/kylep/jsc2f">GitHub: kylep/jsc2f</a>.</p>
<hr>
<h1>Create a Release on GitHub</h1>
<p>Log into your GitHub project and click on releases. If you can't find the link,
you can also just append <code>/releases</code> to the end of the URL to get there.</p>
<p>Click "Create a new release". Enter your tag and branch, along with a release
title and description.</p>
<p>Under the Assets section of the new release there's a (tar.gz) link to download
the source code. If you right click it and copy the URL, that's used next in
the setup.py file.</p>
<hr>
<h1>Write a setup.py file</h1>
<p>Here's mine. Check GitHub to see if I've changed it.
The <code>download_url</code> value comes from the GitHub release.</p>
<pre><code class="language-python">from setuptools import setup

# Without this you'll get no description on the pypi site
with open("README.md", "r") as fh:
    long_description = fh.read()

setup(
    name='jsc2f',
    packages=['jsc2f'], # Alternatively you can use find_packages()
    version='0.1', # match the github release you make later
    license='MIT', # match the LICENSE file in the project
    description='Saves a JSON fields SQLs cell to a file, or UPDATE it back',
    author='Kyle Pericak',
    author_email='kyle@pericak.com',
    url='kyle.pericak.com/jsc2f', # either your own page or github
    download_url='https://github.com/kylep/jsc2f/archive/v0.1.tar.gz',
    keywords=['SQL', 'JSON', 'file'],
    install_requires=[
        'click',
        'mysql-connector'
    ],
    entry_points='''
        [console_scripts]
        jsc2f=jsc2f.cli:cli
    ''',
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Information Technology',
        'Topic :: Database',
        'Programming Language :: Python :: 3',
        'License :: OSI Approved :: MIT License',
        'Natural Language :: English'

</code></pre>
<h2>Picking a topic:</h2>
<p>I used <a href="https://pypi.org/pypi?%3Aaction=list_classifiers">this classifier list</a></p>
<hr>
<h1>Write setup.cfg</h1>
<p>This goes in the project root next to setup.py.</p>
<p><code>vi setup.cfg</code></p>
<pre><code class="language-ini">[metadata]
description-file = README.md
</code></pre>
<h1>Creating a source distribution</h1>
<p>For this you use the python <a href="https://docs.python.org/2/distutils/sourcedist.html">sdist</a>
tool and <a href="https://pypi.org/project/twine/">twine</a>.</p>
<p>From the git project root:</p>
<pre><code class="language-bash">pip install twine

# This will create dist/
python setup.py sdist

# Twine uploads the dist/ files to Pypi
twine upload --skip-existing dist/*
# Enter your username and password for PyPI
</code></pre>


<p>Often I need to operate on a huge JSON string that's saved in a MaraiDB
database. Here's how I do that.</p>
<p>It needs to be said that raw SQL queries like this are generally a bad idea
compared to something like <a href="https://docs.sqlalchemy.org/en/13/orm/">SQLAlchemy's ORM</a>.
Having said that, this example is way smaller and more portable.</p>
<p>The way this works is you open up an interactive python session, ideally in
a virtualenv. Set the variable values at the top. Run the two functions as
needed. In another window/pane, edit the JSON files.</p>
<p>Another option is to use <code>jq</code> with interactive MySQL commands. I'm not covering
that in this post, though.</p>
<h1>Environment setup</h1>
<p>This is best done in a venv if <code>mysql-connector</code> doesn't need to be installed
on the base system.</p>
<pre><code class="language-bash">pip install mysql-connector
</code></pre>
<h1>The code</h1>
<p>It's not a bad idea to write this into a parameterized CLI or whatever.
I just run this in an interactive python shell in a dedicated window/pane.</p>
<pre><code class="language-python">import json
import mysql.connector as mariadb

# Fill in the blanks
db_host= ''  # IP address
db_port = 3306
db_username = ''
db_password = ''
db_name = ''
table_name = ''
column_name = ''
where = ''  # (optional) where = 'WHERE `x` == 1'


def save_to_file(filename):
    """Read the json data from the database, save to filename"""
    conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
    cursor = conn.cursor()
    read_query = 'SELECT {} from {}{}'.format(column_name, table_name, where)
    cursor.execute(read_query)
    data = cursor.fetchone()[0]
    conn.close()
    jdata = json.dumps(json.loads(data), indent=4, sort_keys=True)
    with open(filename, 'w+') as out_file:
        out_file.write(jdata)
    print('done')


def update_from_file(filename):
    """Read data from filename, write to database"""
    with open(filename) as json_file:
        jdata = json.load(json_file)
    data = json.dumps(jdata)
    conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
    cursor = conn.cursor()
    update_query = "UPDATE {} SET {} = '{}'{}".format(table_name, column_name,
                                                      data, where)
    cursor.execute(update_query)
    conn.commit()
    conn.close()
    print('done')
</code></pre>
<h1>Usage</h1>
<p>From an interactive python terminal, paste the above script in.</p>
<p>Then to save the file, run something like:</p>
<pre><code class="language-python">save_to_file(filename='/home/kyle/sqldata.json')
</code></pre>
<p>Then you can update the file using an editor <code>vi /home/kyle/sqldata.json</code>.</p>
<p>Back in the Python terminal, you can then push the data back up.</p>
<pre><code class="language-python">update_from_file(filename='/home/kyle/sqldata.json')
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page contains python-related mini-guides and minor posts.</p>
<hr>
<h3>Table of Contents</h3>
<ul>
<li><a href="#pylint-ignore-comments">Pylint Ignore Comments</a>
<ul>
<li><a href="#turn-off-linting-for-now">Turn off Linting for now</a></li>
</ul>
</li>
<li><a href="#get-an-object-from-list-of-objects-where-the-object-property-has-value-x">Get an object from list of objects where the object property has value x</a>
<ul>
<li><a href="#a-subset-list-is-expected-list-comprehension">A subset list is expected: list comprehension</a></li>
<li><a href="#get-one-match-next-w-generator-expression">Get one match: next() w/ generator expression</a></li>
</ul>
</li>
<li><a href="#iterate-through-dictionary">Iterate through dictionary</a></li>
<li><a href="#print-json-nicely-in-ascii-no-unicode-u-prefix">Print JSON nicely in ASCII (no unicode 'u' prefix)</a></li>
<li><a href="#replace-a-string-in-a-file">Replace a string in a file</a></li>
<li><a href="#get-environment-variable">Get environment variable</a></li>
<li><a href="#use-jinja2-for-templating">Use Jinja2 for templating</a>
<ul>
<li><a href="#jinja2-templating-with-files">Jinja2 templating with files</a></li>
</ul>
</li>
<li><a href="#execute-sql-query-against-mysql-or-mariadb">Execute SQL query against MySQL or MariaDB</a></li>
<li><a href="#read-from-std-in-bash-pipe">Read from std in (bash pipe)</a></li>
<li><a href="#custom-exceptions">Custom Exceptions</a></li>
<li><a href="#cant-run-pip-after-updating">Can't run pip after updating</a></li>
</ul>
<h1>Pylint Ignore Comments</h1>
<p>I can never remember the syntax to ignore certain Pylint warnings. Yeah you can
turn them off everywhere but usually I just want to ignore a certain occurrence.</p>
<pre><code class="language-python"># - This file is a mess and I don't want pylint here:
# pylint: disable-all

# - troublesome import not being found:
# pylint: disable=import-error

# - redefined-builtin (W0622):
# pylint: disable=W0622

# - too-many-public-methods (R0904):
# pylint: disable=R0903

# - too-many-arguments (R0913):
# pylint: disable-msg=R0913

# - too-many-locals (R0914)
# pylint: disable=R0914

# - Invalid constant name:
# pylint: disable=C0103

# - Catching too general exception:
# pylint: disable=W0703

# - Using python3 compatible print statement makes python2 linter mad
# pylint: disable=superfluous-parens

# - Ignore too-long line (this needs both the pylint command and the trailing no-qa)
# pylint: disable=line-too-long
binaryValue = b'QVdTOmV5SndZWGxzYjJGa0lqb2lVSFkwWXpKeFNEaHZXVkkxVmtwRmVXdFhXamhKUVRGcGNFWk1kelJKZEV0V1UwSk9'  # noqa: E50

# - Wrong Import Position (Required when putting python version check up top)
# flake8: noqa=402
# pylint: disable=wrong-import-position
</code></pre>
<h2>Turn off Linting for now</h2>
<p>In VIM, run:</p>
<pre><code>:ALEToggle
</code></pre>
<h1>Get an object from list of objects where the object property has value x</h1>
<h2>A subset list is expected: list comprehension</h2>
<p>See <a href="https://docs.python.org/3/tutorial/datastructures.html">Section 5.1.3 - List comprehension</a></p>
<pre><code class="language-python"># This will return a list of objects whith a value
[obj for obj in haystack_list if obj.value == search]

# Example of list comprehension to get the even numbers in a list
nums = [1, 2, 3, 4, 5, 6, 7, 8]
even_nums = [num for num in nums if num % 2 == 0]
</code></pre>
<h2>Get one match: next() w/ generator expression</h2>
<p>The <code>next()</code> method will return the first element found in an iterable.</p>
<p>See <a href="https://www.python.org/dev/peps/pep-0289/">PEP 289 - Generator expressions</a></p>
<pre><code class="language-python"># Raises a StopIteration exception if no match is found
needle = next((obj for obj in haystack_list if obj.value == value))

# To return a value instead of raising an exception, pass a second argument
needle = next((obj for obj in haystack_list if obj.value == search), None)

# Example of what Next is really doing. This returns 1.
next(iter([1,2,3]))
</code></pre>
<hr>
<h1>Iterate through dictionary</h1>
<pre><code class="language-python">ex_dict = {'a': 1, 'b': 2, 'c': 3}
for key in ex_dict:
    print(key)
    print(ext_dict[key]) # value
</code></pre>
<p>This works because <code>ex_dict.keys() == list(ex_dict)</code></p>
<hr>
<h1>Print JSON nicely in ASCII (no unicode 'u' prefix)</h1>
<p>This is like using <a href="https://docs.python.org/3/library/pprint.html">pprint</a> on
the JSON object but without the unicode prefixes.</p>
<pre><code class="language-python">data = ... # the json string
jdata = json.dumps(json.loads(data),  indent=4, sort_keys=True)
print(jdata)
</code></pre>
<hr>
<h1>Replace a string in a file</h1>
<p>Though honestly, <a href="https://www.geeksforgeeks.org/sed-command-in-linux-unix-with-examples/">sed</a>
is usually a better option. Or if there are a lot of values, I like to use a
jinja template.</p>
<pre><code class="language-python">file_path = '/home/user/example_file'
find_str = 'REPLACE_THIS'
replace_str = 'WITH_THIS'

# Open the file to read the value
with open(file_path, 'r') as read_file:
    file_content = read_file.read()

# Replace the string
file_content = file_content.replace(find_str, replace_str)

# Write the file
with open(file_path, 'w') as write_file:
    write_file.write(replaced_content)
</code></pre>
<hr>
<h1>Get environment variable</h1>
<p>I mostly use this with <a href="https://docs.docker.com/engine/reference/builder/">Dockerfile ENV values</a>.</p>
<pre><code class="language-python">import os

environment_var_name = 'MY_ENV_VAR'
environment_var_vlue os.environ[environment_var_name]
</code></pre>
<hr>
<h1>Use Jinja2 for templating</h1>
<p>I use this for config files, like <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_templating.html">how Ansible does it</a>.</p>
<pre><code class="language-python">from jinja2 import Template

replacements = {'USERNAME': 'kyle', 'PASSWORD': 'example'}
template_text = 'connection = mysql://{{USERNAME}}:{{PASSWORD}}@127.0.0.1'
template = Template(template_text)
replaced_text = template.render(**replacements)
</code></pre>
<h2>Jinja2 templating with files</h2>
<p>Same idea as above but it reads a .j2 file as input and writes it to a config
file. Can combine this nicely with environment variables to write config files
as part of a Docker start script.</p>
<pre><code class="language-python">from jinja2 import Template

def apply_template(jinja2_file, output_file, replacements):
    """Replace the replacements values in jinja2_file, write to output_file"""
    with open(jinja2_file, 'r') as j2_file:
        j2_text = j2_file.read()
    template = Template(j2_text)
    replaced_text = template.render(**replacements)
    with open(output_file, 'w+') as write_file:
        write_file.write(replaced_text)

</code></pre>
<hr>
<h1>Execute SQL query against MySQL or MariaDB</h1>
<pre><code class="language-python">import mysql.connector as mariadb

conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
cursor = conn.cursor()
cursor.execute(sql_query)
conn.commit() # if it was a write action
conn.close()
</code></pre>
<hr>
<h1>Read from std in (bash pipe)</h1>
<p>For instance I have a file named x that has a big string where I want to
replace commas with spaces. Sed works but I can never remember the syntax.</p>
<pre><code class="language-python">cat x | python -c "import sys; print(sys.stdin.read())"

# Example of using python instead of sed to replace , with spaces
cat x | python -c "import sys; print(sys.stdin.read().replace(',',' '))"

# You can use this in fun ways like combining it with bash loops
cat x | python -c \
  "import sys; print(sys.stdin.read().replace(',','\n'))" | while read line; do
  echo $line
done
</code></pre>
<hr>
<h1>Custom Exceptions</h1>
<pre><code class="language-python">class MyCustomException(Error):
    pass
</code></pre>
<hr>
<h1>Can't run pip after updating</h1>
<p>So you ran into a problem because ubuntu ships an ancient version of pip, and
some dependency package like <code>subprocess32</code> is breaking your install. A forum
post says you should update pip.</p>
<pre><code class="language-bash">pip install -U pip
</code></pre>
<p>Except now pip won't run at all.</p>
<pre><code class="language-bash">pip install python-openstackclient
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in &#x3C;module>
    from pip import main
ImportError: cannot import name main
</code></pre>
<p>I think maybe the pip executable file moved after the update.
To fix it:</p>
<pre><code class="language-bash">hash -r pip
</code></pre>
<p>Now pip works again.</p>
<p>As an aside, if <code>subprocess32</code> really was your problem, updating pip didn't
help. For some reason you have to use apt to install <code>python-subprocess32</code>.</p>


<p>This guide covers how I set up a dedicated modded Minecraft Feed The Beast
server on spare laptop so my wife and I could play together.</p>
<p><strong>NOTE</strong>: Ubuntu 18.04 doesn't work at all. It uses a newer version of Java
which doesn't work with Minecraft's launcher.</p>
<hr>
<h1>Server Setup</h1>
<h2>Make the VM</h2>
<p>I'm just using a laptop with Windows 8.0 and Hyper-V for this. It has 16GB RAM,
4 Cores + Hyper-threading, and a 1TB SSD.</p>
<p>Here are the server VM specs I used:</p>
<ul>
<li>Operating System: Ubuntu 16.04 Server</li>
<li>RAM: 12GB</li>
<li>Disk: 200GB (Thin provisioned)</li>
</ul>
<p>Also, set a static IP address.</p>
<h2>Download the Modded Server Code</h2>
<p>Open your browser to <a href="https://www.curseforge.com/minecraft/modpacks/ftb-ultimate-reloaded">the mod site</a>
and find the link to the <a href="https://www.feed-the-beast.com/projects/ftb-ultimate-reloaded/files">server files</a>.</p>
<p>You can download the server files from your browser and SCP it over or download
them to the server directly with wget:</p>
<pre><code class="language-bash">mkdir -p ~/ftb
cd ~/ftb
wget https://media.forgecdn.net/files/2778/970/FTBUltimateReloadedServer_1.9.0.zip

</code></pre>
<p>Unzip the server file archive</p>
<pre><code class="language-bash">apt-get update
apt-get install -y unzip
unzip FTBUltimateReloadedServer_1.9.0.zip
</code></pre>
<h2>Configure the Server</h2>
<p><code>vi settings.sh</code></p>
<pre><code class="language-bash"># Set the RAM to 10GB
export MAX_RAM="10240M"
</code></pre>
<h2>Install Java JRE</h2>
<pre><code class="language-bash">apt-get install -y default-jre
</code></pre>
<h2>Launch the Server</h2>
<p>I like to console into the server and start a Tmux session, then SSH to it
and join that session to keep the server running. You can also just make a
systemd unit if you like.</p>
<p>From the <code>~/ftb</code> folder start the server interactively:</p>
<pre><code class="language-bash">bash ~/ftb/ServerStart.sh
</code></pre>
<hr>
<h1>Client Setup</h1>
<h2>Suggested Option: FTB Launcher</h2>
<p>Download the official FTB launcher from <a href="https://www.feed-the-beast.com/">here</a>.</p>
<p>It's really intuitive, no instructions are needed.</p>
<h2>Alternate Option 1: MultiMC</h2>
<p>This option has my favorite UI, but it seems to run into Java problems on
Windows sometimes.</p>
<p>MultiMC works offline, and is strictly about launching Minecraft.
It has some nice features like a window that seems to <code>tail -f</code> the Minecraft
log. It can also set different resource usage values per mod.</p>
<ol>
<li>Download <a href="https://multimc.org/">Multi-MC</a>, install, and launch it.</li>
<li>Click Add Instance</li>
<li>Choose FTB Legacy > FTB Ultimate Reloaded. OK.</li>
<li>Wait for the download to finish</li>
<li>Double-click the new icon with the mod name</li>
<li>If prompted to go to the account manager, say yes</li>
<li>If not in the settings window, open it. Go to Java. Increase the RAM.
It looks like 5120MB works well, with 512 for PermGen.</li>
<li>Close the settings.</li>
<li>Double-click on the installed mod to launch it</li>
</ol>
<h2>Alternative Option 2: Twitch Client</h2>
<p><strong>Note</strong>: The twitch launcher is a pain. It refuses to go offline and shows you
tons of stuff not related to modding Minecraft.
Unless you already use Twitch for other games or enjoy watching streamers,
I suggest you try MultiMC first.</p>
<p><strong>Update</strong>: This is pretty much discontinued now, definitely avoid it.</p>
<p>From a Windows client OS, go to <a href="https://www.twitch.tv/downloads">twitch's download site</a>
and download the <strong>twitch app for windows</strong>.</p>
<p>Run the installer. You might need to make a Twitch account if you don't have
one, which is a pain but they bought Curse so there's no real better way to
do this.</p>
<ol>
<li>Log in. If you don't have an account, try MultiMC, then make one if it
doesn't work for you.</li>
<li>At the top of the screen, go to Mods.</li>
<li>Click on Minecraft. If its not installed yet, its probably greyed out.</li>
<li>Click Install.</li>
<li>When the install is done you can go to "Browse FTB Modpacks" up top.</li>
<li>Click FTB Ultimate Reloaded</li>
<li>Click Install</li>
</ol>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to make a Docker registry on AWS Elastic Container
Registry (ECR).
To secure the registry, least-privilege roles are created and assigned to
service account users in the AWS AIM tool</p>
<hr>
<h3>Table of contents</h3>
<ul>
<li><a href="#create-an-aws-ecr-repository">Create an AWS ECR Repository</a></li>
<li><a href="#create-an-aws-iam-user-for-the-registry">Create an AWS IAM User For the Registry</a>
<ul>
<li><a href="#create-groups-for-ecr-access">Create Groups for ECR Access</a>
<ul>
<li><a href="#make-a-group-with-write-access">Make a group with write access</a></li>
<li><a href="#make-a-read-only-group">Make a read-only group</a></li>
</ul>
</li>
<li><a href="#create-users-for-ecr-access">Create Users for ECR Access</a></li>
</ul>
</li>
<li><a href="#connect-docker-to-the-new--registry">Connect Docker to the new  Registry</a>
<ul>
<li><a href="#install-the-aws-cli">Install the AWS CLI</a></li>
<li><a href="#configure-the-aws-cli-user">Configure the AWS CLI USER</a></li>
<li><a href="#get-the-docker-login-command">Get the docker login command</a></li>
</ul>
</li>
<li><a href="#push-images-to-the-ecr-registry">Push Images to the ECR Registry</a></li>
</ul>
<h1>Create an AWS ECR Repository</h1>
<p>Navigate to the
<a href="https://ca-central-1.console.aws.amazon.com/ecr/repositories">AWS ECR Repositories page</a>
and click Create a repository.</p>
<p>Fill in the form to name your repository.</p>
<h1>Create an AWS IAM User For the Registry</h1>
<p>Its a good idea to have user accounts with write access, and others with
read-only access. This way you can push to the registry with your privileged
user but you never need to expose those credentials outside your development
or CI/CD environment.</p>
<h2>Create Groups for ECR Access</h2>
<p>First, review the AWS Group access levels <a href="https://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr_managed_policies.html">here</a>.</p>
<h3>Make a group with write access</h3>
<p>Users in this group can push to existing registries but they can't use the CLI
to make new ones.</p>
<ol>
<li>Go to the <a href="https://console.aws.amazon.com/iam">AWS IAM page</a></li>
<li>Click Groups on the sidebar</li>
<li>Name the group, ex: <code>ecr-write</code></li>
<li>Attach Policy: <code>AmazonEC2ContainerRegistryPowerUser</code>, next</li>
<li>Create Group</li>
</ol>
<h3>Make a read-only group</h3>
<p>Its the same procedure as above, but use the read-only policy instead:
<code>AmazonEC2ContainerRegistryReadOnly</code>.</p>
<p>Consider also making an admin group with the
<code>AmazonEC2ContainerRegistryFullAccess</code> permission.</p>
<h2>Create Users for ECR Access</h2>
<p>Make two users, one for pushing and one for pulling. You can share the user
will read access to anyone who needs to consume the images.</p>
<p>To make a user:</p>
<ol>
<li>Go to the <a href="https://console.aws.amazon.com/iam">AWS IAM page</a></li>
<li>Click Users on the sidebar</li>
<li>Add User</li>
<li>Enter user name, check Programmatic Access, next</li>
<li>Choose from the groups you just created to grant read/write access, next</li>
<li>Skip the tags, next</li>
<li>Create User. Be sure to save the secret access key in your password manager.</li>
</ol>
<h1>Connect Docker to the new  Registry</h1>
<h2>Install the AWS CLI</h2>
<p>The one from apt is really old and doesn't work. Use pip.</p>
<pre><code class="language-bash">apt-get install -y python python-pip
pip install aws awscli
</code></pre>
<p>Test it out</p>
<pre><code class="language-bash">python -m awscli help
</code></pre>
<h2>Configure the AWS CLI USER</h2>
<p>When prompted, enter your key, secret key, region (ca-central-1), and output
format (none):</p>
<pre><code class="language-bash">aws configure
</code></pre>
<h2>Get the docker login command</h2>
<p>Use the AWS CLI to fetch and execute the docker login command. You can also
export this command to a file to run it, but there's no point since the token
gets invalidated after 12 hours.</p>
<pre><code class="language-bash">$(aws ecr get-login --no-include-email --region ca-central-1)
</code></pre>
<hr>
<h1>Push Images to the ECR Registry</h1>
<p>If you click on the Push Commands button in the ECR registry page you'll get
examples programmatically generated for your registry.</p>
<p>Pushing is kind of unintuitive. Unlike a normal <code>registry:2</code> docker registry,
ECR won't automatically create your repositories for you. You have to manually
create each one. You can use the web UI, or log in as an admin user and run the
following:</p>
<pre><code class="language-bash">aws ecr create-repository --repository-name kolla/ubuntu-source-base
</code></pre>
<p>For scripting, this is a handy way of creating the repo if it doesn't exist:</p>
<pre><code class="language-bash">aws ecr describe-repositories --repository-names $repo_name \
  || aws ecr create-repository --repository-name $repo_name
</code></pre>
<p>Then you can push.</p>
<pre><code class="language-bash">docker push $myId.dkr.ecr.ca-central-1.amazonaws.com/$repo:$tag
</code></pre>
<p>If you first create the repository, you get an error like this: <code>name unknown: The repository with name 'kolla/ubuntu-source-base' does not exist in the registry with id '...'</code></p>


Kyle Pericak

"It works in my environment"

Categories

Tags