<hr>
<p><strong>Note</strong>: There are a lot of ways to do this, I'm only documenting the approach
I chose. For instance, I'm using the master branch and <code>docs/</code>, but you can use
another branch too.</p>
<hr>
<h1>Initial Setup</h1>
<h2>CNAME file</h2>
<p>Pick a domain name. When I was writing this post, I was setting up the GitHub
Pages site for <a href="http://bwdt.breqwatr.com">Breqwatr Deployment Tool</a>, so I used
<code>bwdt.breqwatr.com</code>.</p>
<p>Create a file in your project's master branch named <code>docs/CNAME</code>, and put your
desired FQDN in that file. Push it to master.</p>
<h2>CNAME DNS Entry</h2>
<p>I'm using CloudFlare, so I logged in there and made a new CNAME entry pointing
<code>bwdt.breqwatr.com</code> to <code>breqwatr.github.io</code>.</p>
<h2>Repository Setup</h2>
<p>As the repository owner, open the repo up in GitHub and go to the Settings
page. Look for the GitHub Pages option.</p>
<ul>
<li>Set your source to <code>master branch /docs folder</code>.</li>
<li>Choose a theme</li>
<li>Confirm custom domain in the "custom domain" field</li>
</ul>
<hr>
<h1>Write the config file</h1>
<p>Pull the changes GitHub made to your registry, then edit <code>docs/_config.yml</code>.</p>
<p>Give your page a title and description.</p>
<pre><code class="language-yaml">theme: jekyll-theme-slate
title: Breqwatr Deployment Tool
description: A private cloud deployment and management toolkit
</code></pre>
<hr>
<h1>Write Content</h1>
<p>Create a file named <code>docs/index.md</code> and write your page content here.</p>
<p>GitHub Pages will link this <code>index.md</code> file to the <code>/</code> path of your chosen
domain.</p>
<p>The content is written in Markdown, which is really intuitive and carries over
nicely from any <code>README.md</code> files you'll have written before.</p>
<h2>Sub-pages</h2>
<p>GitHub Pages supports breaking your content up into more than one static page.
Create a new content file such as <code>installation.md</code>, and GitHub Pages will
host it with a slug of <code>/installation</code>.</p>
<p>From the index file, link to your new secondary file using Markdown. Here's an
example of linking to an installation guide from a table of contents list.</p>
<pre><code class="language-text">Table of contents

- [Installation](/installation.html)
</code></pre>


<p>This post covers some of the ways you can use Ceph from Python</p>
<p>The two libraries used here:</p>
<ul>
<li><a href="https://docs.ceph.com/docs/master/rados/api/python/">rados</a></li>
<li><a href="https://docs.ceph.com/docs/giant/rbd/librbdpy/">rbd</a></li>
</ul>
<hr>
<h1>Setup</h1>
<h2>Deploy ceph.conf</h2>
<p>Grab the ceph.conf file from your cluster or make a new valid one and place
it on the filesystem of the server which will execute the Python code.</p>
<p>Also copy the keyring over. Check <a href="https://docs.ceph.com/docs/master/rados/api/python/">here</a>
if you want to use anything other than the admin keyring.</p>
<h2>Install Dependencies</h2>
<p>Ceph doesn't distribute a pure python library. Instead it seems to rely on some
C stuff that needs to be installed, so you need to use apt instead of pip.
This is a pain, since so far as I can tell you can't isolate these dependencies
inside a virtual environment.</p>
<pre><code class="language-bash">apt-get install python-rados
</code></pre>
<hr>
<h1>Examples</h1>
<p>Be sure that the rados and rbd libraries, and <code>ceph.conf</code> are deployed first.</p>
<h2>Instantiate rados and rbd instances</h2>
<p>The <code>cluster</code> and <code>rbd_inst</code> variables will be used through these examples.
The <code>rbd_inst</code> methods will expect <code>ioctx</code> as their first argument.</p>
<pre><code class="language-python">import rbd
import rados

cluster = rados.Rados(conffile='/etc/ceph/ceph.conf')
cluster.connect()

pool_name = 'volumes'
ioctx = cluster.open_ioctx(pool_name)
rbd_inst = rbd.RBD()
</code></pre>
<h1>Usage examples</h1>
<pre><code class="language-python"># Get cluster stats
# Returns dict with kb, num_objects, kb_avail, and kb_used
cluster.get_cluster_stats()

# Get cluster health by issuing a command to ceph's monitor
cmd = {"prefix":"status", "format":"json"}
## mon_command returns a tuple of (return code, data/buffer, errors)
command_result = cluster.mon_command(json.dumps(cmd), b'', timeout=5)
status = json.loads(command_result[1])

# Get Pool usage stats
# Returns num_objects_degraded, num_kb, num_wr, num_rd, num_objects_unfound,
#   num_objects_missing_on_primary, num_object_clones, num_objects,
#   num_object_copies, num_bytes, num_rd_kb, num_wr_kb
ioctx.get_stats()

# List pools
cluster.list_pools()

# Create pool
pool_name = 'test_pool'
cluster.create_pool(pool_name)

# Delete pool
cluster.delete_pool(pool_name)


# List rados block devices
rbd_inst.list(ioctx)

# Create block device
rbd_name = 'sample-device'
size_gb = 10
rbd_inst.create(ioctx, rbd_name, size_gb)

# Query a block device (image)
image = rbd.Image(ioctx, rbd_name)

# Resize image
new_size=20
image.resize(new_size)

# Create a snapshot
snap_name = 'sample-snap'
image.create_snap(snap_name)

# List snaps of image
image.list_snaps()

# Revert to snapshot
snap_name = 'sample-snap'
image.rollback_to_snap(snap_name)

# Delete a snapshot
image.remove_snap(snap_name)

# Close an open image (when you're done working with it)
image.close()

# Delete an image/volume. Must not have snapshots. Must be closed.
rbd_inst.remove(ioctx, rbd_name)
</code></pre>


<h3>Table of Contents</h3>
<ul>
<li><a href="#create-ci-token">Create CI token</a></li>
<li><a href="#add-token-as-circleci-env-var">Add token as CircleCI env var</a></li>
<li><a href="#write-circleciconfigyml">Write .circleci/config.yml</a>
<ul>
<li><a href="#example-file">Example File</a></li>
<li><a href="#locally-validating-your-config">Locally validating your config</a></li>
</ul>
</li>
</ul>
<h1>Create CI token</h1>
<p>This will do the usual browser-based login system, so if you're using a VM then
be sure your lolcahost:9005 goes to the server's :9005.</p>
<pre><code class="language-bash">firebase login:ci
</code></pre>
<h1>Add token as CircleCI env var</h1>
<ol>
<li>Log into CircleCI</li>
<li>Choose your GitHub project</li>
<li>Use the Hello World build template</li>
<li>Assign that token as <code>$FIREBASE_DEPLOY_TOKEN</code> in the project's environment
variable settings.
<ol>
<li>Navigate to your project in CircleCI</li>
<li>Click the gear on the top-right to open this project's settings</li>
<li>Open environment variables page</li>
<li>Add Variable. Name: <code>FIREBASE_DEPLOY_TOKEN</code>, value is your token.</li>
<li>Add another variable for <code>FIREBASE_PROJECT</code> and set it to your Firebase
project's name.</li>
</ol>
</li>
</ol>
<h1>Write .circleci/config.yml</h1>
<p>I found <a href="https://medium.com/static-void-academy/easy-peasy-ci-cd-w-circleci-282bc85ddcf5">this guide</a>
useful.</p>
<h2>Example File</h2>
<pre><code class="language-yaml">version: 2
jobs:
  build:
    docker:
      # Using latest can cause failures, but I hope it will help me keep up
      - image: circleci/node:latest
    steps:
      - checkout
      - run: npm install --prefix=./firebase-deploy firebase-tools
      - run: >
          ./firebase-deploy/node_modules/.bin/firebase deploy
          --project=$FIREBASE_PROJECT
          --token=$FIREBASE_DEPLOY_TOKEN
</code></pre>
<h2>Locally validating your config</h2>
<p>You can use CircleCI's local-ci. Note that this was tested on a
<a href="https://github.com/kylep/dev-vm">dev Ubuntu server</a>.</p>
<p><strong>Install</strong> - <a href="https://circleci.com/docs/2.0/local-cli/#installation">official docs</a></p>
<pre><code class="language-bash">snap install docker circleci
snap connect circleci:docker docker
</code></pre>
<p>Test your configuration file. Navigate to your project directory first.</p>
<pre><code class="language-bash">circleci config validate
# or
circleci config validate &#x3C;config file path>
</code></pre>


<h1>Create a Firebase Project</h1>
<p>Open the <a href="https://console.firebase.google.com/">Firebase console</a> and follow
the wizard. It's pretty straightforward. I bound my project to a Google Cloud
Platform project I was already using and used the Blaze plan.</p>
<p>The wizard is intuitive, I won't cover its steps here.</p>
<p>Navigate into your project and "Add Firebase to your web app". Once that's done
you can use the CLI to deploy the app.</p>
<hr>
<h1>Install Firebase CLI</h1>
<p>Installing Firebase CLI requires the <code>npm</code> command from NodeJS.
If you want to configure your dev environment in Ansible, check out my
<a href="http://localhost:8000/nodejs-ansible.html">Installing Node.js with Ansible</a>
post.</p>
<pre><code class="language-bash"># Add the key for the nodesource Apt repository
wget -qO https://deb.nodesource.com/gpgkey/nodesource.gpg.key | sudo apt-key add -

# Add the nodesource apt repo
echo "deb https://deb.nodesource.com/node_13.x bionic main" > /etc/apt/sources.d/nodesource
apt-get update

# Install NodeJS to get the npm command
apt-get install -y nodejs

# Install firebase-tools
npm install firebase-tools
</code></pre>
<h2>Logging In</h2>
<p>If your CLI and browser are on the same system, just run <code>firebase login</code> and
ignore the rest of this section.</p>
<p>I haven't found a way to log into Firebase CLI that doesn't need a web browser.
The limitation was inconvenient since I develop inside an Ubuntu server VM.</p>
<p>The options I figure would work are:</p>
<ol>
<li>Using a local Vagrant VM, forward the port that the oauth site tries to
redirect to on localhost. This was the easiest for my local dev environment,
so it's what I did. You copy the URL from your CLI into your workstation's
browser, then have it redirect <code>http://localhost:9005/</code> to your VM. Here's
an example of how to forward the port in a Vagrantfile:
<code>config.vm.network :forwarded_port, host: 9005, guest: 9005</code></li>
<li>Install Chrome in Ubuntu server then launch it with
<a href="/x11-forwarding-ubuntu.html">X11 Forwarding</a></li>
<li>Create a reverse SSH tunnel from a port on your laptop to the development
server. <a href="/dial-home-device.html">Here's a post where I've done SSH port forwarding</a></li>
<li>Use the <code>firebase login:ci</code> command and the
<code>--token</code> argument. This option was my fallback, but you still need to
install <code>firebase-cli</code> on your actual workstation with a browser to use it.</li>
</ol>
<p>Once successfully authenticated your browser will show a message saying
<code>Woohoo! Firebase CLI Login Successful</code>.</p>
<hr>
<h1>Deploy Hello World</h1>
<p>Make an index file in a directory for this app</p>
<pre><code class="language-bash">mkdir app
cd app
mkdir public
echo "&#x3C;h1>Hello World!&#x3C;/h1>" > public/index.html
</code></pre>
<h2>firebase init</h2>
<p>Initialize the directory as an application. This will present you with an
intuitive wizard.</p>
<pre><code class="language-bash">firebase init

# When prompted,
#  choose only Hosting
#  keep your index file
#  decide if you'll use a single page app
</code></pre>
<h3>Single-page app setting</h3>
<p>Running <code>firebase init</code> generates a <code>firebase.json</code> file for your project.
In single-page mode, this file will rewrite all traffic to index.html. You can
edit the file to change this behaviour. The rewrite block in <code>"hosting":</code> looks
like this:</p>
<pre><code class="language-json">    "rewrites": [{
      "source": "**",
      "destination": "/index.html"
    }],
</code></pre>
<h2>firebase deploy</h2>
<p>Deploy the default app</p>
<pre><code class="language-bash">firebase deploy
</code></pre>
<p>The deploy command will print a Hosting URL. The Hello World site can be loaded
from there.</p>


<p>This post covers how I use Ansible to install Node.js. I googled around for a
solution to this and thought the solutions I found looked way too complicated.</p>
<hr>
<h3>Table of Contents</h3>
<ul>
<li><a href="#the-problem">The Problem</a>
<ul>
<li><a href="#what-about-nvm">What about NVM?</a></li>
<li><a href="#chris-leanodejs">chris-lea/node.js</a></li>
<li><a href="#curl-x--bash">curl <em>x</em> | bash</a></li>
</ul>
</li>
<li><a href="#the-solution">The Solution</a></li>
<li><a href="#ansible-example">Ansible Example</a>
<ul>
<li><a href="#add-nodesource-repository-to-apts-sourceslist">Add nodesource repository to Apt's sources.list</a></li>
<li><a href="#install-nodejs">Install Node.js</a></li>
<li><a href="#use-npm-normally">Use NPM normally</a></li>
</ul>
</li>
</ul>
<h1>The Problem</h1>
<p>In theory, installing Node.js should be as easy as running
<code>apt-get install nodejs</code>. The problem is that Node.js versions seem to move
pretty quickly and the one distributed by Cannonical in Ubuntu Bionic is
already too old to run some applications. In my case I was trying to install
<code>firebase-tools</code>, the CLI for Google Firebase, and it wouldn't run with Apt's
version of the package.</p>
<h2>What about NVM?</h2>
<p>NVM, or Node Version Manager, is a nice little application for installing
various versions of Node.js. It kind of reminds me of python virtualenv.
The problem with NVM is that it relies on setting environment variables and
defining the <code>nvm</code> function using the <code>.bashrc</code> file. Ansible doesn't read
that file when it runs.</p>
<p>In theory, the Ansible <code>npm</code> module supports accepting an executable path,
specifically to support NVM, but I had to do some ugly shell commands to
collect that path and even afterwards it wouldn't actually work. Specifically
running the <code>npm</code> binary deployed by NVM without scoping the <code>.bashrc</code> file
throws stack traces instead of installing your app.</p>
<h2>chris-lea/node.js</h2>
<p>A lot of the sites I found were adding the Apt PPA <code>'ppa:chris-lea/node.js'</code>.
I don't know what that is, but it doesn't sit well with me. Just seems like
something that wont age well, might be insecure, and is probably fragile.</p>
<p>To make matters worse, when I tried it out it didn't seem to support Bionic.</p>
<h2>curl <em>x</em> | bash</h2>
<p>Some examples I saw mentioned just curling various URLs and piping them to bash
to run a script.
Sometimes I'm OK with doing this, but that sort of installation method doesn't
really seem appropriate for an Ansible playbook. If nothing else it'd be hard
to make those tasks idempotent. Plus, we can do better.</p>
<hr>
<h1>The Solution</h1>
<p>If you look at the <a href="https://github.com/nodesource/distributions">nodesource/distributions GitHub page</a>,
they have some really nice documentation in their <code>README.md</code>. Under the
<strong>Manual Installation</strong> section you can see that they have a nice, stable
looking Apt repository URL. This worked great for me, and is trivial to
set up Ansible to work with.</p>
<hr>
<h1>Ansible Example</h1>
<h2>Add nodesource repository to Apt's sources.list</h2>
<pre><code class="language-yaml">- name: "Add nodejs apt key"
  apt_key:
    url: https://deb.nodesource.com/gpgkey/nodesource.gpg.key
    state: present

- name: "Add nodejs 13.x ppa for apt repo"
  apt_repository:
    repo: deb https://deb.nodesource.com/node_13.x bionic main
    update_cache: yes
</code></pre>
<h2>Install Node.js</h2>
<pre><code class="language-yaml">- name: "Install nodejs"
  apt:
    update_cache: yes
    name: nodejs
    state: present
</code></pre>
<h2>Use NPM normally</h2>
<p>With this version of Node.js, I was able to install and run the application.</p>
<pre><code class="language-yaml">- name: "Install NPM-distributed command-line tools"
  npm:
    global: yes
    name: "{{ item }}"
  with_items:
    - firebase-tools
</code></pre>


<p><strong>This post is linked to from the <a href="/blog-website.html">Blog Website Project</a></strong></p>
<hr>
<p>This static website is hosted using Google Cloud Storage, which is great except
that it doesn't support HTTPS. There's no really good excuse to not support
HTTPS on a website today.</p>
<p>To close that gap, CloudFlare can handle the HTTPS negotiation for you and it
takes no time at all to set up.</p>
<h1>Configure HTTPS in CloudFlare</h1>
<p>This guide assumes CloudFlare is already acting as a CDN for your site.</p>
<ol>
<li>Go to the CloudFlare DNS page and find the A record or CNAME for your FQDN</li>
<li>Ensure that the cloud icon is yellow and not grey, indicating that the CDN
is in use and not bypassing CloudFlare.</li>
<li>Click on the padlock labelled SSL/TLS</li>
<li>Go to the Overview tab</li>
<li>Select Flexible.... Basically that's it, you're good.
HTTPS is now running on your site. CloudFlare is a really great product!
There's more that can be done, though.</li>
<li>Go to the Edge Certificates tab</li>
<li>Set "Always Use HTTPS" to On</li>
</ol>
<p>In my testing the "Always Use HTTPS" redirect change didn't take effect
immediately. After about 15 minutes 301's were coming back as expected.</p>
<pre><code class="language-text">vagrant@dev-env:~/$ curl -i http://kyle.pericak.com
HTTP/1.1 301 Moved Permanently
Date: Tue, 03 Dec 2019 00:52:25 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Tue, 03 Dec 2019 01:52:25 GMT
Location: https://kyle.pericak.com/
Server: cloudflare
CF-RAY: 53f18ecdec10b677-YWG
</code></pre>
<p><a href="https://support.cloudflare.com/hc/en-us/articles/204144518">You can find more details and official documentation here.</a></p>


<p>As of Vim7, spell-checking is a built-in feature.</p>
<p>The spell checker will highlight incorrectly spelled words in red.
Light blue highlights indicate words which are only found in another dictionary
(such as "color" from <code>en_us</code>).</p>
<hr>
<h1>Commands</h1>
<h2>Enable/Disable</h2>
<p>Enable spellcheck</p>
<ul>
<li><code>set</code> for all buffers, <code>setlocal</code> for the current one</li>
<li><code>en_ca</code> will use Canadian spelling.</li>
</ul>
<pre><code class="language-text">:set spell spelllang=en_ca
:setlocal spell spelllang=en_ca
</code></pre>
<p>Or if you're not picky, the <code>spelllang</code> argument can be omitted. I'm not sure
what the default is, but I figure it's probable <code>en</code> which uses all the English
dictionaries together.</p>
<pre><code>:set spell
</code></pre>
<p>Disable spellcheck:</p>
<pre><code class="language-text">:set nospell
</code></pre>
<h2>Using Spell-Check</h2>
<p>Move to the next misspelled word</p>
<pre><code>]s
</code></pre>
<p>previous misspelled word</p>
<pre><code>[s
</code></pre>
<p>Highlight a misspelled word. Show possible replacements.
Press return with no input to go back.</p>
<pre><code>z=
</code></pre>
<p>Accept the spelling of a word:</p>
<pre><code class="language-text">zg
</code></pre>
<p>Mark a word's spelling as wrong</p>
<pre><code class="language-text">zw
</code></pre>


<p>Docker supports shared memory. The default is only 64MB though which is TINY.</p>
<p>The small size became a problem during a containerized OpenStack upgrade.</p>
<p>This will be a really short post because I haven't done much playing with the
shared memory yet, I just think its really neat.</p>
<hr>
<h1>Change the /dev/shm size on the system</h1>
<p>This lets you change the size of shared memory on a host.</p>
<pre><code class="language-bash">mount -o remount,size=1G /dev/shm
</code></pre>
<hr>
<h1>Update Docker Daemon's Default Shared Memory Size</h1>
<p>This will update the default across the whole Docker service.</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
  "default-shm-size": "512M"
}
</code></pre>
<hr>
<h1>Set a Containers Shared Memory Size</h1>
<p>Here's how to change it for a specific container at runtime.</p>
<pre><code class="language-bash">docker run -it --shm-size=512M
</code></pre>


<p><strong>This post is linked to from the <a href="/open-source-cli-project.html">FOSS Python App Project</a></strong></p>
<hr>
<p>At a high level, the steps to upload a package to PyPi are as follows:</p>
<ol>
<li>Write the project code</li>
<li>Create a release on GitHub and put it in the <code>download_url</code> of setup.py</li>
<li>Write a setup.py file to build and describe the project</li>
<li>Write a setup.cfg file pointing to the readme</li>
<li>create a source distribution</li>
</ol>
<h1>Write the project code</h1>
<p>I'm not going to cover how to build a python project in this post, but here's
the very small and simple project I used when learning this procedure:
<a href="https://github.com/kylep/jsc2f">GitHub: kylep/jsc2f</a>.</p>
<hr>
<h1>Create a Release on GitHub</h1>
<p>Log into your GitHub project and click on releases. If you can't find the link,
you can also just append <code>/releases</code> to the end of the URL to get there.</p>
<p>Click "Create a new release". Enter your tag and branch, along with a release
title and description.</p>
<p>Under the Assets section of the new release there's a (tar.gz) link to download
the source code. If you right click it and copy the URL, that's used next in
the setup.py file.</p>
<hr>
<h1>Write a setup.py file</h1>
<p>Here's mine. Check GitHub to see if I've changed it.
The <code>download_url</code> value comes from the GitHub release.</p>
<pre><code class="language-python">from setuptools import setup

# Without this you'll get no description on the pypi site
with open("README.md", "r") as fh:
    long_description = fh.read()

setup(
    name='jsc2f',
    packages=['jsc2f'], # Alternatively you can use find_packages()
    version='0.1', # match the github release you make later
    license='MIT', # match the LICENSE file in the project
    description='Saves a JSON fields SQLs cell to a file, or UPDATE it back',
    author='Kyle Pericak',
    author_email='kyle@pericak.com',
    url='kyle.pericak.com/jsc2f', # either your own page or github
    download_url='https://github.com/kylep/jsc2f/archive/v0.1.tar.gz',
    keywords=['SQL', 'JSON', 'file'],
    install_requires=[
        'click',
        'mysql-connector'
    ],
    entry_points='''
        [console_scripts]
        jsc2f=jsc2f.cli:cli
    ''',
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Information Technology',
        'Topic :: Database',
        'Programming Language :: Python :: 3',
        'License :: OSI Approved :: MIT License',
        'Natural Language :: English'

</code></pre>
<h2>Picking a topic:</h2>
<p>I used <a href="https://pypi.org/pypi?%3Aaction=list_classifiers">this classifier list</a></p>
<hr>
<h1>Write setup.cfg</h1>
<p>This goes in the project root next to setup.py.</p>
<p><code>vi setup.cfg</code></p>
<pre><code class="language-ini">[metadata]
description-file = README.md
</code></pre>
<h1>Creating a source distribution</h1>
<p>For this you use the python <a href="https://docs.python.org/2/distutils/sourcedist.html">sdist</a>
tool and <a href="https://pypi.org/project/twine/">twine</a>.</p>
<p>From the git project root:</p>
<pre><code class="language-bash">pip install twine

# This will create dist/
python setup.py sdist

# Twine uploads the dist/ files to Pypi
twine upload --skip-existing dist/*
# Enter your username and password for PyPI
</code></pre>


<p>Often I need to operate on a huge JSON string that's saved in a MaraiDB
database. Here's how I do that.</p>
<p>It needs to be said that raw SQL queries like this are generally a bad idea
compared to something like <a href="https://docs.sqlalchemy.org/en/13/orm/">SQLAlchemy's ORM</a>.
Having said that, this example is way smaller and more portable.</p>
<p>The way this works is you open up an interactive python session, ideally in
a virtualenv. Set the variable values at the top. Run the two functions as
needed. In another window/pane, edit the JSON files.</p>
<p>Another option is to use <code>jq</code> with interactive MySQL commands. I'm not covering
that in this post, though.</p>
<h1>Environment setup</h1>
<p>This is best done in a venv if <code>mysql-connector</code> doesn't need to be installed
on the base system.</p>
<pre><code class="language-bash">pip install mysql-connector
</code></pre>
<h1>The code</h1>
<p>It's not a bad idea to write this into a parameterized CLI or whatever.
I just run this in an interactive python shell in a dedicated window/pane.</p>
<pre><code class="language-python">import json
import mysql.connector as mariadb

# Fill in the blanks
db_host= ''  # IP address
db_port = 3306
db_username = ''
db_password = ''
db_name = ''
table_name = ''
column_name = ''
where = ''  # (optional) where = 'WHERE `x` == 1'


def save_to_file(filename):
    """Read the json data from the database, save to filename"""
    conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
    cursor = conn.cursor()
    read_query = 'SELECT {} from {}{}'.format(column_name, table_name, where)
    cursor.execute(read_query)
    data = cursor.fetchone()[0]
    conn.close()
    jdata = json.dumps(json.loads(data), indent=4, sort_keys=True)
    with open(filename, 'w+') as out_file:
        out_file.write(jdata)
    print('done')


def update_from_file(filename):
    """Read data from filename, write to database"""
    with open(filename) as json_file:
        jdata = json.load(json_file)
    data = json.dumps(jdata)
    conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
    cursor = conn.cursor()
    update_query = "UPDATE {} SET {} = '{}'{}".format(table_name, column_name,
                                                      data, where)
    cursor.execute(update_query)
    conn.commit()
    conn.close()
    print('done')
</code></pre>
<h1>Usage</h1>
<p>From an interactive python terminal, paste the above script in.</p>
<p>Then to save the file, run something like:</p>
<pre><code class="language-python">save_to_file(filename='/home/kyle/sqldata.json')
</code></pre>
<p>Then you can update the file using an editor <code>vi /home/kyle/sqldata.json</code>.</p>
<p>Back in the Python terminal, you can then push the data back up.</p>
<pre><code class="language-python">update_from_file(filename='/home/kyle/sqldata.json')
</code></pre>


<p><strong>This post is linked to from the <a href="/blog-website.html">Blog Website Project</a></strong></p>
<hr>
<p>Nobody wants to test in production.</p>
<p>This guide covers how to preview your Pelican content locally as you write it.
Every time you save a markdown file or update the Pelican theme, the entire
site will re-generate and be hosted by a local web-server for you to review.</p>
<h1>Install Dependencies</h1>
<ul>
<li><a href="https://docs.docker.com/install/">Docker</a></li>
<li><a href="https://git-scm.com/book/en/v2/Getting-Started-Installing-Git">Git</a></li>
<li>Text Editor: Your choice, I use <a href="https://github.com/vim/vim">Vim</a></li>
<li>Browser: Chrome has nice developer tools but anything works</li>
</ul>
<hr>
<h1>Get the Pelican Image</h1>
<p><strong>Option 1: Build your own</strong></p>
<p>This is the approach I suggest. You can see the guide I wrote explaining how to
build a Pelican image <a href="/docker-pelican-image.html">here</a>.</p>
<p><strong>Option 2: Use a pre-made image</strong>
I've shared my image for public use <a href="https://hub.docker.com/r/kpericak/pelican">here</a>.
If you plan on using it, you're best bet is to check how I use it on GitHub.</p>
<hr>
<h1>Launch a Persistent Pelican Container</h1>
<p>Once you've either pulled or built the image, it will be listed in your Docker
image list.</p>
<pre><code class="language-bash">docker image list
</code></pre>
<p>Set the following variables. Note the paths should be absolute (vs relative).</p>
<ul>
<li><code>$image_name</code>: docker image name</li>
<li><code>$conf_file</code>: path to <code>pelicanconf.py</code> configuration file</li>
<li><code>$content_dir</code>: path to <code>content/</code> directory, where markdown files reside</li>
<li><code>$output_dir</code>: path to <code>output/</code> directory, where generated files will land.</li>
<li><code>$listen_port</code>: Which port to listen on for the dev web-server</li>
</ul>
<p>Optionally, you can set a theme path too. If you omit this and its associated
volume mount then the theme built into the Pelican image will be used instead.
This enables theme development.</p>
<ul>
<li><code>$theme_path</code>: Path to the Pelican theme</li>
</ul>
<p>Start the Docker container in daemon mode (<code>-d</code>)  as persistent container.
By setting the run command to <code>tail -f /dev/null</code>, the container will never
stop itself and instead sit idle until told to do otherwise.</p>
<p>Mount the directories specified by those variables into the container.</p>
<p>Forward <code>$listen_port</code> to the workstation so the site can be opened in a local
browser.</p>
<pre><code class="language-bash">image_name=""
conf_file=""
content_dir=""
output_dir=""
theme_path=""
listen_port=8000

docker run -d \
  --name pelican \
  -v $conf_file:/pelicanconf.py \
  -v $conftent_dir:/content \
  -v $output_dir:/output \
  -v $theme_path:/theme" \
  -p 0.0.0.0:$listen_port:8000 \
  image_name \
  tail -f /dev/null
</code></pre>
<hr>
<h1>Launch Development Web-Server</h1>
<p>At this point you have a pelican container running, but it isn't doing
anything. The files are all mounted in, and it's prepared to forward its local
port 8000 to your specified workstation port, so all that's left is to start
Pelican's built-in development service.</p>
<p>This service will watch both the markdown files and the theme files for
changes, then reload the site files. It will also host the website on a
non-production web-server.</p>
<pre><code class="language-bash">docker exec -it pelican \
  pelican \
    -s /pelicanconf.py \
    --debug \
    --autoreload \
    --listen \
    --port 8000 \
    --bind 0.0.0.0 \
    /content \
    -o /output
</code></pre>


<h1>What is Pelican</h1>
<p><a href="https://github.com/getpelican/pelican">Pelican</a> is a free and open source
static website generator. It takes a collection of files
written in <a href="https://en.wikipedia.org/wiki/Markdown">Markdown</a> as its input,
applies a theme and configuration, then outputs a full static website.</p>
<p>A static website is a collection of HTML and CSS files that don't need any
database or back-end programming language. These websites can be hosted by any
web-server regardless of language support.</p>
<hr>
<h1>Creating a Pelican Project</h1>
<p>While not necessary, its a really good idea to make a git project to source
control your blog content.</p>
<p>The project will hold the configuration for Pelican, which defines the plugins,
variables for the style-sheet, and the site name. This project also holds all
the markdown files which make up the blog content.</p>
<h2>Project Layout</h2>
<p>The files should be placed in a structure as follows.</p>
<ul>
<li>Configuration goes in the <code>pelicanconf.py</code> file.</li>
<li>Blog posts go in <code>content/posts</code>, and can be sorted into subdirectories.
The nested directory structure has no impact on how the site is rendered.</li>
<li>Static pages go in <code>content/pages</code>. These are handled specially by the
templating engine.</li>
<li><code>content/images</code> stores image files</li>
<li><code>output/</code> doesn't need to exist here, but it's used by the <code>-o</code> (output)
argument in Pelican as the directory to store the generated website files.
This directory should be ignored by Git.</li>
</ul>
<pre><code class="language-text">.
+-- pelicanconf.py
+-- content/
|   +-- posts/
|       +-- first.md
|       +-- how-this-site-works/
|           +-- building-a-website.md
|   +-- pages/
|       +-- about.md
|   +-- images/
|       +-- pelican.png
+-- output/
</code></pre>
<hr>
<h1>Pelican &#x26; Markdown Syntax Reference</h1>
<h2>Post Metadata</h2>
<p>The top of each post starts with some metadata that Pelican uses to scaffold
the site.</p>
<pre><code class="language-text">title: Title of the post or page
summary: Detailed summary
slug: short-name-goes-in-the-url
category: whereThisWillBeSorted
tags: tag1, tag2
date: 2019-08-28
modified: 2019-08-28
status: published or draft
</code></pre>
<ul>
<li><code>title</code>: defines the string at the top of the article and in the index</li>
<li><code>summary</code>: used in the index pages. If not defined, a large and
somewhat ugly subset of the top part of your post will auto-populate this
field.</li>
<li><code>slug</code>: defines the URL from which the post page is accessible</li>
<li><code>category</code>: place this post into a category, accessible from the sidebar in
this particular theme.</li>
<li><code>tags</code>: a comma delimited list of indexed sub-categories. Can have many.</li>
<li><code>date</code>: When the page was authored</li>
<li><code>modified</code>: When the page was last changed</li>
<li><code>status</code>: either <code>published</code> or <code>draft</code>. Defines whether the page is ready to
be displayed to the public.</li>
</ul>
<p>You can also define custom attributes to use in your themes. I've modified a
theme to allow custom thumbnail files for the index and a blog image which look
like this:</p>
<pre><code class="language-text">image: pelican.png
thumbnail: pelican-thumb.png
</code></pre>
<h2>Headers</h2>
<pre><code class="language-text">Regular text
# Big header (h1)
## Smaller Header (h2)
### Even smaller header
</code></pre>
<h2>Horizontal Line</h2>
<pre><code class="language-text">---
</code></pre>
<h2>Table of Contents</h2>
<ul>
<li>
<ul>
<li><a href="#numbered-list">Numbered List</a></li>
<li><a href="#bullet-list">Bullet List</a></li>
<li><a href="#html">HTML</a></li>
<li><a href="#links">Links</a></li>
<li><a href="#images">Images</a></li>
</ul>
</li>
<li><a href="#preview-the-site">Preview the Site</a>
<ul>
<li><a href="#pull-the-pelican-image">Pull the Pelican Image</a></li>
<li><a href="#launch-the-pelican-container">Launch the Pelican container</a></li>
<li><a href="#launch-the-local-web-server-and-auto-rebuild-service">Launch the local web server and auto-rebuild service</a></li>
<li><a href="#browse-to-your-site">Browse to your site</a></li>
</ul>
</li>
<li><a href="#reference-links">Reference Links</a></li>
<li><a href="#next-up">Next Up</a></li>
</ul>
<h2>Numbered List</h2>
<p>The numbers will automatically increment.</p>
<pre><code class="language-text">1. First item
1. Second item
1. Third Item
</code></pre>
<h2>Bullet List</h2>
<pre><code class="language-text">- First Item
- Second Item
    - Nested List item with 4 spaces before the dash
- Third Item
</code></pre>
<h2>HTML</h2>
<p>Markdown files can also support raw HTML content, allowing you a lot of
flexibility.</p>
<pre><code class="language-html">&#x3C;table class="mytable>
  &#x3C;tr>&#x3C;th>Header&#x3C;/th>&#x3C;/tr>
  &#x3C;tr>&#x3C;td>Cell&#x3C;/td>&#x3C;/td>&#x3C;/tr>
&#x3C;/table>
</code></pre>
<h2>Links</h2>
<p>Linked text goes in square brackets the URL in round brackets.</p>
<pre><code class="language-text">[A web link](https://example.com)
</code></pre>
<h2>Images</h2>
<pre><code class="language-text">![Alt Text](/images/blog_workflow.png)
</code></pre>
<hr>
<h1>Preview the Site</h1>
<p>The content is ready to be converted into a website. Use a Pelican container to
generate the site. The pelican container can also act as a local web server to
preview changes before pushing them.</p>
<h2>Pull the Pelican Image</h2>
<p>Pull a pelican image from GCR or Docker Hub.
Mine is <code>gcr.io/kylepericak/pelican</code> on GCR and <code>kpericak/pelican</code> on Docker
Hub. The GCR image is private, since they charge me for the downloads.</p>
<pre><code class="language-bash">docker pull gcr.io/kylepericak/pelican
# or
docker pull kpericak/pelican
</code></pre>
<h2>Launch the Pelican container</h2>
<p>Run this from the project root. It will mount the content/ and output/
directories into a new container named "pelican". It will also forward the
port Pelican's web server listens on (8000) to the workstation's port 8888.</p>
<p>The Pelican auto-reload feature looks like maybe its multithreaded or something,
so running it as the root process of the docker container doesn't work.
Instead, launch the container with the <code>tail -f /dev/null</code> command so it just
idles.</p>
<pre><code class="language-bash">docker run -d \
  --name pelican \
  -v $(pwd)/content:/content \
  -v $(pwd)/output:/output \
  -p 8888:8000 \
  gcr.io/kylepericak/pelican \
  tail -f /dev/null
</code></pre>
<h2>Launch the local web server and auto-rebuild service</h2>
<p>This will build the site files, launch the web server, and watch for new
file changes so it can automatically rebuild.</p>
<p>The example <code>docker exec</code> command runs interactively. I like to run it in a
dedicated tmux pane to show if anything has gone wrong.</p>
<pre><code class="language-bash">docker exec -it pelican \
  pelican \
    -s /pelicanconf.py \
    --debug \
    --autoreload \
    --listen \
    /content \
    -o /output
</code></pre>
<h2>Browse to your site</h2>
<p><a href="http://localhost:8888">http://localhost:8888</a></p>
<hr>
<h1>Reference Links</h1>
<ul>
<li><a href="http://docs.getpelican.com/en/3.6.3/content.html">Official Pelican Syntax Reference</a></li>
</ul>
<hr>
<h1>Next Up</h1>
<p>Now that the content is written, its time to
<a href="/htsw-4-gcs-static-website.html">host the static website on Google Cloud Storage</a>.</p>


<p><strong>This post is linked to from the <a href="/blog-website.html">Blog Website Project</a></strong></p>
<hr>
<p>Pelican is a free and open source static website generator written in Python.
It accepts a theme, a config file, and a directory of markdown documents as
input and uses them to generate the HTML and CSS files that make up a static
website.</p>
<p>This post covers how to build a Docker image that contains the Pelican software
and the theme which I chose to use with it. This image is reusable and
<a href="https://hub.docker.com/r/kpericak/pelican">freely available on Docker Hub</a>.</p>
<h1>Dockerfile</h1>
<p>As with any Docker image, the most important piece is the Dockerfile.</p>
<p>You can see the current file I'm using <a href="https://github.com/kylep/pelican/blob/master/Dockerfile">here</a>.</p>
<p>In this Dockerfile, Pelican, its theme, and plugins are installed to the image.</p>
<pre><code class="language-text">FROM ubuntu:bionic
ADD requirements.txt ./
RUN apt-get update \
    &#x26;&#x26; apt-get install -y --no-install-recommends \
         git \
         python \
         python-pip \
         python-setuptools \
    &#x26;&#x26; rm -rf /var/lib/apt/lists/* \
    &#x26;&#x26; pip install -r requirements.txt \
    &#x26;&#x26; git clone https://github.com/kylep/voidy-bootstrap.git /theme \
    &#x26;&#x26; mkdir -p /tmp-plugins/ /plugins \
    &#x26;&#x26; git clone --recurse \
           https://github.com/getpelican/pelican-plugins.git /tmp-plugins \
    &#x26;&#x26; cp -r /tmp-plugins/series /plugins/ \
    &#x26;&#x26; rm -r /tmp-plugins
</code></pre>
<h3>FROM ubuntu:bionic</h3>
<p>This section specifies the Docker base image. The above docker file uses
<a href="https://hub.docker.com/_/ubuntu/">the official Docker Hub Ubuntu image</a>.</p>
<h3>ADD requirements.txt</h3>
<p>The requirements.txt file lists the python packages which Pelican depends upon.
Docker's <code>ADD</code> will copy that file into the image at the specified path.</p>
<p>The minimum packages required are:</p>
<pre><code class="language-text">pelican
markdown
</code></pre>
<h3>RUN ...</h3>
<p>The run task is basically a bash script all squished into one line. While its
harder to read with all the line continuation and <code>&#x26;&#x26;</code> symbols, using them
leads to Docker only building a single layer.</p>
<p><a href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/">See the best practices for writing docker images</a>.</p>
<p>Since this is an Ubuntu image, <code>apt</code> is the package manager. It installs Git
and some python packages. Git is required to pull the Pelican theme from
Github.</p>
<p>Only the minimum Python packages are installed by apt, since pip is the
preferred python package manager.</p>
<p><code>rm -rf /var/lib/apt/lists/*</code> removes the leftover apt artifacts, helping to
shrink the Docker image.</p>
<p>After the theme is pulled from GitHub, Pelican's open source plugins are also
pulled. The ones desired are copied to <code>/plugins</code>, then the (huge) checkout is
deleted to reduce image size.</p>
<hr>
<h1>Running the Docker Image</h1>
<p>Locally, you need to mount the content into the container. Something like this
will work:</p>
<pre><code class="language-bash">docker run \
  --name pelican \
  -v $(pwd):/workspace \
  gcr.io/kylepericak/pelican:latest \
  pelican -o /workspace/output -s /workspace/pelicanconf.py
</code></pre>
<p>The above command will load the <code>content/</code> and <code>pelicanconf.py</code> files into
Pelican, and write the static website to <code>output/</code>.</p>


Kyle Pericak

"It works in my environment"

Categories

Tags