<p>You've deployed a server in a customer environment and you don't have remote
network access configured. The customer is OK with you being able to remote
into their network, but its not practical to set you up with a VPN. Enter the
"Dial-Home Device".</p>
<h1 id="what-is-the-dial-home-device">What is the Dial-Home Device</h1>
<p>It's just an Ubuntu server that repeatedly tries to dial an internet connected
jump box in order to open a reverse session. As soon as the networking is set
up, it will dial home and open itself up for remote administration.</p>
<p>Also, <a href="https://stargate.fandom.com/wiki/Dial_Home_Device">this.</a></p>
<hr>
<h1 id="deploy-the-pivot-server">Deploy the Pivot Server</h1>
<h2 id="whats-a-pivot-server">What's a Pivot Server?</h2>
<p>This is an internet-connected server such as a VPS that the DHD will SSH to.</p>
<p>You'll SSH to it, then from there you can SSH through the DHD's reverse tunnel.</p>
<h2 id="steps-to-deploy">Steps to Deploy</h2>
<h3 id="launch-a-vps">Launch a VPS</h3>
<p>Using AWS, GCP, Azure, Linode, Digital Ocean, so on, launch an
internet-connected Ubuntu server. It needs to have a static, internet routed IP
address. Its not a bad idea to make a DNS entry for it too, so you can change
the IP down the road.</p>
<p>SSH to the server to confirm that it works.</p>
<h3 id="configure-remote-user">Configure Remote User</h3>
<p>These steps will create a user named <code>remoteuser</code></p>
<pre><code class="language-bash"># Create the user
useradd remoteuser

# Set a password for the user
passwd remoteuser

# Create the home &#x26; ssh directories for the user
mkdir -p /home/remoteuser/.ssh
chown -R remoteuser:remoteuser /home/remoteuser

# Disable shell access to the user, so it can't run commands on the pivot
usermod -s /bin/false remoteuser
</code></pre>
<hr>
<h1 id="deploy-the-dial-home-device">Deploy the Dial-Home Device</h1>
<p>Install Ubuntu Server on a physical server or VM at the remote location. This
server should have network connectivity to whatever resource you want to
manage, and outbound internet access (it can ping google).</p>
<h2 id="create-an-ssh-key-on-the-dhd">Create an SSH key on the DHD</h2>
<p>On the DHD server, you may already have SSH keys. Look in <code>~/.ssh</code>. If not,
run the following to create one. Don't use a passphrase, just hit enter when
prompted.</p>
<pre><code class="language-bash">ssh-keygen
</code></pre>
<p>Collect the public key for the next step:</p>
<pre><code class="language-bash">cat ~/.ssh/id_rsa.pub
</code></pre>
<p>For convenience, I like to copy the key to the root user too. Optional.</p>
<h2 id="authorize-the-ssh-public-key-on-the-pivot-serer">Authorize the SSH Public Key on the Pivot Serer</h2>
<p>Back on the Pivot server, edit the remote user's <code>authorized_keys</code> file to
grant this SSH key-pair access. Paste the SSH key at the end of the file</p>
<pre><code class="language-bash">vi /home/remoteuser/.ssh/authorized_keys
</code></pre>
<h2 id="test-a-remote-connection">Test a remote connection</h2>
<h3 id="initiate-the-connection">Initiate the connection</h3>
<p>From the DHD server, open the reverse tunnel manually to make sure it works.</p>
<ul>
<li>Replace the username as needed</li>
<li>The <code>$pivot</code> variable should be the IP or FQDN of the pivot server</li>
<li>Pick a random port in the range 1024-49151 for the listen port.</li>
</ul>
<pre><code class="language-bash">user="remoteuser"
pivot="support.example.com"
port=22222
ssh -fN -R 0.0.0.0:$port:127.0.0.1:22 $user@$pivot
</code></pre>
<h3 id="connect-through-the-pivot">Connect through the pivot</h3>
<p>From the pivot server, SSH to the DHD. This example assumes your username
is <code>myuser</code>.</p>
<pre><code class="language-bash">ssh -p 22222 myuser@127.0.0.1
</code></pre>
<p>Consider adding your public key to the <code>~/.ssh/authorized_keys</code> file now so
passwords aren't needed.</p>
<p>Now you can jump through the pivot into the DHD.</p>
<hr>
<h1 id="persist-the-connection">Persist the Connection</h1>
<h2 id="build-the-dial-home-command">Build the dial-home command</h2>
<p>From any directory, create the script:</p>
<p><code>vi dial-home</code></p>
<pre><code class="language-bash">#!/usr/bin/env bash

# Check if the reverse tunnel  is already open
port=22222
if [[ $(ps aux | grep ssh | grep $port) ]]; then
  # reverse tunnel is open
  exit 0
fi

# tunnel is not open, open it
user="remoteuser"
pivot="support.example.com"
ssh -fN -R 0.0.0.0:$port:127.0.0.1:22 $user@$pivot
</code></pre>
<p>Make the script executable</p>
<pre><code class="language-bash">chmod +x dial-home
</code></pre>
<p>Move the script to somewhere in the <code>$PATH</code></p>
<pre><code class="language-bash">sudo mv dial-home /usr/local/bin/
</code></pre>
<p>Now the <code>dial-home</code> command will open the tunnel.</p>
<h2 id="make-cron-dial-home-every-minute">Make cron dial-home every minute</h2>
<p>The connection will die for lots of reasons. Make it bring itself back online
automatically using cron.</p>
<p>From the DHD, make sure cron is running (by default, it is)</p>
<pre><code class="language-bash">systemctl status cron
</code></pre>
<p>Edit the crontab</p>
<p><code>sudo vi /etc/crontab</code></p>
<p>and insert this line, where <code>myuser</code> is your username on the DHD.
If you use root, you have to also give root the ssh private key.</p>
<pre><code class="language-text">* * * * *   myuser dial-home
</code></pre>
<p>Reload cron</p>
<pre><code class="language-bash">service cron reload
</code></pre>
<p>Then kill the session (using <code>ps aux</code> and <code>kill</code>) then watch it come back.</p>
<p>Done. Now the server is a dial-home device. It will constantly dial your pivot
server and re-open the session whenever its closed. To lock the server out,
just remove its public key from <code>remoteuser</code>'s <code>authorized_keys</code> file.</p>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page lists some of the errors that I've personally encountered,
and how I fixed them. I might also include errors I never fixed in case anyone
wants to talk about it.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#docker-login-fails">Docker Login Fails</a></li>
<li><a href="#runaway-memory-usage-in-ceph-osd-process">Runaway memory usage in Ceph OSD process</a></li>
<li><a href="#chrome-blocks-https127001-with-neterr_cert_revoked">Chrome blocks https://127.0.0.1 with NET::ERR_CERT_REVOKED</a></li>
<li><a href="#failed-command-aws-ecr-get-login">Failed command: aws ecr get-login</a></li>
<li><a href="#pure-storage-iscsi-with-kolla-openstack---fdisk-freezes">Pure Storage iSCSI with Kolla OpenStack - fdisk freezes</a></li>
<li><a href="#weird-chars-when-pasting-on-mac-os">Weird chars when pasting on Mac OS</a></li>
</ul>
<h1 id="docker-login-fails">Docker Login Fails</h1>
<pre><code>Error saving credentials: error storing credentials - err: exit status 1, out: `Cannot autolaunch D-Bus without X11 $DISPLAY`
</code></pre>
<p>I had this happen after running both Ceph-Ansible then Kolla-Ansible on an Ubuntu server, so it could be either of those that set up the conditions for this to happen. <a href="https://github.com/docker/cli/issues/1136">this solution</a> worked for me:</p>
<pre><code class="language-bash">apt-get install -y gnupg2 pass
</code></pre>
<hr>
<h1 id="runaway-memory-usage-in-ceph-osd-process">Runaway memory usage in Ceph OSD process</h1>
<p>It looked like a memory leak. Each OSD process was using around 30GB of RAM.</p>
<p>It turns out that Ceph-Ansible has the following logic when generating its
<code>ceph.conf</code> file:</p>
<pre><code># this math
roles/ceph-config/templates/ceph.conf.j2:{% set _osd_memory_target = (ansible_memtotal_mb * 1048576 * hci_safety_factor / _num_osds) | int %}
# sets osd memory target
roles/ceph-config/templates/ceph.conf.j2:osd memory target = {{ _osd_memory_target | default(osd_memory_target) }}
</code></pre>
<p>In my case, this set the value WAY too high. In one of my deployments, it was
set to <code>37853425827</code>, which is like 35GB (PER OSD SERVICE!).</p>
<p>The 4GB default is much more sane, <code>4294967296</code>.</p>
<pre><code># ceph.conf
[osd]
osd memory target = 4294967296
</code></pre>
<hr>
<h1 id="chrome-blocks-https127001-with-neterr_cert_revoked">Chrome blocks <a href="https://127.0.0.1">https://127.0.0.1</a> with NET::ERR_CERT_REVOKED</h1>
<p>This comes up when I reverse tunnel to a remove web server, forwarding it to my workstation. Chrome has decided to not let you do that. They give this errror:</p>
<pre><code class="language-text">Your connection is not private
Attackers might be trying to steal your information from 127.0.0.1 (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_REVOKED
</code></pre>
<p>To bypass that, open <code>chrome://flags/#allow-insecure-localhost</code> and set allow-insecure-localhost to enabled. Click relaunch. This will nuke your current browser…</p>
<p>Once it relaunches you'll be able to open the site.</p>
<hr>
<h1 id="failed-command-aws-ecr-get-login">Failed command: aws ecr get-login</h1>
<p>This error was caused by a bad NTP config. Renew the NTP lease to fix it.</p>
<pre><code>An error occurred (InvalidSignatureException) when calling the GetAuthorizationToken operation: Signature expired: 20190911T174538Z is now earlier than 20190912T004835Z (20190912T010335Z - 15 min.)
</code></pre>
<hr>
<h1 id="pure-storage-iscsi-with-kolla-openstack---fdisk-freezes">Pure Storage iSCSI with Kolla OpenStack - fdisk freezes</h1>
<p>This was a weird one. When using the Kolla containers project to make a
Cinder-volume container that has the <code>purecinder</code> plugin, I was able to get
OpenStack to create the volumes in Pure but it couldn't mount them to the hosts
for things like creating a volume with the <code>--image</code> argument. That made the
volumes I could create basically useless.</p>
<p>These errors were in the logs:</p>
<pre><code class="language-text"># Lots of these in cinder's log
Trying to connect to iSCSI portal

# also from cinder's log, also recurring:
iscsiadm stderr output when getting sessions: iscsiadm: No active sessions

# in DMESG
FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
</code></pre>
<p>The weirdest part was that the <code>fdisk -l</code> command would totally lock up and
the process running it couldn't be killed even with <code>kill -9</code>. The iSCSI volume
never got properly mounted but Pure support said they could see some data
coming into it.</p>
<p><strong>Root cause</strong>: MTU mismatch. Of course. I had set MTU 9000 on the iSCSI ports,
my host ports, and the switchports heading to the Pure Storage iSCSI
interfaces. I had forgotten to set it on the switch heading to the Dell servers
being used for OpenStack.</p>
<hr>
<h1 id="weird-chars-when-pasting-on-mac-os">Weird chars when pasting on Mac OS</h1>
<p>When you paste sometimes you get <code>^[[200~</code> at the start and <code>^[[201~</code> at the
end of the pasted content. Or maybe <code>0~</code> and <code>1~</code>.</p>
<p>To resolve, run:</p>
<pre><code class="language-bash">printf "\e[?2004l"^C
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page contains operations-related mini-guides and minor posts.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</a></li>
<li><a href="#convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</a></li>
<li><a href="#journalctl-commands">Journalctl commands</a></li>
<li><a href="#get-real-ram-use-of-processes">Get real RAM use of processes</a>
<ul>
<li><a href="#of-ceph">of ceph</a></li>
<li><a href="#of-vms">of VMs</a></li>
</ul>
</li>
<li><a href="#bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</a></li>
<li><a href="#create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</a>
<ul>
<li><a href="#use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</a></li>
</ul>
</li>
<li><a href="#test-mtu-from-ubuntu">Test MTU from Ubuntu</a></li>
<li><a href="#find-big-files-on-mac-os">Find big files on Mac OS</a></li>
<li><a href="#renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</a></li>
<li><a href="#create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</a></li>
<li><a href="#passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</a></li>
<li><a href="#vim-tasks">VIM Tasks</a>
<ul>
<li><a href="#find--replace">Find &#x26; Replace</a></li>
<li><a href="#spell-check-commands">Spell-Check commands</a></li>
</ul>
</li>
<li><a href="#use-an-insecure-docker-registry">Use an Insecure Docker Registry</a></li>
<li><a href="#ansible-ignore-errors">Ansible ignore errors</a></li>
<li><a href="#how-to-tail--f-on-dmesg">How to tail -f on dmesg</a></li>
<li><a href="#ubuntu-downgrade-a-package">Ubuntu Downgrade a package</a></li>
<li><a href="#make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</a></li>
<li><a href="#bash-print-all-but-first-line">Bash: Print all but first line</a></li>
<li><a href="#bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</a></li>
</ul>
<h1 id="find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</h1>
<pre><code>cat /proc/net/bonding/bond0
</code></pre>
<hr>
<h1 id="convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</h1>
<pre><code class="language-bash">openssl pkcs7 -print_certs -in certnew.p7b -out cert.pem
</code></pre>
<hr>
<h1 id="journalctl-commands">Journalctl commands</h1>
<p>Watch the logs as they come in:</p>
<pre><code class="language-bash">journalctl -f
</code></pre>
<p>When journalctl logs too much and you need to pair it down:</p>
<pre><code class="language-bash"># to 30 days
journalctl --vacuum-time=30d

# to 500MB
journalctl --vacuum-size=500M
</code></pre>
<p>Show for a specific service</p>
<pre><code class="language-bash">journalctl ceph*
</code></pre>
<hr>
<h1 id="get-real-ram-use-of-processes">Get real RAM use of processes</h1>
<p>You can always use top or htop or whatever. I use this to specifically get the
info about those processes.</p>
<h2 id="of-ceph">of ceph</h2>
<pre><code>ps aux | grep ceph | grep -v -e grep -e qemu -e "\[" | while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); id=$(echo $line | awk '{print $16}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tid: $id,    \tRAM: ($size)"; done
</code></pre>
<h2 id="of-vms">of VMs</h2>
<p>for KVM</p>
<pre><code>ps aux | grep qemu | grep -v -e grep| while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); guest=$(echo $line | awk '{print $13}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tguest: $guest,   \t\tRAM: ($size)" ; done
</code></pre>
<h1 id="bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</h1>
<p>As of Mac Catalina, developing a site using self-signed certs is a bigger pain
than before. Yeah you can go get the key and add it, but that's a nuisance.
It looks like the Chrome team built something they call an
"interstitial bypass keyword" to deal with that. That's one cool feature name.</p>
<p><strong>Obvious disclaimer</strong> If you're not a developer doing this to access your own
site, don't do this.</p>
<p>To bypass the warning, type</p>
<pre><code>thisisunsafe
</code></pre>
<p>Apparently it used to be <code>badidea</code> but they rotated it since it got too well
known. They'll probably do it again someday. I suspect people are using this
on sites they "trust" without really knowing any better and getting hurt by
it.</p>
<hr>
<h1 id="create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</h1>
<p>Useful for simulating a real drive when building a test environment. I use this to test OpenStack in a VM as a Ceph or LVM back-end.</p>
<pre><code class="language-bash"># create empty file of all zeros
sudo dd if=/dev/zero of=/root/virtual-disk bs=1M count=512
# show list of loopback devices. Note its not there
losetup --list
# create the loopback device
losetup /dev/loop0 /root/virtual-disk
# confirm it worked
losetup --list
</code></pre>
<p>You can now treat it like a normal volume. Mount it, put a filesystem on it, whatever. Example:</p>
<pre><code class="language-bash">pvcreate /dev/loop0
vgcreate cinder-volumes /dev/loop0
</code></pre>
<h2 id="use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</h2>
<p>I found <a href="https://unix.stackexchange.com/questions/418322/persistent-lvm-device-with-loopback-devices-by-fstab">this useful systemd unit example</a> that didn't quite work for me, but was close. Here's mine:</p>
<p><code>vi /etc/systemd/system/loops-setup.service</code></p>
<pre><code class="language-text">[Unit]
Description=Setup loopback devices

DefaultDependencies=no
Conflicts=umount.target

Requires=lvm2-lvmetad.service mnt-host.mount
Before=local-fs.target umount.target
After=lvm2-lvmetad.service mnt-host.mount

[Service]
ExecStart=/sbin/losetup /dev/loop0 /root/virtual-disk
ExecStop=/sbin/losetup -d /dev/loop0

RemainAfterExit=yes
Type=oneshot

[Install]
WantedBy=local-fs-pre.target
</code></pre>
<p>Then enable it</p>
<pre><code class="language-bash">systemctl enable loops-setup
</code></pre>
<p>Reboot, then check <code>vgdisplay</code> (if that's your use case). It'll be there.</p>
<hr>
<h1 id="test-mtu-from-ubuntu">Test MTU from Ubuntu</h1>
<p>You would not believe the strange problems that an MTU issue can be the root cause of.</p>
<pre><code class="language-bash">ping -M do -s 1472 &#x3C;remote server>
ping -M do -s 8900 &#x3C;remote server>
</code></pre>
<hr>
<h1 id="find-big-files-on-mac-os">Find big files on Mac OS</h1>
<p>My SSD is always almost full. Here are useful commands for cleaning up:</p>
<pre><code class="language-bash"># Find all the files over 1G
sudo find / -type f -size +1G -exec ls -lh {} \; | awk '{ print $9 ": " $5 }' 2>/dev/null

# List files in Downloads directory by size
ls -lhS ~/Downloads
</code></pre>
<h1 id="renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</h1>
<pre><code class="language-bash">service ntp stop
ntpd -gq
service ntp start
</code></pre>
<hr>
<h1 id="create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</h1>
<pre><code class="language-bash">aws ecr describe-repositories --repository-names $repo_name \
    || aws ecr create-repository --repository-name $repo_name
</code></pre>
<hr>
<h1 id="passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</h1>
<p>Run <code>visudo</code> as root.</p>
<pre><code class="language-bash"># as root
export EDITOR=vim
visudo
</code></pre>
<p>Add the user (replace <code>exampleUser</code>) at the end.</p>
<pre><code class="language-text">exampleUser ALL=(ALL) NOPASSWD: ALL
</code></pre>
<p>For those who don't use vim, a quick reminder:
<code>i</code> enter to insert mode , <code>[esc]</code> to exit insert mode, <code>:wq</code> to save and quit.</p>
<hr>
<h1 id="vim-tasks">VIM Tasks</h1>
<h2 id="find--replace">Find &#x26; Replace</h2>
<pre><code class="language-bash"># replace findThis with ReplaceWithThis
:%s/findThis/ReplaceWithThis/g
</code></pre>
<h2 id="spell-check-commands">Spell-Check commands</h2>
<pre><code class="language-text"># enable/disable
:set spell spelllang=en_ca
:setlocal spell spelllang=en_ca
:set nospell

# next/last
]s
[s

# suggest change
z=

# add/remove word from dictionary
zg
zw`
</code></pre>
<hr>
<h1 id="use-an-insecure-docker-registry">Use an Insecure Docker Registry</h1>
<p>To use a registry with no HTTP cert:</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
    "insecure-registries" : ["myregistrydomain.com:5000"]
}
</code></pre>
<hr>
<h1 id="ansible-ignore-errors">Ansible ignore errors</h1>
<p>The command is <code>ignore_errors: yes</code></p>
<p>Just throw this into the task that's failing. It will still fail, but the play
won't stop.</p>
<p>Example:</p>
<pre><code class="language-yml">- name: Ensure the docker service is running
  script: ../tools/validate-docker-execute.sh
    ignore_errors: yes
</code></pre>
<hr>
<h1 id="how-to-tail--f-on-dmesg">How to tail -f on dmesg</h1>
<pre><code class="language-bash">dmesg -wH
</code></pre>
<hr>
<h1 id="ubuntu-downgrade-a-package">Ubuntu Downgrade a package</h1>
<pre><code class="language-bash"># Find the available versions under Versions:
apt-cache showpkg &#x3C;package>

# Install one of those
apt-get install &#x3C;package>=&#x3C;version>
</code></pre>
<hr>
<h1 id="make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</h1>
<p>Useful for OS that ship with python3 but not python2.
Edit the inventory, find a group of the groups and apply this var.</p>
<pre><code class="language-ini">[baremetal:vars]
ansible_python_interpreter=/usr/bin/python3
</code></pre>
<hr>
<h1 id="bash-print-all-but-first-line">Bash: Print all but first line</h1>
<p>Use <code>tail -n +2</code>.
This was unintuitive to me because +1 seems like it should do it.</p>
<pre><code class="language-bash"># Example
docker images | awk '{print $3}' | tail -n +2
</code></pre>
<hr>
<h1 id="bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</h1>
<p>Use trap.</p>
<ul>
<li><a href="http://redsymbol.net/articles/bash-exit-traps/">This is a nice post covering lots of examples</a>.</li>
<li><a href="https://github.com/GoogleCloudPlatform/endpoints-quickstart/blob/master/scripts/deploy_api.sh">Here's where I first saw it used</a></li>
</ul>
<pre><code class="language-bash">temp_file=$(mktemp)

cleanup() {
  rm "$temp_file"
}

trap cleanup EXIT
</code></pre>


<p>How to run a docker container on AWS ECS (Elastic Compute Cloud)  using the AWS
web console. A container from the Elastic Container Registry will be launched.
It will run on Amazon's <a href="https://aws.amazon.com/fargate/">Fargate Container-as-a-Service</a>.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#launch-a-container-on-fargate">Launch a container on Fargate</a></li>
<li><a href="#publish-containers-network-to-the-internet">Publish container's network to the internet</a></li>
<li><a href="#using-a-custom-domain-name">Using a custom domain name</a></li>
</ul>
<h1 id="launch-a-container-on-fargate">Launch a container on Fargate</h1>
<ol>
<li>Upload an image to AWS ECR. <a href="/aws-ecr.html">See my guide on using AWS ECR</a>.</li>
<li>Open <a href="https://ca-central-1.console.aws.amazon.com/ecs/">AWS ECS</a></li>
<li>Under Container Definition, select Configure</li>
<li>Fill in the form
<ol>
<li>For the image URL, use the URL of the image uploaded to
ECR. I was testing a reverse proxy image, so it looked like this:
<code>000000000000.dkr.ecr.ca-central-1.amazonaws.com/test/revproxy:latest</code></li>
<li>I have my nginx reverse proxy a 256MB soft limit for RAM</li>
<li>For my reverse proxy I made port mappings for port 80 and 443</li>
<li>No need to fill out any of the advanced fields</li>
</ol>
</li>
<li>Fill out the Task Definition section, click Next</li>
<li>Define your service
<ol>
<li>Without a load-balancer, the only way to access your container will be by
using <a href="https://aws.amazon.com/blogs/compute/access-private-applications-on-aws-fargate-using-amazon-api-gateway-privatelink/">PrivateLink</a>.</li>
<li>During the initial cluster create wizard you can choose an Application
Load Balancer. That seems to work well.</li>
</ol>
</li>
<li>Name the Fargate cluster, next</li>
<li>Click create</li>
</ol>
<p>It will take a little while. Once its finished you can click on View Service.</p>
<hr>
<h1 id="publish-containers-network-to-the-internet">Publish container's network to the internet</h1>
<p>Navigate to the newly created service and go to the Details tab. From there you
can see a Load Balancing section. It should have a link to the new load
balancer.</p>
<p>You can also access this page from the left nav-bar by going to Load Balancing,
then choosing the new load balancer.</p>
<p>In the Description tab here, you can find the DNS name that AWS has assigned
to this load balancer. Unfortunately you can't depend on the IP address behind
this FQDN to stay the same.</p>
<hr>
<h1 id="using-a-custom-domain-name">Using a custom domain name</h1>
<p>I'm working on that. Its not intuitive. If you know how, I'd like to hear about
it..</p>


<p>Docker supports shared memory. The default is only 64MB though which is TINY.</p>
<p>The small size became a problem during a containerized OpenStack upgrade.</p>
<p>This will be a really short post because I haven't done much playing with the
shared memory yet, I just think its really neat.</p>
<hr>
<h1 id="change-the-devshm-size-on-the-system">Change the /dev/shm size on the system</h1>
<p>This lets you change the size of shared memory on a host.</p>
<pre><code class="language-bash">mount -o remount,size=1G /dev/shm
</code></pre>
<hr>
<h1 id="update-docker-daemons-default-shared-memory-size">Update Docker Daemon's Default Shared Memory Size</h1>
<p>This will update the default across the whole Docker service.</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
  "default-shm-size": "512M"
}
</code></pre>
<hr>
<h1 id="set-a-containers-shared-memory-size">Set a Containers Shared Memory Size</h1>
<p>Here's how to change it for a specific container at runtime.</p>
<pre><code class="language-bash">docker run -it --shm-size=512M
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This page covers how to do some things in Git.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#use-vim-for-commits">Use Vim for commits</a></li>
<li><a href="#rename-branch">Rename Branch</a></li>
<li><a href="#roll-back-merge">Roll Back Merge</a>
<ul>
<li><a href="#option-1-abort-the-merge">Option 1: Abort the merge</a></li>
<li><a href="#option-2-reset-to-the-last-commit">Option 2: Reset to the last commit</a></li>
</ul>
</li>
<li><a href="#delete-a-file-from-git-history">Delete a File From Git History</a>
<ul>
<li><a href="#if-you-dont-push-with---force">If you don't push with --force</a></li>
</ul>
</li>
</ul>
<h1 id="use-vim-for-commits">Use Vim for commits</h1>
<p>Ugh, Nano. Kidding, but seriously here's how to use Vim instead.</p>
<pre><code class="language-bash">git config --global core.editor "vim"
</code></pre>
<hr>
<h1 id="rename-branch">Rename Branch</h1>
<pre><code class="language-bash"># Check out the branch to rename
git checkout development
# Rename the branch
git branch -m master
# Push the change
git push origin :development master
</code></pre>
<p>If you're using BitBucket and it fails to delete the branch, it might be set
as your MAIN branch. You need to change that before it can be deleted. You can
change it in the settings page.</p>
<hr>
<h1 id="roll-back-merge">Roll Back Merge</h1>
<p>If you do a git merge and its really ugly, you might want to undo that.</p>
<p>First you did the commit</p>
<pre><code class="language-bash">git checkout master
git merge --squash development
</code></pre>
<p>Then you got a TON of <code>CONFLICT</code> messages and a really dirty state.</p>
<h2 id="option-1-abort-the-merge">Option 1: Abort the merge</h2>
<p>This one's my preference. There might be a few <code>~</code> files left over to <code>rm</code>
manually.</p>
<pre><code class="language-bash">git merge --abort
</code></pre>
<h2 id="option-2-reset-to-the-last-commit">Option 2: Reset to the last commit</h2>
<p>First, get the commit right before this one from git log</p>
<pre><code class="language-bash">git log -n 1
</code></pre>
<p>Copy the commit hash</p>
<pre><code class="language-bash">git reset --hard &#x3C;commit hash>
</code></pre>
<p>You may need to clean up some files after that.</p>
<p><strong>extra</strong>
Usually, the next thing you wanna do is try the merge again with
<code>--stattheirs</code>, or</p>
<pre><code class="language-bash">git merge --squash --strategy-option=theirs development
</code></pre>
<hr>
<h1 id="delete-a-file-from-git-history">Delete a File From Git History</h1>
<p>It's important to note, as GitHub highlights <a href="https://help.github.com/en/articles/removing-sensitive-data-from-a-repository">in their documentation</a>,
if you've uploaded a secret password or something, you still need to change it.</p>
<p>The file I removed was called <code>sqldata.json</code>, and it was in my project root.
From the project root, I ran:</p>
<pre><code class="language-bash">git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch sqldata.json" \
  --prune-empty --tag-name-filter cat -- --all

git push origin --force --all
</code></pre>
<h2 id="if-you-dont-push-with---force">If you don't push with --force</h2>
<p>When I tried to push, though, I got an error. I'd forgotten to use <code>--force</code>.</p>
<pre><code class="language-bash">To https://github.com/kylep/jsc2f.git
 ! [rejected]        master -> master (non-fast-forward)
 error: failed to push some refs to 'https://github.com/kylep/jsc2f.git'
 hint: Updates were rejected because the tip of your current branch is behind
 hint: its remote counterpart. Integrate the remote changes (e.g.
 hint: 'git pull ...') before pushing again.
 hint: See the 'Note about fast-forwards' in 'git push --help' for details.
</code></pre>
<p>Then, when I tried to pull, I got</p>
<pre><code class="language-bash">From https://github.com/kylep/jsc2f
 + ca7d7cc...710d37b master     -> origin/master  (forced update)
 fatal: refusing to merge unrelated histories
</code></pre>
<p>If you're getting errors like that, maybe you need to run push with <code>--force</code>.</p>
<pre><code class="language-bash">git push origin --force --all
</code></pre>


<p><strong>This post is linked to from the <a href="/open-source-cli-project.html">FOSS Python App Project</a></strong></p>
<hr>
<p>At a high level, the steps to upload a package to PyPi are as follows:</p>
<ol>
<li>Write the project code</li>
<li>Create a release on GitHub and put it in the <code>download_url</code> of setup.py</li>
<li>Write a setup.py file to build and describe the project</li>
<li>Write a setup.cfg file pointing to the readme</li>
<li>create a source distribution</li>
</ol>
<h1 id="write-the-project-code">Write the project code</h1>
<p>I'm not going to cover how to build a python project in this post, but here's
the very small and simple project I used when learning this procedure:
<a href="https://github.com/kylep/jsc2f">GitHub: kylep/jsc2f</a>.</p>
<hr>
<h1 id="create-a-release-on-github">Create a Release on GitHub</h1>
<p>Log into your GitHub project and click on releases. If you can't find the link,
you can also just append <code>/releases</code> to the end of the URL to get there.</p>
<p>Click "Create a new release". Enter your tag and branch, along with a release
title and description.</p>
<p>Under the Assets section of the new release there's a (tar.gz) link to download
the source code. If you right click it and copy the URL, that's used next in
the setup.py file.</p>
<hr>
<h1 id="write-a-setuppy-file">Write a setup.py file</h1>
<p>Here's mine. Check GitHub to see if I've changed it.
The <code>download_url</code> value comes from the GitHub release.</p>
<pre><code class="language-python">from setuptools import setup

# Without this you'll get no description on the pypi site
with open("README.md", "r") as fh:
    long_description = fh.read()

setup(
    name='jsc2f',
    packages=['jsc2f'], # Alternatively you can use find_packages()
    version='0.1', # match the github release you make later
    license='MIT', # match the LICENSE file in the project
    description='Saves a JSON fields SQLs cell to a file, or UPDATE it back',
    author='Kyle Pericak',
    author_email='kyle@pericak.com',
    url='kyle.pericak.com/jsc2f', # either your own page or github
    download_url='https://github.com/kylep/jsc2f/archive/v0.1.tar.gz',
    keywords=['SQL', 'JSON', 'file'],
    install_requires=[
        'click',
        'mysql-connector'
    ],
    entry_points='''
        [console_scripts]
        jsc2f=jsc2f.cli:cli
    ''',
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Information Technology',
        'Topic :: Database',
        'Programming Language :: Python :: 3',
        'License :: OSI Approved :: MIT License',
        'Natural Language :: English'

</code></pre>
<h2 id="picking-a-topic">Picking a topic:</h2>
<p>I used <a href="https://pypi.org/pypi?%3Aaction=list_classifiers">this classifier list</a></p>
<hr>
<h1 id="write-setupcfg">Write setup.cfg</h1>
<p>This goes in the project root next to setup.py.</p>
<p><code>vi setup.cfg</code></p>
<pre><code class="language-ini">[metadata]
description-file = README.md
</code></pre>
<h1 id="creating-a-source-distribution">Creating a source distribution</h1>
<p>For this you use the python <a href="https://docs.python.org/2/distutils/sourcedist.html">sdist</a>
tool and <a href="https://pypi.org/project/twine/">twine</a>.</p>
<p>From the git project root:</p>
<pre><code class="language-bash">pip install twine

# This will create dist/
python setup.py sdist

# Twine uploads the dist/ files to Pypi
twine upload --skip-existing dist/*
# Enter your username and password for PyPI
</code></pre>


<p>Often I need to operate on a huge JSON string that's saved in a MaraiDB
database. Here's how I do that.</p>
<p>It needs to be said that raw SQL queries like this are generally a bad idea
compared to something like <a href="https://docs.sqlalchemy.org/en/13/orm/">SQLAlchemy's ORM</a>.
Having said that, this example is way smaller and more portable.</p>
<p>The way this works is you open up an interactive python session, ideally in
a virtualenv. Set the variable values at the top. Run the two functions as
needed. In another window/pane, edit the JSON files.</p>
<p>Another option is to use <code>jq</code> with interactive MySQL commands. I'm not covering
that in this post, though.</p>
<h1 id="environment-setup">Environment setup</h1>
<p>This is best done in a venv if <code>mysql-connector</code> doesn't need to be installed
on the base system.</p>
<pre><code class="language-bash">pip install mysql-connector
</code></pre>
<h1 id="the-code">The code</h1>
<p>It's not a bad idea to write this into a parameterized CLI or whatever.
I just run this in an interactive python shell in a dedicated window/pane.</p>
<pre><code class="language-python">import json
import mysql.connector as mariadb

# Fill in the blanks
db_host= ''  # IP address
db_port = 3306
db_username = ''
db_password = ''
db_name = ''
table_name = ''
column_name = ''
where = ''  # (optional) where = 'WHERE `x` == 1'


def save_to_file(filename):
    """Read the json data from the database, save to filename"""
    conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
    cursor = conn.cursor()
    read_query = 'SELECT {} from {}{}'.format(column_name, table_name, where)
    cursor.execute(read_query)
    data = cursor.fetchone()[0]
    conn.close()
    jdata = json.dumps(json.loads(data), indent=4, sort_keys=True)
    with open(filename, 'w+') as out_file:
        out_file.write(jdata)
    print('done')


def update_from_file(filename):
    """Read data from filename, write to database"""
    with open(filename) as json_file:
        jdata = json.load(json_file)
    data = json.dumps(jdata)
    conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
    cursor = conn.cursor()
    update_query = "UPDATE {} SET {} = '{}'{}".format(table_name, column_name,
                                                      data, where)
    cursor.execute(update_query)
    conn.commit()
    conn.close()
    print('done')
</code></pre>
<h1 id="usage">Usage</h1>
<p>From an interactive python terminal, paste the above script in.</p>
<p>Then to save the file, run something like:</p>
<pre><code class="language-python">save_to_file(filename='/home/kyle/sqldata.json')
</code></pre>
<p>Then you can update the file using an editor <code>vi /home/kyle/sqldata.json</code>.</p>
<p>Back in the Python terminal, you can then push the data back up.</p>
<pre><code class="language-python">update_from_file(filename='/home/kyle/sqldata.json')
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page contains python-related mini-guides and minor posts.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#pylint-ignore-comments">Pylint Ignore Comments</a>
<ul>
<li><a href="#turn-off-linting-for-now">Turn off Linting for now</a></li>
</ul>
</li>
<li><a href="#get-an-object-from-list-of-objects-where-the-object-property-has-value-x">Get an object from list of objects where the object property has value x</a>
<ul>
<li><a href="#a-subset-list-is-expected-list-comprehension">A subset list is expected: list comprehension</a></li>
<li><a href="#get-one-match-next-w-generator-expression">Get one match: next() w/ generator expression</a></li>
</ul>
</li>
<li><a href="#iterate-through-dictionary">Iterate through dictionary</a></li>
<li><a href="#print-json-nicely-in-ascii-no-unicode-u-prefix">Print JSON nicely in ASCII (no unicode 'u' prefix)</a></li>
<li><a href="#replace-a-string-in-a-file">Replace a string in a file</a></li>
<li><a href="#get-environment-variable">Get environment variable</a></li>
<li><a href="#use-jinja2-for-templating">Use Jinja2 for templating</a>
<ul>
<li><a href="#jinja2-templating-with-files">Jinja2 templating with files</a></li>
</ul>
</li>
<li><a href="#execute-sql-query-against-mysql-or-mariadb">Execute SQL query against MySQL or MariaDB</a></li>
<li><a href="#read-from-std-in-bash-pipe">Read from std in (bash pipe)</a></li>
<li><a href="#custom-exceptions">Custom Exceptions</a></li>
<li><a href="#cant-run-pip-after-updating">Can't run pip after updating</a></li>
</ul>
<h1 id="pylint-ignore-comments">Pylint Ignore Comments</h1>
<p>I can never remember the syntax to ignore certain Pylint warnings. Yeah you can
turn them off everywhere but usually I just want to ignore a certain occurrence.</p>
<pre><code class="language-python"># - This file is a mess and I don't want pylint here:
# pylint: disable-all

# - troublesome import not being found:
# pylint: disable=import-error

# - redefined-builtin (W0622):
# pylint: disable=W0622

# - too-many-public-methods (R0904):
# pylint: disable=R0903

# - too-many-arguments (R0913):
# pylint: disable-msg=R0913

# - too-many-locals (R0914)
# pylint: disable=R0914

# - Invalid constant name:
# pylint: disable=C0103

# - Catching too general exception:
# pylint: disable=W0703

# - Using python3 compatible print statement makes python2 linter mad
# pylint: disable=superfluous-parens

# - Ignore too-long line (this needs both the pylint command and the trailing no-qa)
# pylint: disable=line-too-long
binaryValue = b'QVdTOmV5SndZWGxzYjJGa0lqb2lVSFkwWXpKeFNEaHZXVkkxVmtwRmVXdFhXamhKUVRGcGNFWk1kelJKZEV0V1UwSk9'  # noqa: E50

# - Wrong Import Position (Required when putting python version check up top)
# flake8: noqa=402
# pylint: disable=wrong-import-position
</code></pre>
<h2 id="turn-off-linting-for-now">Turn off Linting for now</h2>
<p>In VIM, run:</p>
<pre><code>:ALEToggle
</code></pre>
<h1 id="get-an-object-from-list-of-objects-where-the-object-property-has-value-x">Get an object from list of objects where the object property has value x</h1>
<h2 id="a-subset-list-is-expected-list-comprehension">A subset list is expected: list comprehension</h2>
<p>See <a href="https://docs.python.org/3/tutorial/datastructures.html">Section 5.1.3 - List comprehension</a></p>
<pre><code class="language-python"># This will return a list of objects whith a value
[obj for obj in haystack_list if obj.value == search]

# Example of list comprehension to get the even numbers in a list
nums = [1, 2, 3, 4, 5, 6, 7, 8]
even_nums = [num for num in nums if num % 2 == 0]
</code></pre>
<h2 id="get-one-match-next-w-generator-expression">Get one match: next() w/ generator expression</h2>
<p>The <code>next()</code> method will return the first element found in an iterable.</p>
<p>See <a href="https://www.python.org/dev/peps/pep-0289/">PEP 289 - Generator expressions</a></p>
<pre><code class="language-python"># Raises a StopIteration exception if no match is found
needle = next((obj for obj in haystack_list if obj.value == value))

# To return a value instead of raising an exception, pass a second argument
needle = next((obj for obj in haystack_list if obj.value == search), None)

# Example of what Next is really doing. This returns 1.
next(iter([1,2,3]))
</code></pre>
<hr>
<h1 id="iterate-through-dictionary">Iterate through dictionary</h1>
<pre><code class="language-python">ex_dict = {'a': 1, 'b': 2, 'c': 3}
for key in ex_dict:
    print(key)
    print(ext_dict[key]) # value
</code></pre>
<p>This works because <code>ex_dict.keys() == list(ex_dict)</code></p>
<hr>
<h1 id="print-json-nicely-in-ascii-no-unicode-u-prefix">Print JSON nicely in ASCII (no unicode 'u' prefix)</h1>
<p>This is like using <a href="https://docs.python.org/3/library/pprint.html">pprint</a> on
the JSON object but without the unicode prefixes.</p>
<pre><code class="language-python">data = ... # the json string
jdata = json.dumps(json.loads(data),  indent=4, sort_keys=True)
print(jdata)
</code></pre>
<hr>
<h1 id="replace-a-string-in-a-file">Replace a string in a file</h1>
<p>Though honestly, <a href="https://www.geeksforgeeks.org/sed-command-in-linux-unix-with-examples/">sed</a>
is usually a better option. Or if there are a lot of values, I like to use a
jinja template.</p>
<pre><code class="language-python">file_path = '/home/user/example_file'
find_str = 'REPLACE_THIS'
replace_str = 'WITH_THIS'

# Open the file to read the value
with open(file_path, 'r') as read_file:
    file_content = read_file.read()

# Replace the string
file_content = file_content.replace(find_str, replace_str)

# Write the file
with open(file_path, 'w') as write_file:
    write_file.write(replaced_content)
</code></pre>
<hr>
<h1 id="get-environment-variable">Get environment variable</h1>
<p>I mostly use this with <a href="https://docs.docker.com/engine/reference/builder/">Dockerfile ENV values</a>.</p>
<pre><code class="language-python">import os

environment_var_name = 'MY_ENV_VAR'
environment_var_vlue os.environ[environment_var_name]
</code></pre>
<hr>
<h1 id="use-jinja2-for-templating">Use Jinja2 for templating</h1>
<p>I use this for config files, like <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_templating.html">how Ansible does it</a>.</p>
<pre><code class="language-python">from jinja2 import Template

replacements = {'USERNAME': 'kyle', 'PASSWORD': 'example'}
template_text = 'connection = mysql://{{USERNAME}}:{{PASSWORD}}@127.0.0.1'
template = Template(template_text)
replaced_text = template.render(**replacements)
</code></pre>
<h2 id="jinja2-templating-with-files">Jinja2 templating with files</h2>
<p>Same idea as above but it reads a .j2 file as input and writes it to a config
file. Can combine this nicely with environment variables to write config files
as part of a Docker start script.</p>
<pre><code class="language-python">from jinja2 import Template

def apply_template(jinja2_file, output_file, replacements):
    """Replace the replacements values in jinja2_file, write to output_file"""
    with open(jinja2_file, 'r') as j2_file:
        j2_text = j2_file.read()
    template = Template(j2_text)
    replaced_text = template.render(**replacements)
    with open(output_file, 'w+') as write_file:
        write_file.write(replaced_text)

</code></pre>
<hr>
<h1 id="execute-sql-query-against-mysql-or-mariadb">Execute SQL query against MySQL or MariaDB</h1>
<pre><code class="language-python">import mysql.connector as mariadb

conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
cursor = conn.cursor()
cursor.execute(sql_query)
conn.commit() # if it was a write action
conn.close()
</code></pre>
<hr>
<h1 id="read-from-std-in-bash-pipe">Read from std in (bash pipe)</h1>
<p>For instance I have a file named x that has a big string where I want to
replace commas with spaces. Sed works but I can never remember the syntax.</p>
<pre><code class="language-python">cat x | python -c "import sys; print(sys.stdin.read())"

# Example of using python instead of sed to replace , with spaces
cat x | python -c "import sys; print(sys.stdin.read().replace(',',' '))"

# You can use this in fun ways like combining it with bash loops
cat x | python -c \
  "import sys; print(sys.stdin.read().replace(',','\n'))" | while read line; do
  echo $line
done
</code></pre>
<hr>
<h1 id="custom-exceptions">Custom Exceptions</h1>
<pre><code class="language-python">class MyCustomException(Error):
    pass
</code></pre>
<hr>
<h1 id="cant-run-pip-after-updating">Can't run pip after updating</h1>
<p>So you ran into a problem because ubuntu ships an ancient version of pip, and
some dependency package like <code>subprocess32</code> is breaking your install. A forum
post says you should update pip.</p>
<pre><code class="language-bash">pip install -U pip
</code></pre>
<p>Except now pip won't run at all.</p>
<pre><code class="language-bash">pip install python-openstackclient
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in &#x3C;module>
    from pip import main
ImportError: cannot import name main
</code></pre>
<p>I think maybe the pip executable file moved after the update.
To fix it:</p>
<pre><code class="language-bash">hash -r pip
</code></pre>
<p>Now pip works again.</p>
<p>As an aside, if <code>subprocess32</code> really was your problem, updating pip didn't
help. For some reason you have to use apt to install <code>python-subprocess32</code>.</p>


<p>This guide covers how I set up a dedicated modded Minecraft Feed The Beast
server on spare laptop so my wife and I could play together.</p>
<p><strong>NOTE</strong>: Ubuntu 18.04 doesn't work at all. It uses a newer version of Java
which doesn't work with Minecraft's launcher.</p>
<hr>
<h1 id="server-setup">Server Setup</h1>
<h2 id="make-the-vm">Make the VM</h2>
<p>I'm just using a laptop with Windows 8.0 and Hyper-V for this. It has 16GB RAM,
4 Cores + Hyper-threading, and a 1TB SSD.</p>
<p>Here are the server VM specs I used:</p>
<ul>
<li>Operating System: Ubuntu 16.04 Server</li>
<li>RAM: 12GB</li>
<li>Disk: 200GB (Thin provisioned)</li>
</ul>
<p>Also, set a static IP address.</p>
<h2 id="download-the-modded-server-code">Download the Modded Server Code</h2>
<p>Open your browser to <a href="https://www.curseforge.com/minecraft/modpacks/ftb-ultimate-reloaded">the mod site</a>
and find the link to the <a href="https://www.feed-the-beast.com/projects/ftb-ultimate-reloaded/files">server files</a>.</p>
<p>You can download the server files from your browser and SCP it over or download
them to the server directly with wget:</p>
<pre><code class="language-bash">mkdir -p ~/ftb
cd ~/ftb
wget https://media.forgecdn.net/files/2778/970/FTBUltimateReloadedServer_1.9.0.zip

</code></pre>
<p>Unzip the server file archive</p>
<pre><code class="language-bash">apt-get update
apt-get install -y unzip
unzip FTBUltimateReloadedServer_1.9.0.zip
</code></pre>
<h2 id="configure-the-server">Configure the Server</h2>
<p><code>vi settings.sh</code></p>
<pre><code class="language-bash"># Set the RAM to 10GB
export MAX_RAM="10240M"
</code></pre>
<h2 id="install-java-jre">Install Java JRE</h2>
<pre><code class="language-bash">apt-get install -y default-jre
</code></pre>
<h2 id="launch-the-server">Launch the Server</h2>
<p>I like to console into the server and start a Tmux session, then SSH to it
and join that session to keep the server running. You can also just make a
systemd unit if you like.</p>
<p>From the <code>~/ftb</code> folder start the server interactively:</p>
<pre><code class="language-bash">bash ~/ftb/ServerStart.sh
</code></pre>
<hr>
<h1 id="client-setup">Client Setup</h1>
<h2 id="suggested-option-ftb-launcher">Suggested Option: FTB Launcher</h2>
<p>Download the official FTB launcher from <a href="https://www.feed-the-beast.com/">here</a>.</p>
<p>It's really intuitive, no instructions are needed.</p>
<h2 id="alternate-option-1-multimc">Alternate Option 1: MultiMC</h2>
<p>This option has my favorite UI, but it seems to run into Java problems on
Windows sometimes.</p>
<p>MultiMC works offline, and is strictly about launching Minecraft.
It has some nice features like a window that seems to <code>tail -f</code> the Minecraft
log. It can also set different resource usage values per mod.</p>
<ol>
<li>Download <a href="https://multimc.org/">Multi-MC</a>, install, and launch it.</li>
<li>Click Add Instance</li>
<li>Choose FTB Legacy > FTB Ultimate Reloaded. OK.</li>
<li>Wait for the download to finish</li>
<li>Double-click the new icon with the mod name</li>
<li>If prompted to go to the account manager, say yes</li>
<li>If not in the settings window, open it. Go to Java. Increase the RAM.
It looks like 5120MB works well, with 512 for PermGen.</li>
<li>Close the settings.</li>
<li>Double-click on the installed mod to launch it</li>
</ol>
<h2 id="alternative-option-2-twitch-client">Alternative Option 2: Twitch Client</h2>
<p><strong>Note</strong>: The twitch launcher is a pain. It refuses to go offline and shows you
tons of stuff not related to modding Minecraft.
Unless you already use Twitch for other games or enjoy watching streamers,
I suggest you try MultiMC first.</p>
<p><strong>Update</strong>: This is pretty much discontinued now, definitely avoid it.</p>
<p>From a Windows client OS, go to <a href="https://www.twitch.tv/downloads">twitch's download site</a>
and download the <strong>twitch app for windows</strong>.</p>
<p>Run the installer. You might need to make a Twitch account if you don't have
one, which is a pain but they bought Curse so there's no real better way to
do this.</p>
<ol>
<li>Log in. If you don't have an account, try MultiMC, then make one if it
doesn't work for you.</li>
<li>At the top of the screen, go to Mods.</li>
<li>Click on Minecraft. If its not installed yet, its probably greyed out.</li>
<li>Click Install.</li>
<li>When the install is done you can go to "Browse FTB Modpacks" up top.</li>
<li>Click FTB Ultimate Reloaded</li>
<li>Click Install</li>
</ol>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to make a Docker registry on AWS Elastic Container
Registry (ECR).
To secure the registry, least-privilege roles are created and assigned to
service account users in the AWS AIM tool</p>
<hr>
<h3 id="table-of-contents">Table of contents</h3>
<ul>
<li><a href="#create-an-aws-ecr-repository">Create an AWS ECR Repository</a></li>
<li><a href="#create-an-aws-iam-user-for-the-registry">Create an AWS IAM User For the Registry</a>
<ul>
<li><a href="#create-groups-for-ecr-access">Create Groups for ECR Access</a>
<ul>
<li><a href="#make-a-group-with-write-access">Make a group with write access</a></li>
<li><a href="#make-a-read-only-group">Make a read-only group</a></li>
</ul>
</li>
<li><a href="#create-users-for-ecr-access">Create Users for ECR Access</a></li>
</ul>
</li>
<li><a href="#connect-docker-to-the-new--registry">Connect Docker to the new  Registry</a>
<ul>
<li><a href="#install-the-aws-cli">Install the AWS CLI</a></li>
<li><a href="#configure-the-aws-cli-user">Configure the AWS CLI USER</a></li>
<li><a href="#get-the-docker-login-command">Get the docker login command</a></li>
</ul>
</li>
<li><a href="#push-images-to-the-ecr-registry">Push Images to the ECR Registry</a></li>
</ul>
<h1 id="create-an-aws-ecr-repository">Create an AWS ECR Repository</h1>
<p>Navigate to the
<a href="https://ca-central-1.console.aws.amazon.com/ecr/repositories">AWS ECR Repositories page</a>
and click Create a repository.</p>
<p>Fill in the form to name your repository.</p>
<h1 id="create-an-aws-iam-user-for-the-registry">Create an AWS IAM User For the Registry</h1>
<p>Its a good idea to have user accounts with write access, and others with
read-only access. This way you can push to the registry with your privileged
user but you never need to expose those credentials outside your development
or CI/CD environment.</p>
<h2 id="create-groups-for-ecr-access">Create Groups for ECR Access</h2>
<p>First, review the AWS Group access levels <a href="https://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr_managed_policies.html">here</a>.</p>
<h3 id="make-a-group-with-write-access">Make a group with write access</h3>
<p>Users in this group can push to existing registries but they can't use the CLI
to make new ones.</p>
<ol>
<li>Go to the <a href="https://console.aws.amazon.com/iam">AWS IAM page</a></li>
<li>Click Groups on the sidebar</li>
<li>Name the group, ex: <code>ecr-write</code></li>
<li>Attach Policy: <code>AmazonEC2ContainerRegistryPowerUser</code>, next</li>
<li>Create Group</li>
</ol>
<h3 id="make-a-read-only-group">Make a read-only group</h3>
<p>Its the same procedure as above, but use the read-only policy instead:
<code>AmazonEC2ContainerRegistryReadOnly</code>.</p>
<p>Consider also making an admin group with the
<code>AmazonEC2ContainerRegistryFullAccess</code> permission.</p>
<h2 id="create-users-for-ecr-access">Create Users for ECR Access</h2>
<p>Make two users, one for pushing and one for pulling. You can share the user
will read access to anyone who needs to consume the images.</p>
<p>To make a user:</p>
<ol>
<li>Go to the <a href="https://console.aws.amazon.com/iam">AWS IAM page</a></li>
<li>Click Users on the sidebar</li>
<li>Add User</li>
<li>Enter user name, check Programmatic Access, next</li>
<li>Choose from the groups you just created to grant read/write access, next</li>
<li>Skip the tags, next</li>
<li>Create User. Be sure to save the secret access key in your password manager.</li>
</ol>
<h1 id="connect-docker-to-the-new--registry">Connect Docker to the new  Registry</h1>
<h2 id="install-the-aws-cli">Install the AWS CLI</h2>
<p>The one from apt is really old and doesn't work. Use pip.</p>
<pre><code class="language-bash">apt-get install -y python python-pip
pip install aws awscli
</code></pre>
<p>Test it out</p>
<pre><code class="language-bash">python -m awscli help
</code></pre>
<h2 id="configure-the-aws-cli-user">Configure the AWS CLI USER</h2>
<p>When prompted, enter your key, secret key, region (ca-central-1), and output
format (none):</p>
<pre><code class="language-bash">aws configure
</code></pre>
<h2 id="get-the-docker-login-command">Get the docker login command</h2>
<p>Use the AWS CLI to fetch and execute the docker login command. You can also
export this command to a file to run it, but there's no point since the token
gets invalidated after 12 hours.</p>
<pre><code class="language-bash">$(aws ecr get-login --no-include-email --region ca-central-1)
</code></pre>
<hr>
<h1 id="push-images-to-the-ecr-registry">Push Images to the ECR Registry</h1>
<p>If you click on the Push Commands button in the ECR registry page you'll get
examples programmatically generated for your registry.</p>
<p>Pushing is kind of unintuitive. Unlike a normal <code>registry:2</code> docker registry,
ECR won't automatically create your repositories for you. You have to manually
create each one. You can use the web UI, or log in as an admin user and run the
following:</p>
<pre><code class="language-bash">aws ecr create-repository --repository-name kolla/ubuntu-source-base
</code></pre>
<p>For scripting, this is a handy way of creating the repo if it doesn't exist:</p>
<pre><code class="language-bash">aws ecr describe-repositories --repository-names $repo_name \
  || aws ecr create-repository --repository-name $repo_name
</code></pre>
<p>Then you can push.</p>
<pre><code class="language-bash">docker push $myId.dkr.ecr.ca-central-1.amazonaws.com/$repo:$tag
</code></pre>
<p>If you first create the repository, you get an error like this: <code>name unknown: The repository with name 'kolla/ubuntu-source-base' does not exist in the registry with id '...'</code></p>


<p><strong>This post is linked to from the <a href="/openstack.html">OpenStack Deep Dive Project</a></strong></p>
<hr>
<p><a href="https://docs.openstack.org/cinder/latest/">OpenStack Cinder</a> needs some
special software installed to work with certain storage backends.</p>
<p><a href="https://docs.openstack.org/kolla/latest/">Kolla</a> builds Docker containers for
OpenStack.</p>
<p>This post documents the steps to customize how Kolla builds containers. In
particular, these steps add the Pure Storage plugin to the Cinder-Volume
service's Docker image.</p>
<hr>
<h1 id="install-kolla">Install Kolla</h1>
<p>This example uses the Rocky checkout to make Rocky based images. Use the stable
branch matching your OpenStack version else things get weird.</p>
<pre><code class="language-bash">apt-get install -y git python python-pip
git clone https://github.com/openstack/kolla.git
get checkout stable/rocky
cd kolla
pip install .
</code></pre>
<h2 id="configure-kolla">Configure Kolla</h2>
<p>Use tox's <code>genconfig envlist</code> to generate a commented config file:</p>
<pre><code class="language-bash">tox -e genconfig
mkdir -p /etc/kolla
mv etc/kolla/kolla-build.conf /etc/kolla/
</code></pre>
<p>Then make any changes to the new config file as needed.</p>
<hr>
<h1 id="add-the-modifications">Add the Modifications</h1>
<p>Here's an example of adding the Pure Storage's plugin to the cinder_volume
image.</p>
<p><code>vi /etc/kolla/template-overrides.j2</code></p>
<pre><code class="language-jinja2">{% extends parent_template %}

# Cinder Volume
{% block cinder_volume_ubuntu_setup %}
RUN pip install purestorage
{% endblock %}
</code></pre>
<hr>
<h1 id="build-the-image-w-modifications">Build the Image w/ Modifications</h1>
<pre><code class="language-bash">kolla-build \
  --tag rocky-test-20190826-01 \
  --template-override /etc/kolla/template-overrides.j2  \
  cinder-volume
</code></pre>
<p>That's it. You can now use this image with the included plugin.</p>


<p><strong>This post is linked to from the <a href="/project-email.html">Automated Emails Project</a></strong></p>
<h1 id="what-is-spf">What is SPF?</h1>
<p>Sender Policy Framework</p>
<p>SPF uses a DNS attribute assigned to the sending domain name that lists which
hosts are expected to send emails for that domain.</p>
<h1 id="how-to-use-spf">How to use SPF</h1>
<h2 id="find-your-external-ip-address">Find your external IP address</h2>
<p>Many VMs use NAT to accesss the internet. Their <code>ifconfig</code> output won't match
the actual internet IP that the emails will send from.</p>
<p>To check your external IP address, execute:</p>
<pre><code class="language-bash">dig +short myip.opendns.com @resolver1.opendns.com
</code></pre>
<p>Below, the output of this command will be used as <code>&#x3C;ext_ip></code></p>
<p>If you have a static public IP address you can use it as <code>&#x3C;ext_ip></code></p>
<h2 id="set-the-spf-dns-record">Set the SPF DNS Record</h2>
<p>There used to be a site with reference data at openspf.org but they ran out of
funding so it went offline. You can still get to its reference page from the
<a href="http://web.archive.org/web/20190224184030/http://www.openspf.org/SPF_Record_Syntax">WayBackMachine</a>.</p>
<p>To configure SPF, first
go to your domain registrar's website and create a new entry for your domain.
In Cloudflare, you just go to the DNS page. Cloudflare supports both SPF and
TXT records, some registrars will only offer TXT. If yours offers both, use
both for this.</p>
<p>Create a record for the domain your emails will send from. For instance, if
your emails will send from <a href="mailto:noreply@alerts.example.com">noreply@alerts.example.com</a>, create an entry for
alerts.example.com.</p>
<p>Check the above linked reference for other options such as using A records,
but here's an example of an SPF record stating that emails are expected from
the IP address 1.2.3.4. You can also use subnet masks if you own a range.</p>
<pre><code>v=spf ip4:1.2.3.4 -all
</code></pre>
<p>Set your DNS SPF record, and that's it, you're done. Receiving "milters"
might check this and add a few points or whatever towards your sender's
trustworthiness.</p>


<h1 id="setup">Setup</h1>
<p>This setup was done on an <a href="https://www.intel.ca/content/www/ca/en/products/boards-kits/nuc.html">Intel Nuc</a>.</p>
<p>The Nuc is running Ubuntu Server 18.04 Bionic, CLI only. It has a wired
connection to a rack switch on an access port, and a wireless connection to the
office WiFi.</p>
<p>The server will host 192.168.0.1 and act as a gateway for the wired servers.
It will do NAT just like a home router does to ensure there are no routing
problems with the upstream layer 3 devices.</p>
<p>This setup begins where my <a href="ubuntu-bionic-wifi">Ubuntu WiFi Guide</a> left off.
Check it out if you're not sure how to connect to the WiFi from an Ubuntu
server.</p>
<h1 id="enable-ip-routing">Enable IP Routing</h1>
<p>Check if it's enabled in your config file. This command will just print the
file and ignore any comments or blank lines.</p>
<pre><code>cat /etc/sysctl.conf | grep -v -e "#" -v -e "^$"
</code></pre>
<p>If you don't have this entry, edit the file and insert it:</p>
<pre><code>net.ipv4.ip_forward=1
</code></pre>
<p>If you changed the file, apply your changes:</p>
<pre><code>sysctl -p
</code></pre>
<h1 id="configure-interfaces">Configure Interfaces</h1>
<p>Here's my netplan, with the wireless details changed:</p>
<pre><code>network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s25:
      dhcp4: false
      addresses:
        - 192.168.0.1/24
  wifis:
    wlp2s0:
      dhcp4: true
      access-points:
        "My SSID":
          password: "My Password"
</code></pre>
<p>If you changed your netplan file, apply the changes with</p>
<pre><code>netplan try
</code></pre>
<h1 id="configure-nat">Configure NAT</h1>
<p>We'll use <code>iptables</code> for NAT.</p>
<p>First, make a file that defines the rules you want:</p>
<h2 id="define-iptables-rules-file">Define iptables rules file</h2>
<p>These rules will take any traffic coming from 192.168.0.0/24 that are being
routed out the WiFi interface <code>wlp2s0</code> (the default gateway / default route),
and apply the MASQUERADE rule which will run port address translation, allowing
internet.</p>
<p>There are two tables, <code>nat</code> and <code>filter</code>. The NAT table does the translation,
the filter table defines what's allowed in and out. We need to specify that
traffic is allowed to go:</p>
<ul>
<li>From the Ethernet port to the WiFi port</li>
<li>Back from WiFi to Ethernet, when it is <code>RELATED</code> or <code>ESTABLISHED</code></li>
</ul>
<p><code>vi /etc/iptables_rules.sh</code></p>
<pre><code>#!/usr/bin/env bash
echo "Loading rules..."
iptables -t filter -A FORWARD -i enp0s25 -o wlp2s0 -j ACCEPT
iptables -t filter -A FORWARD -i wlp2s0 -o enp0s25 -m state \
  --state RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o wlp2s0 -j MASQUERADE
echo "Done"
</code></pre>
<p>Make the script executable</p>
<pre><code class="language-bash">chmod +x /etc/iptables_rules.sh
</code></pre>
<h2 id="load-iptables-rules-now">Load iptables rules now</h2>
<p>Run the new file to apply the rules.</p>
<pre><code class="language-bash">/etc/iptables_rules.sh
</code></pre>
<p>You can now use this device as a router without having to configure any special
routes.</p>
<h2 id="configure-iptables-rules-to-load-on-boot">Configure iptables rules to load on boot</h2>
<p>Add the following line to your rc.local file. If it doesn't exist, create it.</p>
<p><code>vi /etc/rc.local</code></p>
<pre><code>/etc/iptables_rules.sh
</code></pre>
<p>If the file didn't exist, make sure that it starts with a shebang:</p>
<pre><code>#!/usr/bin/env bash
</code></pre>
<p>Ensure that it's executable:</p>
<pre><code>chmod +x /etc/rc.local
</code></pre>


<h1 id="setup">Setup</h1>
<p>This setup was done on an <a href="https://www.intel.ca/content/www/ca/en/products/boards-kits/nuc.html">Intel Nuc</a>.</p>
<p>The Nuc is running Ubuntu Server 18.04 Bionic, CLI only. It has a wired
connection to a DHCP-enabled home router initially, but will use wireless
internet after this in finished</p>
<h1 id="find-and-start-the-wireless-device">Find and Start the Wireless Device</h1>
<pre><code class="language-bash">ip link
</code></pre>
<p>Mine was called <code>wlp2s0</code>. Start it:</p>
<pre><code class="language-bash">ip link set dev wlp2s0 up
</code></pre>
<hr>
<h1 id="download--install-wireless-software">Download &#x26; Install wireless software</h1>
<p>Ubuntu server doesn't ship with the required packages to connect to WiFi.</p>
<p>From another Ubuntu server that <strong>does</strong> have internet, plug in a USB stick
and mount it. This example assumes it's mounted to /mnt.</p>
<p>I'm pretty sure that it can connect to WEP without this, and its just the
newer WPA2 networks that basically everyone is using that need this.</p>
<pre><code class="language-bash"># Delete anything currently in the apt cache
apt-get clean

# building some temp package lists, drop them in ~ or wherever
cd ~

# use apt-rdepends to get the dependencies
apt-get install apt-rdepends

# find deps for wireless-tools
apt-rdepends wireless-tools 2>/dev/null | grep "Depends: " | awk '{print $2}' | sort | uniq > packages1
echo wireless-tools >> packages1

# find deps for wpasupplicant
apt-rdepends wpasupplicant 2>/dev/null | grep "Depends: " | awk '{print $2}' | sort | uniq > packages2
echo wpasupplicant >> packages2
cat packages1 packages2 | sort | uniq > packages

# download the debs
cd /var/cache/apt/archives
cat ~/packages | while read p; do echo $p; apt-get download $p; done

# Copy the files
mkdir -p /mnt/debs
cp /var/cache/apt/archives/*.deb /mnt/debs
umount /mnt
</code></pre>
<p>Now remove the USB drive from that system and plug it into the offiline one.</p>
<pre><code class="language-bash"># mount the usb drive
mount /dev/sdb1 /mnt

# move the files to your cache directory
cp /mnt/debs/* /var/cache/apt/archives/

# Install the packages
apt-get install wireless-tools wpasupplicant
</code></pre>
<h1 id="scan-for-ssids">Scan for SSIDs</h1>
<p>The <code>iwlist</code> command can also give you other information about the network if
you don't pipe its output to grep, but it's pretty verbose.</p>
<pre><code class="language-bash">iwlist wlp2s0 scanning | grep -ie ssid
</code></pre>
<h1 id="join-the-wifi-network-with-netplan">Join the WiFi Network with Netplan</h1>
<p>You can use various commands to interactively connect to the network, but I
want this connection to come up with the NUC. Netplan is the new tool used to
configure networks since Ubuntu's Bionic stable release. Here's the netplan
config I used. You can see the wired connection in there too, it doesn't need
to be there once this is finished.</p>
<p>Be sure to replace "<code>MySSID</code>" and "<code>My Password</code>" with your own.</p>
<p><code>vi /etc/netplan/01-netcfg.yaml</code></p>
<pre><code>network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s25:
      dhcp4: yes
  wifis:
    wlp2s0:
      dhcp4: yes
      access-points:
        "MySSID":
          password: "My Password"
</code></pre>
<p>Then apply the changes.</p>
<pre><code>netplan try
</code></pre>
<p>You should have an IP address now in <code>ifconfig</code>. For some reason I didn't get
one right away, so I forced the DHCP request like this:</p>
<pre><code>dhclient wlp2s0 -v
</code></pre>
<p>After that, you can connect to the wireless IP and don't need the wired one.</p>
<h1 id="reference-links">Reference Links</h1>
<ul>
<li><a href="https://askubuntu.com/questions/138472/how-do-i-connect-to-a-wpa-wifi-network-using-the-command-line">AskUbuntu post about wpasupplicant</a></li>
<li><a href="https://ubuntuforums.org/showthread.php?t=2392154">Ubuntuforums.org netplan post</a></li>
</ul>


Kyle Pericak

"It works in my environment"

Categories

Tags