<p>How to run a docker container on AWS ECS (Elastic Compute Cloud)  using the AWS
web console. A container from the Elastic Container Registry will be launched.
It will run on Amazon's <a href="https://aws.amazon.com/fargate/">Fargate Container-as-a-Service</a>.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#launch-a-container-on-fargate">Launch a container on Fargate</a></li>
<li><a href="#publish-containers-network-to-the-internet">Publish container's network to the internet</a></li>
<li><a href="#using-a-custom-domain-name">Using a custom domain name</a></li>
</ul>
<h1 id="launch-a-container-on-fargate">Launch a container on Fargate</h1>
<ol>
<li>Upload an image to AWS ECR. <a href="/aws-ecr.html">See my guide on using AWS ECR</a>.</li>
<li>Open <a href="https://ca-central-1.console.aws.amazon.com/ecs/">AWS ECS</a></li>
<li>Under Container Definition, select Configure</li>
<li>Fill in the form
<ol>
<li>For the image URL, use the URL of the image uploaded to
ECR. I was testing a reverse proxy image, so it looked like this:
<code>000000000000.dkr.ecr.ca-central-1.amazonaws.com/test/revproxy:latest</code></li>
<li>I have my nginx reverse proxy a 256MB soft limit for RAM</li>
<li>For my reverse proxy I made port mappings for port 80 and 443</li>
<li>No need to fill out any of the advanced fields</li>
</ol>
</li>
<li>Fill out the Task Definition section, click Next</li>
<li>Define your service
<ol>
<li>Without a load-balancer, the only way to access your container will be by
using <a href="https://aws.amazon.com/blogs/compute/access-private-applications-on-aws-fargate-using-amazon-api-gateway-privatelink/">PrivateLink</a>.</li>
<li>During the initial cluster create wizard you can choose an Application
Load Balancer. That seems to work well.</li>
</ol>
</li>
<li>Name the Fargate cluster, next</li>
<li>Click create</li>
</ol>
<p>It will take a little while. Once its finished you can click on View Service.</p>
<hr>
<h1 id="publish-containers-network-to-the-internet">Publish container's network to the internet</h1>
<p>Navigate to the newly created service and go to the Details tab. From there you
can see a Load Balancing section. It should have a link to the new load
balancer.</p>
<p>You can also access this page from the left nav-bar by going to Load Balancing,
then choosing the new load balancer.</p>
<p>In the Description tab here, you can find the DNS name that AWS has assigned
to this load balancer. Unfortunately you can't depend on the IP address behind
this FQDN to stay the same.</p>
<hr>
<h1 id="using-a-custom-domain-name">Using a custom domain name</h1>
<p>I'm working on that. Its not intuitive. If you know how, I'd like to hear about
it..</p>


<p>Docker supports shared memory. The default is only 64MB though which is TINY.</p>
<p>The small size became a problem during a containerized OpenStack upgrade.</p>
<p>This will be a really short post because I haven't done much playing with the
shared memory yet, I just think its really neat.</p>
<hr>
<h1 id="change-the-devshm-size-on-the-system">Change the /dev/shm size on the system</h1>
<p>This lets you change the size of shared memory on a host.</p>
<pre><code class="language-bash">mount -o remount,size=1G /dev/shm
</code></pre>
<hr>
<h1 id="update-docker-daemons-default-shared-memory-size">Update Docker Daemon's Default Shared Memory Size</h1>
<p>This will update the default across the whole Docker service.</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
  "default-shm-size": "512M"
}
</code></pre>
<hr>
<h1 id="set-a-containers-shared-memory-size">Set a Containers Shared Memory Size</h1>
<p>Here's how to change it for a specific container at runtime.</p>
<pre><code class="language-bash">docker run -it --shm-size=512M
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This page covers how to do some things in Git.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#use-vim-for-commits">Use Vim for commits</a></li>
<li><a href="#rename-branch">Rename Branch</a></li>
<li><a href="#roll-back-merge">Roll Back Merge</a>
<ul>
<li><a href="#option-1-abort-the-merge">Option 1: Abort the merge</a></li>
<li><a href="#option-2-reset-to-the-last-commit">Option 2: Reset to the last commit</a></li>
</ul>
</li>
<li><a href="#delete-a-file-from-git-history">Delete a File From Git History</a>
<ul>
<li><a href="#if-you-dont-push-with---force">If you don't push with --force</a></li>
</ul>
</li>
</ul>
<h1 id="use-vim-for-commits">Use Vim for commits</h1>
<p>Ugh, Nano. Kidding, but seriously here's how to use Vim instead.</p>
<pre><code class="language-bash">git config --global core.editor "vim"
</code></pre>
<hr>
<h1 id="rename-branch">Rename Branch</h1>
<pre><code class="language-bash"># Check out the branch to rename
git checkout development
# Rename the branch
git branch -m master
# Push the change
git push origin :development master
</code></pre>
<p>If you're using BitBucket and it fails to delete the branch, it might be set
as your MAIN branch. You need to change that before it can be deleted. You can
change it in the settings page.</p>
<hr>
<h1 id="roll-back-merge">Roll Back Merge</h1>
<p>If you do a git merge and its really ugly, you might want to undo that.</p>
<p>First you did the commit</p>
<pre><code class="language-bash">git checkout master
git merge --squash development
</code></pre>
<p>Then you got a TON of <code>CONFLICT</code> messages and a really dirty state.</p>
<h2 id="option-1-abort-the-merge">Option 1: Abort the merge</h2>
<p>This one's my preference. There might be a few <code>~</code> files left over to <code>rm</code>
manually.</p>
<pre><code class="language-bash">git merge --abort
</code></pre>
<h2 id="option-2-reset-to-the-last-commit">Option 2: Reset to the last commit</h2>
<p>First, get the commit right before this one from git log</p>
<pre><code class="language-bash">git log -n 1
</code></pre>
<p>Copy the commit hash</p>
<pre><code class="language-bash">git reset --hard &#x3C;commit hash>
</code></pre>
<p>You may need to clean up some files after that.</p>
<p><strong>extra</strong>
Usually, the next thing you wanna do is try the merge again with
<code>--stattheirs</code>, or</p>
<pre><code class="language-bash">git merge --squash --strategy-option=theirs development
</code></pre>
<hr>
<h1 id="delete-a-file-from-git-history">Delete a File From Git History</h1>
<p>It's important to note, as GitHub highlights <a href="https://help.github.com/en/articles/removing-sensitive-data-from-a-repository">in their documentation</a>,
if you've uploaded a secret password or something, you still need to change it.</p>
<p>The file I removed was called <code>sqldata.json</code>, and it was in my project root.
From the project root, I ran:</p>
<pre><code class="language-bash">git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch sqldata.json" \
  --prune-empty --tag-name-filter cat -- --all

git push origin --force --all
</code></pre>
<h2 id="if-you-dont-push-with---force">If you don't push with --force</h2>
<p>When I tried to push, though, I got an error. I'd forgotten to use <code>--force</code>.</p>
<pre><code class="language-bash">To https://github.com/kylep/jsc2f.git
 ! [rejected]        master -> master (non-fast-forward)
 error: failed to push some refs to 'https://github.com/kylep/jsc2f.git'
 hint: Updates were rejected because the tip of your current branch is behind
 hint: its remote counterpart. Integrate the remote changes (e.g.
 hint: 'git pull ...') before pushing again.
 hint: See the 'Note about fast-forwards' in 'git push --help' for details.
</code></pre>
<p>Then, when I tried to pull, I got</p>
<pre><code class="language-bash">From https://github.com/kylep/jsc2f
 + ca7d7cc...710d37b master     -> origin/master  (forced update)
 fatal: refusing to merge unrelated histories
</code></pre>
<p>If you're getting errors like that, maybe you need to run push with <code>--force</code>.</p>
<pre><code class="language-bash">git push origin --force --all
</code></pre>


<p><strong>This post is linked to from the <a href="/open-source-cli-project.html">FOSS Python App Project</a></strong></p>
<hr>
<p>At a high level, the steps to upload a package to PyPi are as follows:</p>
<ol>
<li>Write the project code</li>
<li>Create a release on GitHub and put it in the <code>download_url</code> of setup.py</li>
<li>Write a setup.py file to build and describe the project</li>
<li>Write a setup.cfg file pointing to the readme</li>
<li>create a source distribution</li>
</ol>
<h1 id="write-the-project-code">Write the project code</h1>
<p>I'm not going to cover how to build a python project in this post, but here's
the very small and simple project I used when learning this procedure:
<a href="https://github.com/kylep/jsc2f">GitHub: kylep/jsc2f</a>.</p>
<hr>
<h1 id="create-a-release-on-github">Create a Release on GitHub</h1>
<p>Log into your GitHub project and click on releases. If you can't find the link,
you can also just append <code>/releases</code> to the end of the URL to get there.</p>
<p>Click "Create a new release". Enter your tag and branch, along with a release
title and description.</p>
<p>Under the Assets section of the new release there's a (tar.gz) link to download
the source code. If you right click it and copy the URL, that's used next in
the setup.py file.</p>
<hr>
<h1 id="write-a-setuppy-file">Write a setup.py file</h1>
<p>Here's mine. Check GitHub to see if I've changed it.
The <code>download_url</code> value comes from the GitHub release.</p>
<pre><code class="language-python">from setuptools import setup

# Without this you'll get no description on the pypi site
with open("README.md", "r") as fh:
    long_description = fh.read()

setup(
    name='jsc2f',
    packages=['jsc2f'], # Alternatively you can use find_packages()
    version='0.1', # match the github release you make later
    license='MIT', # match the LICENSE file in the project
    description='Saves a JSON fields SQLs cell to a file, or UPDATE it back',
    author='Kyle Pericak',
    author_email='kyle@pericak.com',
    url='kyle.pericak.com/jsc2f', # either your own page or github
    download_url='https://github.com/kylep/jsc2f/archive/v0.1.tar.gz',
    keywords=['SQL', 'JSON', 'file'],
    install_requires=[
        'click',
        'mysql-connector'
    ],
    entry_points='''
        [console_scripts]
        jsc2f=jsc2f.cli:cli
    ''',
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Information Technology',
        'Topic :: Database',
        'Programming Language :: Python :: 3',
        'License :: OSI Approved :: MIT License',
        'Natural Language :: English'

</code></pre>
<h2 id="picking-a-topic">Picking a topic:</h2>
<p>I used <a href="https://pypi.org/pypi?%3Aaction=list_classifiers">this classifier list</a></p>
<hr>
<h1 id="write-setupcfg">Write setup.cfg</h1>
<p>This goes in the project root next to setup.py.</p>
<p><code>vi setup.cfg</code></p>
<pre><code class="language-ini">[metadata]
description-file = README.md
</code></pre>
<h1 id="creating-a-source-distribution">Creating a source distribution</h1>
<p>For this you use the python <a href="https://docs.python.org/2/distutils/sourcedist.html">sdist</a>
tool and <a href="https://pypi.org/project/twine/">twine</a>.</p>
<p>From the git project root:</p>
<pre><code class="language-bash">pip install twine

# This will create dist/
python setup.py sdist

# Twine uploads the dist/ files to Pypi
twine upload --skip-existing dist/*
# Enter your username and password for PyPI
</code></pre>


<p>Often I need to operate on a huge JSON string that's saved in a MaraiDB
database. Here's how I do that.</p>
<p>It needs to be said that raw SQL queries like this are generally a bad idea
compared to something like <a href="https://docs.sqlalchemy.org/en/13/orm/">SQLAlchemy's ORM</a>.
Having said that, this example is way smaller and more portable.</p>
<p>The way this works is you open up an interactive python session, ideally in
a virtualenv. Set the variable values at the top. Run the two functions as
needed. In another window/pane, edit the JSON files.</p>
<p>Another option is to use <code>jq</code> with interactive MySQL commands. I'm not covering
that in this post, though.</p>
<h1 id="environment-setup">Environment setup</h1>
<p>This is best done in a venv if <code>mysql-connector</code> doesn't need to be installed
on the base system.</p>
<pre><code class="language-bash">pip install mysql-connector
</code></pre>
<h1 id="the-code">The code</h1>
<p>It's not a bad idea to write this into a parameterized CLI or whatever.
I just run this in an interactive python shell in a dedicated window/pane.</p>
<pre><code class="language-python">import json
import mysql.connector as mariadb

# Fill in the blanks
db_host= ''  # IP address
db_port = 3306
db_username = ''
db_password = ''
db_name = ''
table_name = ''
column_name = ''
where = ''  # (optional) where = 'WHERE `x` == 1'


def save_to_file(filename):
    """Read the json data from the database, save to filename"""
    conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
    cursor = conn.cursor()
    read_query = 'SELECT {} from {}{}'.format(column_name, table_name, where)
    cursor.execute(read_query)
    data = cursor.fetchone()[0]
    conn.close()
    jdata = json.dumps(json.loads(data), indent=4, sort_keys=True)
    with open(filename, 'w+') as out_file:
        out_file.write(jdata)
    print('done')


def update_from_file(filename):
    """Read data from filename, write to database"""
    with open(filename) as json_file:
        jdata = json.load(json_file)
    data = json.dumps(jdata)
    conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
    cursor = conn.cursor()
    update_query = "UPDATE {} SET {} = '{}'{}".format(table_name, column_name,
                                                      data, where)
    cursor.execute(update_query)
    conn.commit()
    conn.close()
    print('done')
</code></pre>
<h1 id="usage">Usage</h1>
<p>From an interactive python terminal, paste the above script in.</p>
<p>Then to save the file, run something like:</p>
<pre><code class="language-python">save_to_file(filename='/home/kyle/sqldata.json')
</code></pre>
<p>Then you can update the file using an editor <code>vi /home/kyle/sqldata.json</code>.</p>
<p>Back in the Python terminal, you can then push the data back up.</p>
<pre><code class="language-python">update_from_file(filename='/home/kyle/sqldata.json')
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page contains python-related mini-guides and minor posts.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#pylint-ignore-comments">Pylint Ignore Comments</a>
<ul>
<li><a href="#turn-off-linting-for-now">Turn off Linting for now</a></li>
</ul>
</li>
<li><a href="#get-an-object-from-list-of-objects-where-the-object-property-has-value-x">Get an object from list of objects where the object property has value x</a>
<ul>
<li><a href="#a-subset-list-is-expected-list-comprehension">A subset list is expected: list comprehension</a></li>
<li><a href="#get-one-match-next-w-generator-expression">Get one match: next() w/ generator expression</a></li>
</ul>
</li>
<li><a href="#iterate-through-dictionary">Iterate through dictionary</a></li>
<li><a href="#print-json-nicely-in-ascii-no-unicode-u-prefix">Print JSON nicely in ASCII (no unicode 'u' prefix)</a></li>
<li><a href="#replace-a-string-in-a-file">Replace a string in a file</a></li>
<li><a href="#get-environment-variable">Get environment variable</a></li>
<li><a href="#use-jinja2-for-templating">Use Jinja2 for templating</a>
<ul>
<li><a href="#jinja2-templating-with-files">Jinja2 templating with files</a></li>
</ul>
</li>
<li><a href="#execute-sql-query-against-mysql-or-mariadb">Execute SQL query against MySQL or MariaDB</a></li>
<li><a href="#read-from-std-in-bash-pipe">Read from std in (bash pipe)</a></li>
<li><a href="#custom-exceptions">Custom Exceptions</a></li>
<li><a href="#cant-run-pip-after-updating">Can't run pip after updating</a></li>
</ul>
<h1 id="pylint-ignore-comments">Pylint Ignore Comments</h1>
<p>I can never remember the syntax to ignore certain Pylint warnings. Yeah you can
turn them off everywhere but usually I just want to ignore a certain occurrence.</p>
<pre><code class="language-python"># - This file is a mess and I don't want pylint here:
# pylint: disable-all

# - troublesome import not being found:
# pylint: disable=import-error

# - redefined-builtin (W0622):
# pylint: disable=W0622

# - too-many-public-methods (R0904):
# pylint: disable=R0903

# - too-many-arguments (R0913):
# pylint: disable-msg=R0913

# - too-many-locals (R0914)
# pylint: disable=R0914

# - Invalid constant name:
# pylint: disable=C0103

# - Catching too general exception:
# pylint: disable=W0703

# - Using python3 compatible print statement makes python2 linter mad
# pylint: disable=superfluous-parens

# - Ignore too-long line (this needs both the pylint command and the trailing no-qa)
# pylint: disable=line-too-long
binaryValue = b'QVdTOmV5SndZWGxzYjJGa0lqb2lVSFkwWXpKeFNEaHZXVkkxVmtwRmVXdFhXamhKUVRGcGNFWk1kelJKZEV0V1UwSk9'  # noqa: E50

# - Wrong Import Position (Required when putting python version check up top)
# flake8: noqa=402
# pylint: disable=wrong-import-position
</code></pre>
<h2 id="turn-off-linting-for-now">Turn off Linting for now</h2>
<p>In VIM, run:</p>
<pre><code>:ALEToggle
</code></pre>
<h1 id="get-an-object-from-list-of-objects-where-the-object-property-has-value-x">Get an object from list of objects where the object property has value x</h1>
<h2 id="a-subset-list-is-expected-list-comprehension">A subset list is expected: list comprehension</h2>
<p>See <a href="https://docs.python.org/3/tutorial/datastructures.html">Section 5.1.3 - List comprehension</a></p>
<pre><code class="language-python"># This will return a list of objects whith a value
[obj for obj in haystack_list if obj.value == search]

# Example of list comprehension to get the even numbers in a list
nums = [1, 2, 3, 4, 5, 6, 7, 8]
even_nums = [num for num in nums if num % 2 == 0]
</code></pre>
<h2 id="get-one-match-next-w-generator-expression">Get one match: next() w/ generator expression</h2>
<p>The <code>next()</code> method will return the first element found in an iterable.</p>
<p>See <a href="https://www.python.org/dev/peps/pep-0289/">PEP 289 - Generator expressions</a></p>
<pre><code class="language-python"># Raises a StopIteration exception if no match is found
needle = next((obj for obj in haystack_list if obj.value == value))

# To return a value instead of raising an exception, pass a second argument
needle = next((obj for obj in haystack_list if obj.value == search), None)

# Example of what Next is really doing. This returns 1.
next(iter([1,2,3]))
</code></pre>
<hr>
<h1 id="iterate-through-dictionary">Iterate through dictionary</h1>
<pre><code class="language-python">ex_dict = {'a': 1, 'b': 2, 'c': 3}
for key in ex_dict:
    print(key)
    print(ext_dict[key]) # value
</code></pre>
<p>This works because <code>ex_dict.keys() == list(ex_dict)</code></p>
<hr>
<h1 id="print-json-nicely-in-ascii-no-unicode-u-prefix">Print JSON nicely in ASCII (no unicode 'u' prefix)</h1>
<p>This is like using <a href="https://docs.python.org/3/library/pprint.html">pprint</a> on
the JSON object but without the unicode prefixes.</p>
<pre><code class="language-python">data = ... # the json string
jdata = json.dumps(json.loads(data),  indent=4, sort_keys=True)
print(jdata)
</code></pre>
<hr>
<h1 id="replace-a-string-in-a-file">Replace a string in a file</h1>
<p>Though honestly, <a href="https://www.geeksforgeeks.org/sed-command-in-linux-unix-with-examples/">sed</a>
is usually a better option. Or if there are a lot of values, I like to use a
jinja template.</p>
<pre><code class="language-python">file_path = '/home/user/example_file'
find_str = 'REPLACE_THIS'
replace_str = 'WITH_THIS'

# Open the file to read the value
with open(file_path, 'r') as read_file:
    file_content = read_file.read()

# Replace the string
file_content = file_content.replace(find_str, replace_str)

# Write the file
with open(file_path, 'w') as write_file:
    write_file.write(replaced_content)
</code></pre>
<hr>
<h1 id="get-environment-variable">Get environment variable</h1>
<p>I mostly use this with <a href="https://docs.docker.com/engine/reference/builder/">Dockerfile ENV values</a>.</p>
<pre><code class="language-python">import os

environment_var_name = 'MY_ENV_VAR'
environment_var_vlue os.environ[environment_var_name]
</code></pre>
<hr>
<h1 id="use-jinja2-for-templating">Use Jinja2 for templating</h1>
<p>I use this for config files, like <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_templating.html">how Ansible does it</a>.</p>
<pre><code class="language-python">from jinja2 import Template

replacements = {'USERNAME': 'kyle', 'PASSWORD': 'example'}
template_text = 'connection = mysql://{{USERNAME}}:{{PASSWORD}}@127.0.0.1'
template = Template(template_text)
replaced_text = template.render(**replacements)
</code></pre>
<h2 id="jinja2-templating-with-files">Jinja2 templating with files</h2>
<p>Same idea as above but it reads a .j2 file as input and writes it to a config
file. Can combine this nicely with environment variables to write config files
as part of a Docker start script.</p>
<pre><code class="language-python">from jinja2 import Template

def apply_template(jinja2_file, output_file, replacements):
    """Replace the replacements values in jinja2_file, write to output_file"""
    with open(jinja2_file, 'r') as j2_file:
        j2_text = j2_file.read()
    template = Template(j2_text)
    replaced_text = template.render(**replacements)
    with open(output_file, 'w+') as write_file:
        write_file.write(replaced_text)

</code></pre>
<hr>
<h1 id="execute-sql-query-against-mysql-or-mariadb">Execute SQL query against MySQL or MariaDB</h1>
<pre><code class="language-python">import mysql.connector as mariadb

conn = mariadb.connect(host=db_host, port=db_port, user=db_username,
                           password=db_password, database=db_name)
cursor = conn.cursor()
cursor.execute(sql_query)
conn.commit() # if it was a write action
conn.close()
</code></pre>
<hr>
<h1 id="read-from-std-in-bash-pipe">Read from std in (bash pipe)</h1>
<p>For instance I have a file named x that has a big string where I want to
replace commas with spaces. Sed works but I can never remember the syntax.</p>
<pre><code class="language-python">cat x | python -c "import sys; print(sys.stdin.read())"

# Example of using python instead of sed to replace , with spaces
cat x | python -c "import sys; print(sys.stdin.read().replace(',',' '))"

# You can use this in fun ways like combining it with bash loops
cat x | python -c \
  "import sys; print(sys.stdin.read().replace(',','\n'))" | while read line; do
  echo $line
done
</code></pre>
<hr>
<h1 id="custom-exceptions">Custom Exceptions</h1>
<pre><code class="language-python">class MyCustomException(Error):
    pass
</code></pre>
<hr>
<h1 id="cant-run-pip-after-updating">Can't run pip after updating</h1>
<p>So you ran into a problem because ubuntu ships an ancient version of pip, and
some dependency package like <code>subprocess32</code> is breaking your install. A forum
post says you should update pip.</p>
<pre><code class="language-bash">pip install -U pip
</code></pre>
<p>Except now pip won't run at all.</p>
<pre><code class="language-bash">pip install python-openstackclient
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in &#x3C;module>
    from pip import main
ImportError: cannot import name main
</code></pre>
<p>I think maybe the pip executable file moved after the update.
To fix it:</p>
<pre><code class="language-bash">hash -r pip
</code></pre>
<p>Now pip works again.</p>
<p>As an aside, if <code>subprocess32</code> really was your problem, updating pip didn't
help. For some reason you have to use apt to install <code>python-subprocess32</code>.</p>


<p>This guide covers how I set up a dedicated modded Minecraft Feed The Beast
server on spare laptop so my wife and I could play together.</p>
<p><strong>NOTE</strong>: Ubuntu 18.04 doesn't work at all. It uses a newer version of Java
which doesn't work with Minecraft's launcher.</p>
<hr>
<h1 id="server-setup">Server Setup</h1>
<h2 id="make-the-vm">Make the VM</h2>
<p>I'm just using a laptop with Windows 8.0 and Hyper-V for this. It has 16GB RAM,
4 Cores + Hyper-threading, and a 1TB SSD.</p>
<p>Here are the server VM specs I used:</p>
<ul>
<li>Operating System: Ubuntu 16.04 Server</li>
<li>RAM: 12GB</li>
<li>Disk: 200GB (Thin provisioned)</li>
</ul>
<p>Also, set a static IP address.</p>
<h2 id="download-the-modded-server-code">Download the Modded Server Code</h2>
<p>Open your browser to <a href="https://www.curseforge.com/minecraft/modpacks/ftb-ultimate-reloaded">the mod site</a>
and find the link to the <a href="https://www.feed-the-beast.com/projects/ftb-ultimate-reloaded/files">server files</a>.</p>
<p>You can download the server files from your browser and SCP it over or download
them to the server directly with wget:</p>
<pre><code class="language-bash">mkdir -p ~/ftb
cd ~/ftb
wget https://media.forgecdn.net/files/2778/970/FTBUltimateReloadedServer_1.9.0.zip

</code></pre>
<p>Unzip the server file archive</p>
<pre><code class="language-bash">apt-get update
apt-get install -y unzip
unzip FTBUltimateReloadedServer_1.9.0.zip
</code></pre>
<h2 id="configure-the-server">Configure the Server</h2>
<p><code>vi settings.sh</code></p>
<pre><code class="language-bash"># Set the RAM to 10GB
export MAX_RAM="10240M"
</code></pre>
<h2 id="install-java-jre">Install Java JRE</h2>
<pre><code class="language-bash">apt-get install -y default-jre
</code></pre>
<h2 id="launch-the-server">Launch the Server</h2>
<p>I like to console into the server and start a Tmux session, then SSH to it
and join that session to keep the server running. You can also just make a
systemd unit if you like.</p>
<p>From the <code>~/ftb</code> folder start the server interactively:</p>
<pre><code class="language-bash">bash ~/ftb/ServerStart.sh
</code></pre>
<hr>
<h1 id="client-setup">Client Setup</h1>
<h2 id="suggested-option-ftb-launcher">Suggested Option: FTB Launcher</h2>
<p>Download the official FTB launcher from <a href="https://www.feed-the-beast.com/">here</a>.</p>
<p>It's really intuitive, no instructions are needed.</p>
<h2 id="alternate-option-1-multimc">Alternate Option 1: MultiMC</h2>
<p>This option has my favorite UI, but it seems to run into Java problems on
Windows sometimes.</p>
<p>MultiMC works offline, and is strictly about launching Minecraft.
It has some nice features like a window that seems to <code>tail -f</code> the Minecraft
log. It can also set different resource usage values per mod.</p>
<ol>
<li>Download <a href="https://multimc.org/">Multi-MC</a>, install, and launch it.</li>
<li>Click Add Instance</li>
<li>Choose FTB Legacy > FTB Ultimate Reloaded. OK.</li>
<li>Wait for the download to finish</li>
<li>Double-click the new icon with the mod name</li>
<li>If prompted to go to the account manager, say yes</li>
<li>If not in the settings window, open it. Go to Java. Increase the RAM.
It looks like 5120MB works well, with 512 for PermGen.</li>
<li>Close the settings.</li>
<li>Double-click on the installed mod to launch it</li>
</ol>
<h2 id="alternative-option-2-twitch-client">Alternative Option 2: Twitch Client</h2>
<p><strong>Note</strong>: The twitch launcher is a pain. It refuses to go offline and shows you
tons of stuff not related to modding Minecraft.
Unless you already use Twitch for other games or enjoy watching streamers,
I suggest you try MultiMC first.</p>
<p><strong>Update</strong>: This is pretty much discontinued now, definitely avoid it.</p>
<p>From a Windows client OS, go to <a href="https://www.twitch.tv/downloads">twitch's download site</a>
and download the <strong>twitch app for windows</strong>.</p>
<p>Run the installer. You might need to make a Twitch account if you don't have
one, which is a pain but they bought Curse so there's no real better way to
do this.</p>
<ol>
<li>Log in. If you don't have an account, try MultiMC, then make one if it
doesn't work for you.</li>
<li>At the top of the screen, go to Mods.</li>
<li>Click on Minecraft. If its not installed yet, its probably greyed out.</li>
<li>Click Install.</li>
<li>When the install is done you can go to "Browse FTB Modpacks" up top.</li>
<li>Click FTB Ultimate Reloaded</li>
<li>Click Install</li>
</ol>


<p><strong>This post is linked to from the <a href="/aws.html">AWS: Deep Dive Project</a></strong></p>
<hr>
<p>This post covers how to make a Docker registry on AWS Elastic Container
Registry (ECR).
To secure the registry, least-privilege roles are created and assigned to
service account users in the AWS AIM tool</p>
<hr>
<h3 id="table-of-contents">Table of contents</h3>
<ul>
<li><a href="#create-an-aws-ecr-repository">Create an AWS ECR Repository</a></li>
<li><a href="#create-an-aws-iam-user-for-the-registry">Create an AWS IAM User For the Registry</a>
<ul>
<li><a href="#create-groups-for-ecr-access">Create Groups for ECR Access</a>
<ul>
<li><a href="#make-a-group-with-write-access">Make a group with write access</a></li>
<li><a href="#make-a-read-only-group">Make a read-only group</a></li>
</ul>
</li>
<li><a href="#create-users-for-ecr-access">Create Users for ECR Access</a></li>
</ul>
</li>
<li><a href="#connect-docker-to-the-new--registry">Connect Docker to the new  Registry</a>
<ul>
<li><a href="#install-the-aws-cli">Install the AWS CLI</a></li>
<li><a href="#configure-the-aws-cli-user">Configure the AWS CLI USER</a></li>
<li><a href="#get-the-docker-login-command">Get the docker login command</a></li>
</ul>
</li>
<li><a href="#push-images-to-the-ecr-registry">Push Images to the ECR Registry</a></li>
</ul>
<h1 id="create-an-aws-ecr-repository">Create an AWS ECR Repository</h1>
<p>Navigate to the
<a href="https://ca-central-1.console.aws.amazon.com/ecr/repositories">AWS ECR Repositories page</a>
and click Create a repository.</p>
<p>Fill in the form to name your repository.</p>
<h1 id="create-an-aws-iam-user-for-the-registry">Create an AWS IAM User For the Registry</h1>
<p>Its a good idea to have user accounts with write access, and others with
read-only access. This way you can push to the registry with your privileged
user but you never need to expose those credentials outside your development
or CI/CD environment.</p>
<h2 id="create-groups-for-ecr-access">Create Groups for ECR Access</h2>
<p>First, review the AWS Group access levels <a href="https://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr_managed_policies.html">here</a>.</p>
<h3 id="make-a-group-with-write-access">Make a group with write access</h3>
<p>Users in this group can push to existing registries but they can't use the CLI
to make new ones.</p>
<ol>
<li>Go to the <a href="https://console.aws.amazon.com/iam">AWS IAM page</a></li>
<li>Click Groups on the sidebar</li>
<li>Name the group, ex: <code>ecr-write</code></li>
<li>Attach Policy: <code>AmazonEC2ContainerRegistryPowerUser</code>, next</li>
<li>Create Group</li>
</ol>
<h3 id="make-a-read-only-group">Make a read-only group</h3>
<p>Its the same procedure as above, but use the read-only policy instead:
<code>AmazonEC2ContainerRegistryReadOnly</code>.</p>
<p>Consider also making an admin group with the
<code>AmazonEC2ContainerRegistryFullAccess</code> permission.</p>
<h2 id="create-users-for-ecr-access">Create Users for ECR Access</h2>
<p>Make two users, one for pushing and one for pulling. You can share the user
will read access to anyone who needs to consume the images.</p>
<p>To make a user:</p>
<ol>
<li>Go to the <a href="https://console.aws.amazon.com/iam">AWS IAM page</a></li>
<li>Click Users on the sidebar</li>
<li>Add User</li>
<li>Enter user name, check Programmatic Access, next</li>
<li>Choose from the groups you just created to grant read/write access, next</li>
<li>Skip the tags, next</li>
<li>Create User. Be sure to save the secret access key in your password manager.</li>
</ol>
<h1 id="connect-docker-to-the-new--registry">Connect Docker to the new  Registry</h1>
<h2 id="install-the-aws-cli">Install the AWS CLI</h2>
<p>The one from apt is really old and doesn't work. Use pip.</p>
<pre><code class="language-bash">apt-get install -y python python-pip
pip install aws awscli
</code></pre>
<p>Test it out</p>
<pre><code class="language-bash">python -m awscli help
</code></pre>
<h2 id="configure-the-aws-cli-user">Configure the AWS CLI USER</h2>
<p>When prompted, enter your key, secret key, region (ca-central-1), and output
format (none):</p>
<pre><code class="language-bash">aws configure
</code></pre>
<h2 id="get-the-docker-login-command">Get the docker login command</h2>
<p>Use the AWS CLI to fetch and execute the docker login command. You can also
export this command to a file to run it, but there's no point since the token
gets invalidated after 12 hours.</p>
<pre><code class="language-bash">$(aws ecr get-login --no-include-email --region ca-central-1)
</code></pre>
<hr>
<h1 id="push-images-to-the-ecr-registry">Push Images to the ECR Registry</h1>
<p>If you click on the Push Commands button in the ECR registry page you'll get
examples programmatically generated for your registry.</p>
<p>Pushing is kind of unintuitive. Unlike a normal <code>registry:2</code> docker registry,
ECR won't automatically create your repositories for you. You have to manually
create each one. You can use the web UI, or log in as an admin user and run the
following:</p>
<pre><code class="language-bash">aws ecr create-repository --repository-name kolla/ubuntu-source-base
</code></pre>
<p>For scripting, this is a handy way of creating the repo if it doesn't exist:</p>
<pre><code class="language-bash">aws ecr describe-repositories --repository-names $repo_name \
  || aws ecr create-repository --repository-name $repo_name
</code></pre>
<p>Then you can push.</p>
<pre><code class="language-bash">docker push $myId.dkr.ecr.ca-central-1.amazonaws.com/$repo:$tag
</code></pre>
<p>If you first create the repository, you get an error like this: <code>name unknown: The repository with name 'kolla/ubuntu-source-base' does not exist in the registry with id '...'</code></p>


<p><strong>This post is linked to from the <a href="/openstack.html">OpenStack Deep Dive Project</a></strong></p>
<hr>
<p><a href="https://docs.openstack.org/cinder/latest/">OpenStack Cinder</a> needs some
special software installed to work with certain storage backends.</p>
<p><a href="https://docs.openstack.org/kolla/latest/">Kolla</a> builds Docker containers for
OpenStack.</p>
<p>This post documents the steps to customize how Kolla builds containers. In
particular, these steps add the Pure Storage plugin to the Cinder-Volume
service's Docker image.</p>
<hr>
<h1 id="install-kolla">Install Kolla</h1>
<p>This example uses the Rocky checkout to make Rocky based images. Use the stable
branch matching your OpenStack version else things get weird.</p>
<pre><code class="language-bash">apt-get install -y git python python-pip
git clone https://github.com/openstack/kolla.git
get checkout stable/rocky
cd kolla
pip install .
</code></pre>
<h2 id="configure-kolla">Configure Kolla</h2>
<p>Use tox's <code>genconfig envlist</code> to generate a commented config file:</p>
<pre><code class="language-bash">tox -e genconfig
mkdir -p /etc/kolla
mv etc/kolla/kolla-build.conf /etc/kolla/
</code></pre>
<p>Then make any changes to the new config file as needed.</p>
<hr>
<h1 id="add-the-modifications">Add the Modifications</h1>
<p>Here's an example of adding the Pure Storage's plugin to the cinder_volume
image.</p>
<p><code>vi /etc/kolla/template-overrides.j2</code></p>
<pre><code class="language-jinja2">{% extends parent_template %}

# Cinder Volume
{% block cinder_volume_ubuntu_setup %}
RUN pip install purestorage
{% endblock %}
</code></pre>
<hr>
<h1 id="build-the-image-w-modifications">Build the Image w/ Modifications</h1>
<pre><code class="language-bash">kolla-build \
  --tag rocky-test-20190826-01 \
  --template-override /etc/kolla/template-overrides.j2  \
  cinder-volume
</code></pre>
<p>That's it. You can now use this image with the included plugin.</p>


<p><strong>This post is linked to from the <a href="/project-email.html">Automated Emails Project</a></strong></p>
<h1 id="what-is-spf">What is SPF?</h1>
<p>Sender Policy Framework</p>
<p>SPF uses a DNS attribute assigned to the sending domain name that lists which
hosts are expected to send emails for that domain.</p>
<h1 id="how-to-use-spf">How to use SPF</h1>
<h2 id="find-your-external-ip-address">Find your external IP address</h2>
<p>Many VMs use NAT to accesss the internet. Their <code>ifconfig</code> output won't match
the actual internet IP that the emails will send from.</p>
<p>To check your external IP address, execute:</p>
<pre><code class="language-bash">dig +short myip.opendns.com @resolver1.opendns.com
</code></pre>
<p>Below, the output of this command will be used as <code>&#x3C;ext_ip></code></p>
<p>If you have a static public IP address you can use it as <code>&#x3C;ext_ip></code></p>
<h2 id="set-the-spf-dns-record">Set the SPF DNS Record</h2>
<p>There used to be a site with reference data at openspf.org but they ran out of
funding so it went offline. You can still get to its reference page from the
<a href="http://web.archive.org/web/20190224184030/http://www.openspf.org/SPF_Record_Syntax">WayBackMachine</a>.</p>
<p>To configure SPF, first
go to your domain registrar's website and create a new entry for your domain.
In Cloudflare, you just go to the DNS page. Cloudflare supports both SPF and
TXT records, some registrars will only offer TXT. If yours offers both, use
both for this.</p>
<p>Create a record for the domain your emails will send from. For instance, if
your emails will send from <a href="mailto:noreply@alerts.example.com">noreply@alerts.example.com</a>, create an entry for
alerts.example.com.</p>
<p>Check the above linked reference for other options such as using A records,
but here's an example of an SPF record stating that emails are expected from
the IP address 1.2.3.4. You can also use subnet masks if you own a range.</p>
<pre><code>v=spf ip4:1.2.3.4 -all
</code></pre>
<p>Set your DNS SPF record, and that's it, you're done. Receiving "milters"
might check this and add a few points or whatever towards your sender's
trustworthiness.</p>


<h1 id="setup">Setup</h1>
<p>This setup was done on an <a href="https://www.intel.ca/content/www/ca/en/products/boards-kits/nuc.html">Intel Nuc</a>.</p>
<p>The Nuc is running Ubuntu Server 18.04 Bionic, CLI only. It has a wired
connection to a rack switch on an access port, and a wireless connection to the
office WiFi.</p>
<p>The server will host 192.168.0.1 and act as a gateway for the wired servers.
It will do NAT just like a home router does to ensure there are no routing
problems with the upstream layer 3 devices.</p>
<p>This setup begins where my <a href="ubuntu-bionic-wifi">Ubuntu WiFi Guide</a> left off.
Check it out if you're not sure how to connect to the WiFi from an Ubuntu
server.</p>
<h1 id="enable-ip-routing">Enable IP Routing</h1>
<p>Check if it's enabled in your config file. This command will just print the
file and ignore any comments or blank lines.</p>
<pre><code>cat /etc/sysctl.conf | grep -v -e "#" -v -e "^$"
</code></pre>
<p>If you don't have this entry, edit the file and insert it:</p>
<pre><code>net.ipv4.ip_forward=1
</code></pre>
<p>If you changed the file, apply your changes:</p>
<pre><code>sysctl -p
</code></pre>
<h1 id="configure-interfaces">Configure Interfaces</h1>
<p>Here's my netplan, with the wireless details changed:</p>
<pre><code>network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s25:
      dhcp4: false
      addresses:
        - 192.168.0.1/24
  wifis:
    wlp2s0:
      dhcp4: true
      access-points:
        "My SSID":
          password: "My Password"
</code></pre>
<p>If you changed your netplan file, apply the changes with</p>
<pre><code>netplan try
</code></pre>
<h1 id="configure-nat">Configure NAT</h1>
<p>We'll use <code>iptables</code> for NAT.</p>
<p>First, make a file that defines the rules you want:</p>
<h2 id="define-iptables-rules-file">Define iptables rules file</h2>
<p>These rules will take any traffic coming from 192.168.0.0/24 that are being
routed out the WiFi interface <code>wlp2s0</code> (the default gateway / default route),
and apply the MASQUERADE rule which will run port address translation, allowing
internet.</p>
<p>There are two tables, <code>nat</code> and <code>filter</code>. The NAT table does the translation,
the filter table defines what's allowed in and out. We need to specify that
traffic is allowed to go:</p>
<ul>
<li>From the Ethernet port to the WiFi port</li>
<li>Back from WiFi to Ethernet, when it is <code>RELATED</code> or <code>ESTABLISHED</code></li>
</ul>
<p><code>vi /etc/iptables_rules.sh</code></p>
<pre><code>#!/usr/bin/env bash
echo "Loading rules..."
iptables -t filter -A FORWARD -i enp0s25 -o wlp2s0 -j ACCEPT
iptables -t filter -A FORWARD -i wlp2s0 -o enp0s25 -m state \
  --state RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o wlp2s0 -j MASQUERADE
echo "Done"
</code></pre>
<p>Make the script executable</p>
<pre><code class="language-bash">chmod +x /etc/iptables_rules.sh
</code></pre>
<h2 id="load-iptables-rules-now">Load iptables rules now</h2>
<p>Run the new file to apply the rules.</p>
<pre><code class="language-bash">/etc/iptables_rules.sh
</code></pre>
<p>You can now use this device as a router without having to configure any special
routes.</p>
<h2 id="configure-iptables-rules-to-load-on-boot">Configure iptables rules to load on boot</h2>
<p>Add the following line to your rc.local file. If it doesn't exist, create it.</p>
<p><code>vi /etc/rc.local</code></p>
<pre><code>/etc/iptables_rules.sh
</code></pre>
<p>If the file didn't exist, make sure that it starts with a shebang:</p>
<pre><code>#!/usr/bin/env bash
</code></pre>
<p>Ensure that it's executable:</p>
<pre><code>chmod +x /etc/rc.local
</code></pre>


<h1 id="setup">Setup</h1>
<p>This setup was done on an <a href="https://www.intel.ca/content/www/ca/en/products/boards-kits/nuc.html">Intel Nuc</a>.</p>
<p>The Nuc is running Ubuntu Server 18.04 Bionic, CLI only. It has a wired
connection to a DHCP-enabled home router initially, but will use wireless
internet after this in finished</p>
<h1 id="find-and-start-the-wireless-device">Find and Start the Wireless Device</h1>
<pre><code class="language-bash">ip link
</code></pre>
<p>Mine was called <code>wlp2s0</code>. Start it:</p>
<pre><code class="language-bash">ip link set dev wlp2s0 up
</code></pre>
<hr>
<h1 id="download--install-wireless-software">Download &#x26; Install wireless software</h1>
<p>Ubuntu server doesn't ship with the required packages to connect to WiFi.</p>
<p>From another Ubuntu server that <strong>does</strong> have internet, plug in a USB stick
and mount it. This example assumes it's mounted to /mnt.</p>
<p>I'm pretty sure that it can connect to WEP without this, and its just the
newer WPA2 networks that basically everyone is using that need this.</p>
<pre><code class="language-bash"># Delete anything currently in the apt cache
apt-get clean

# building some temp package lists, drop them in ~ or wherever
cd ~

# use apt-rdepends to get the dependencies
apt-get install apt-rdepends

# find deps for wireless-tools
apt-rdepends wireless-tools 2>/dev/null | grep "Depends: " | awk '{print $2}' | sort | uniq > packages1
echo wireless-tools >> packages1

# find deps for wpasupplicant
apt-rdepends wpasupplicant 2>/dev/null | grep "Depends: " | awk '{print $2}' | sort | uniq > packages2
echo wpasupplicant >> packages2
cat packages1 packages2 | sort | uniq > packages

# download the debs
cd /var/cache/apt/archives
cat ~/packages | while read p; do echo $p; apt-get download $p; done

# Copy the files
mkdir -p /mnt/debs
cp /var/cache/apt/archives/*.deb /mnt/debs
umount /mnt
</code></pre>
<p>Now remove the USB drive from that system and plug it into the offiline one.</p>
<pre><code class="language-bash"># mount the usb drive
mount /dev/sdb1 /mnt

# move the files to your cache directory
cp /mnt/debs/* /var/cache/apt/archives/

# Install the packages
apt-get install wireless-tools wpasupplicant
</code></pre>
<h1 id="scan-for-ssids">Scan for SSIDs</h1>
<p>The <code>iwlist</code> command can also give you other information about the network if
you don't pipe its output to grep, but it's pretty verbose.</p>
<pre><code class="language-bash">iwlist wlp2s0 scanning | grep -ie ssid
</code></pre>
<h1 id="join-the-wifi-network-with-netplan">Join the WiFi Network with Netplan</h1>
<p>You can use various commands to interactively connect to the network, but I
want this connection to come up with the NUC. Netplan is the new tool used to
configure networks since Ubuntu's Bionic stable release. Here's the netplan
config I used. You can see the wired connection in there too, it doesn't need
to be there once this is finished.</p>
<p>Be sure to replace "<code>MySSID</code>" and "<code>My Password</code>" with your own.</p>
<p><code>vi /etc/netplan/01-netcfg.yaml</code></p>
<pre><code>network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s25:
      dhcp4: yes
  wifis:
    wlp2s0:
      dhcp4: yes
      access-points:
        "MySSID":
          password: "My Password"
</code></pre>
<p>Then apply the changes.</p>
<pre><code>netplan try
</code></pre>
<p>You should have an IP address now in <code>ifconfig</code>. For some reason I didn't get
one right away, so I forced the DHCP request like this:</p>
<pre><code>dhclient wlp2s0 -v
</code></pre>
<p>After that, you can connect to the wireless IP and don't need the wired one.</p>
<h1 id="reference-links">Reference Links</h1>
<ul>
<li><a href="https://askubuntu.com/questions/138472/how-do-i-connect-to-a-wpa-wifi-network-using-the-command-line">AskUbuntu post about wpasupplicant</a></li>
<li><a href="https://ubuntuforums.org/showthread.php?t=2392154">Ubuntuforums.org netplan post</a></li>
</ul>


<p>One pain point I often have is SSH forwarding the ports for a bunch of web
services and IPMI/IDRAC/ILO/Whatever connections through my pivot server to my
workstation. Its really tedious, and having a web UI right on the jump server's
subnet can be really helpful.</p>
<p>In this guide, I show how I launch Chrome on a remote Ubuntu server without
ever installing any graphics tools like Ubuntu Desktop on that server, and
without needing VNC/RDP/PCOIP/etc. X11 forwarding lets the XQuarts process on
a MacBook render the browser, while it still uses the networking on the Ubuntu
server it launched from.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#configure-workstation-macbook-ssh-settings-for-x11-forwarding">Configure Workstation (MacBook) SSH Settings for X11 Forwarding</a>
<ul>
<li><a href="#set-xauthlocation">Set XAuthLocation</a></li>
<li><a href="#enable-forwardagent-and-forwardx11">Enable ForwardAgent and ForwardX11</a></li>
</ul>
</li>
<li><a href="#remotely-launch-graphical-apps-with-xwindows">Remotely Launch Graphical Apps with XWindows</a>
<ul>
<li><a href="#install-chrome">Install Chrome</a></li>
<li><a href="#install-x11-apps-for-xeyes">Install X11-apps for xeyes</a></li>
</ul>
</li>
<li><a href="#launch-forwarded-xwindows-session">Launch Forwarded XWindows Session</a>
<ul>
<li><a href="#open-ssh-session-with-x11-forwarding-enabled">Open SSH session with X11 Forwarding Enabled.</a></li>
</ul>
</li>
<li><a href="#troubleshooting">Troubleshooting</a>
<ul>
<li><a href="#xauth--timeout-in-locking-authority-file-xauthority">xauth:  timeout in locking authority file ~/.Xauthority</a></li>
<li><a href="#display-is-empty-no-xauth-program">$DISPLAY is Empty, No xauth program</a></li>
<li><a href="#x11-connection-rejected-because-of-wrong-authentication">X11 connection rejected because of wrong authentication.</a></li>
</ul>
</li>
<li><a href="#reference-links">Reference Links</a></li>
</ul>
<h1 id="configure-workstation-macbook-ssh-settings-for-x11-forwarding">Configure Workstation (MacBook) SSH Settings for X11 Forwarding</h1>
<p>If you're not using a MacBook, this won't apply.</p>
<h2 id="set-xauthlocation">Set XAuthLocation</h2>
<p>The workstation needs to allow X11 forwarding. Some brilliant update broke it
in my version of Mac OS I had to set the XAuthLocation in <code>ssh_config</code>.</p>
<p><code>vi /etc/ssh/ssh_config</code></p>
<pre><code class="language-bash">Host *
  SendEnv LANG LC_*
  XAuthLocation /usr/X11/bin/xauth
  ServerAliveInterval 60
  ForwardX11Timeout 596h
</code></pre>
<h2 id="enable-forwardagent-and-forwardx11">Enable ForwardAgent and ForwardX11</h2>
<p>My hostname was <code>adc-bmc</code> and its IP was 192.168.2.2. Alter this as needed.
On your workstation:</p>
<p><code>vi ~/.ssh/config</code></p>
<pre><code class="language-bash">Host adc-bmc
  HostName 192.168.2.2
  ForwardAgent yes
  ForwardX11 yes
</code></pre>
<h1 id="remotely-launch-graphical-apps-with-xwindows">Remotely Launch Graphical Apps with XWindows</h1>
<h2 id="install-chrome">Install Chrome</h2>
<p>You can use Firefox or whatever instead. Chrome's install will fail due to
missing dependencies, but apt can sort it out.</p>
<pre><code class="language-bash">wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
apt-get install -f
dpkg -i google-chrome-stable_current_amd64.deb
</code></pre>
<h2 id="install-x11-apps-for-xeyes">Install X11-apps for xeyes</h2>
<p>This is just for fun and testing. Xeyes is kind of neat.</p>
<pre><code>apt-get install -y x11-apps
</code></pre>
<hr>
<h1 id="launch-forwarded-xwindows-session">Launch Forwarded XWindows Session</h1>
<h2 id="open-ssh-session-with-x11-forwarding-enabled">Open SSH session with X11 Forwarding Enabled.</h2>
<p>XWindows can be passed through SSH. Open a new terminal window and connect to
the server using the <code>-X</code> flag to enable X forwarding on the client.</p>
<p>The xeyes application is a fun test.</p>
<pre><code class="language-bash">ssh -X myuser@192.168.2.2 xeyes
</code></pre>
<p>If that works, try Chrome. The extra flags enable compression and use a faster
encryption algorithm. If they cause problems, just use <code>-X</code>.</p>
<pre><code class="language-bash">ssh -XC -c chacha20-poly1305@openssh.com myuser@192.168.2.2 google-chrome
</code></pre>
<h1 id="troubleshooting">Troubleshooting</h1>
<h2 id="xauth--timeout-in-locking-authority-file-xauthority">xauth:  timeout in locking authority file ~/.Xauthority</h2>
<p>As your not-root user, remove and <code>chmod</code> some files.</p>
<pre><code class="language-bash">sudo rm ~/.Xauthority-c
sudo rm ~/.Xauthority-l
sudo rm -r ~/.Xauthority
mkdir ~/.Xauthority
chmod 0600 ~/.Xauthority
</code></pre>
<h2 id="display-is-empty-no-xauth-program">$DISPLAY is Empty, No xauth program</h2>
<p>On your mac, if you connect with <code>ssh -v -X ...</code> and you get an error line
saying <code>No xauth program</code>, then you need to edit your SSH config to specify
the xauth path. Mine was <code>/usr/X11/bin/xauth</code>.</p>
<h2 id="x11-connection-rejected-because-of-wrong-authentication">X11 connection rejected because of wrong authentication.</h2>
<p>If you're on Mac, this can happen when you didn't set the <code>XAuthLocation</code> ssh
configuration.</p>
<h1 id="reference-links">Reference Links</h1>
<ul>
<li><a href="https://linuxize.com/post/how-to-install-google-chrome-web-browser-on-ubuntu-18-04/">linuxize.com - Install chrome</a></li>
<li><a href="https://github.com/dnschneid/crouton/issues/2676">github.com - x11 forwarding tshooting</a></li>
</ul>


<p>This post covers how to create a bash script that runs every minute to send an
email when a ping fails.</p>
<p>You could use <a href="https://www.pingdom.com/">Pingdom</a> but there's no need to pay
someone else if you have compute resources available. Also, this works from
inside your local network, so it can ping <a href="https://tools.ietf.org/html/rfc1918">RFC1918</a>
addresses.</p>
<p>This script watches for ICMP (ping) availability.
You can replace the ping command with a netcat call to watch a port, or run
whatever other test you like.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#building-the-alert-tool">Building the Alert Tool</a>
<ul>
<li><a href="#test-the-mail-command">Test the mail command</a></li>
<li><a href="#create-the-bash-script">Create the bash script</a></li>
</ul>
</li>
<li><a href="#create-cron-job">Create cron job</a>
<ul>
<li>
<ul>
<li><a href="#restart-cron-to-apply-the-changes">Restart Cron to apply the changes</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<h1 id="building-the-alert-tool">Building the Alert Tool</h1>
<p>Before you begin, consider <a href="/emails-postfix-ubuntu.html">setting up a local send-only postfix service</a>.</p>
<h2 id="test-the-mail-command">Test the mail command</h2>
<p>Emails in this script will send using the <code>mail</code> command. Make sure it works
before continuing.</p>
<pre><code class="language-bash">echo "Test alert please ignore" | mail \
    -a "FROM:noreply@alerts.example.com" \
    -s "Test alert" \
    alerts@example.com
</code></pre>
<h2 id="create-the-bash-script">Create the bash script</h2>
<p>Create a directory to place the script.</p>
<pre><code class="language-bash">mkdir -p /etc/cronjobs &#x26;&#x26; cd /etc/cronjobs
chmod +x ping_alert.sh
</code></pre>
<p>Create the script. Here's a commented example of one I wrote:</p>
<p><code>vi ping_alert.sh</code></p>
<pre><code class="language-bash">#!/usr/bin/env bash

# Positional arguments. If debug is set to '1', debug will echo to stdout
target_ip=$1
debug_enabled=$2

# debug function for troubleshooting
function debug () {
    if [[ "$debug_enabled" == "1" ]]; then
        echo $1
    fi
}

# The token file will be written when an email has been sent, and ensure that
# another email doesn't get sent again too soon. This helps prevent your emails
# from being marked as SPAM and is also way less annoying.

token_file='/etc/cronjobs/.email_sent'
sent_recently=false
now_ts=$(date +%s)
if [ -f $token_file ]; then
    file_ts=$(cat $token_file)
    # hour_in_seconds defined how long between emails to wait before another
    # can be sent.
    hour_in_seconds=3600
    hour_from_file_ts=$(($file_ts+$hour_in_seconds))
    if [[ $now_ts -le $hour_from_file_ts ]]; then
        sent_recently=true
    fi
fi

# The script will gracefully exit if its within the cooldown window.
if [[ $sent_recently == true ]]; then
    debug "sent recently, exiting. Check $token_file"
    exit
fi

# Ping the target IP and collect the return code. If the return code is 0 then
# the ping was OK and this script will exit.
debug "Pigning $target_ip..."
ping -c 1 $target_ip >/dev/null
if [[ $? == 0 ]]; then
    debug "ping ok, exiting"
    exit
fi

# Since the script hasn't exited, the ping failed. Send the warning email.
debug "Sending warning email"
msg="Warning: Ping to IP $target_ip failed."
echo $msg | mail \
    -a  "FROM:noreply@alert.example.com" \
    -s "Test alert" \
    alerts@example.com

# Update the token file to reset the cooldown.
debug "Writing new timestamp $now_ts"
# Write the new timestamp
echo $now_ts > $token_file
</code></pre>
<p>Make the script executable:</p>
<pre><code class="language-bash">chmod +x ping_alert.sh
</code></pre>
<p>Give the script a test run against a valid and invalid IP. It should send an
email for the invalid IP but not the valid one.</p>
<pre><code># test that should pass
/etc/cronjobs/ping_alert.sh 1.1.1.1
# test that should fail
/etc/cronjobs/ping_alert.sh 10.2.3.4
</code></pre>
<hr>
<h1 id="create-cron-job">Create cron job</h1>
<p>We'll run this script every minute. Since it has a cool-down timer it won't
send spam.</p>
<p>This example will ping <a href="https://new.blog.cloudflare.com/announcing-1111/">Cloudflare's DNS</a>,
<code>1.1.1.1</code>, so replace the IP with one that you actually watch to watch.</p>
<p><code>vi /etc/crontab</code></p>
<pre><code># Keep whatever is currently there
*  *    * * *   root    /etc/cronjobs/ping_alert.sh 1.1.1.1 0 | tee -a /var/log/ping_alert.log
</code></pre>
<h3 id="restart-cron-to-apply-the-changes">Restart Cron to apply the changes</h3>
<pre><code>systemctl restart cron
</code></pre>
<p>Now the job will run every minute, and send emails at most every hour.</p>


<hr>
<h1 id="scenario">Scenario</h1>
<p>You have three machines:
A: a Macbook laptop connected to the internet
B: an Ubuntu 18.04 server on GCP or AWS used as a pivot server.
Lets call it pivot.example.com, assuming it has a DNS A record.
C: an Ubuntu 18.04 Server at a remote datacenter/office with no inbound rules
configured on the firewall there.</p>
<p>You need to administer server C, from your workstation A, but you're not on the
network of C.</p>
<h1 id="solution-ssh-tunnels">Solution: SSH Tunnels</h1>
<p>SSH can tunnel traffic through a server, effectively proxying the connections.
Here's how this works:</p>
<ul>
<li>C connects to B with a <a href="https://www.howtoforge.com/reverse-ssh-tunneling">reverse tunnel</a>,
so that any connection to port 2022 on B will be proxied to 22 on C.</li>
<li>A connects to B with a reverse tunnel so that any connection to 2022 on A
goes to 2022 on B (which then goes to 22 on C)</li>
<li>Connections from A to 127.0.0.1 on port 22022 act as if they were going to
port 22 on C.</li>
</ul>
<p>This can be done for any TCP service, be it SSH, Web traffic, IPMI, so on.</p>
<h2 id="disclaimer">Disclaimer</h2>
<p>Whoever runs the network you're connecting to might not be happy you did this.
I suggest you talk with them first. It could save you some trouble. There are
security implications to opening a "back door" to a network.</p>
<p>Consider asking for a VPN account to the target network.</p>
<h1 id="commands">Commands</h1>
<p>These examples use the above example port of 2022, but you can forward other
ports and even multiple ports at once as needed.</p>
<h2 id="tunnel-from-target-server-to-pivot-server">Tunnel from target server to pivot server</h2>
<p>From <strong>C</strong>, the remote server,
create the reverse tunnel to <strong>B</strong>, the pivot server.</p>
<pre><code class="language-bash">ssh -fNT -R 2022:localhost:22 ubuntu@pivot.example.com
</code></pre>
<h2 id="tunnel-from-workstation-to-pivot-server">Tunnel from workstation to pivot server</h2>
<p>From <strong>A</strong>, your workstation,
Create a tunnel to <strong>B</strong>, the pivot server, so your own workstation listens
for the traffic.</p>
<p>The sudo is used so you can pick lower ports, if desired. It's optional. The
<code>-i</code> argument is because sudo won't use your regular private key file.</p>
<pre><code class="language-bash">sudo ssh -i ~/.ssh/id_rsa -fNT -L 2022:localhost:2022 ubuntu@pivot.example.com
</code></pre>
<h2 id="connection-from-workstation-to-target-server">Connection from workstation to target server</h2>
<p>Now that you have the two tunnels up, you can connect through them.</p>
<p>On your workstation, SSH through the tunnels:</p>
<pre><code class="language-bash">ssh -p 2022 ubuntu@localhost
</code></pre>
<h1 id="a-neat-trick">A Neat Trick</h1>
<p>In my <a href="x11-forwarding-ubuntu18">x11 forwarding guide</a>, I showed how to open
Chrome on a remote Ubuntu server. You can actually do through through the
tunnels you've just made, allowing you to access internal resources at the
target network.</p>
<pre><code class="language-bash">ssh -XC -c chacha20-poly1305@openssh.com -p 2022 ubuntu@localhost google-chrome
</code></pre>


Kyle Pericak

"It works in my environment"

Categories

Tags