<h1 id="install-kvm">Install KVM</h1>
<p>Install the packages</p>
<pre><code class="language-bash">apt-get install -y qemu-kvm libvirt-bin bridge-utils
</code></pre>
<p>Package notes from <a href="https://help.ubuntu.com/community/KVM/Installation">ubuntu.com</a>:</p>
<ul>
<li>libvirt-bin provides libvirtd which you need to administer qemu and kvm instances using libvirt</li>
<li>qemu-kvm (kvm in Karmic and earlier) is the backend</li>
<li>ubuntu-vm-builder powerful command line tool for building virtual machines</li>
<li>bridge-utils provides a bridge from your network to the virtual machines</li>
</ul>
<p>Add users to groups</p>
<pre><code class="language-bash">adduser `id -un` libvirt
adduser `id -un` kvm
</code></pre>
<p>Verify install</p>
<pre><code class="language-bash">virsh list --all
</code></pre>
<hr>
<h1 id="networking">Networking</h1>
<h2 id="configure-network-bridge">Configure network bridge</h2>
<p>Your network interfaces should be configured with NetPlan right now.
Since bridge-utils is installed, you can use NetPlan to do the rest.</p>
<p>Before doing this, it's a good idea to double-check your IPMI/IDRAC/ILO is working.</p>
<pre><code class="language-yaml"># Example netplan config defining bridge
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: no
    eno2:
      dhcp4: no
  bridges:
    br0:
      interfaces: [eno2]
      addresses: ["10.1.0.14/24"]
      gateway4: 10.1.0.1
      mtu: 1500
      nameservers:
        addresses: ["10.1.0.154"]
      parameters:
        stp: true
        forward-delay: 4
      dhcp4: no
      dhcp6: no
</code></pre>
<hr>
<h1 id="defining-the-vm-using-virt-install">Defining the VM using virt-install</h1>
<p>Install virt-install - it's kind of big, about 160MB</p>
<pre><code class="language-bash">apt-get install -y virtinst
</code></pre>
<h2 id="optional---list-os-variant-options">Optional - list os-variant options</h2>
<pre><code class="language-bash">apt-get install -y libosinfo-bin

osinfo-query os
</code></pre>
<h2 id="create-the-vm">Create the VM</h2>
<ul>
<li><code>--disk bus=</code>: 'ide', 'scsi', 'usb', 'virtio' or 'xen'.</li>
</ul>
<h3 id="booting-from-an-existing-qcow2-image">Booting from an existing qcow2 image</h3>
<pre><code class="language-bash">virt-install \
  --name MyImportedVM \
  --description "Imported virtual machine" \
  --graphics vnc,listen=0.0.0.0 \
  --noautoconsole \  
  --os-type=linux \
  --memory 8192 \
  --vcpus=4 \
  --disk path=/var/lib/libvirt/images/importedVM.qcow2,bus=virtio\
  --boot hd \
  --network bridge:br0,model=virtio
</code></pre>
<h3 id="building-a-new-windows-vm">Building a new Windows VM</h3>
<p>Windows can be difficult on KVM because it doesn't ship with Virtio drivers. Using the non-virtio
disk and network emulation is comparatively slow. Things are made harder because <code>virt-install</code>
doesn't let you use --cdrom twice. To mount two iso files at once (so virtio drivers can be installed),
use the --disk argument with device=cdrom. Also be sure to use bus=ide so Windows can read it before
installing the virtio drivers.</p>
<p>The virtio drivers can be downloaded <a href="https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso">here</a></p>
<pre><code class="language-bash">virt-install \
  --name Server2019 \
  --description "Windows Server 2019 Template" \
  --os-type=windows \
  --memory 8192 \
  --vcpus=4 \
  --disk path=/var/lib/libvirt/images/windows2019.qcow2,bus=virtio,size=30 \
  --disk /var/lib/libvirt/isos/WindowsServer2019StandardCore_1909.iso,device=cdrom,bus=ide \
  --disk /var/lib/libvirt/isos/virtio-win.iso,device=cdrom,bus=ide \
  --graphics vnc,listen=0.0.0.0 \
  --noautoconsole \
  --network bridge=br0,model=virtio
</code></pre>
<ul>
<li>When installing the OS through your VNC viewer, no drives will show up.</li>
<li>Click the "Load Driver" button, and browse to the virtio iso that was mounted.</li>
<li>Expand the <code>viostor</code> folder and choose your OS. Click the <code>amd64</code> subdirectory. Click OK.</li>
<li>RedHat VirtIO SCSI Controller should show up, click Next</li>
<li>Finish the install</li>
</ul>
<p>Once the install finishes the server will turn off.</p>
<p>Edit the server using <code>virsh edit</code> to remove the disk:</p>
<pre><code class="language-bash">virsh edit Server2019
</code></pre>
<p>Choose your editor of choice (vim, obviously), and remove the boot <code>cdrom</code> disk.
Leave the virtio one, you stil need it. It looks like this:</p>
<pre><code class="language-xml">    &#x3C;disk type='file' device='cdrom'>
      &#x3C;driver name='qemu' type='raw'/>
      &#x3C;source file='/var/lib/libvirt/isos/WindowsServer2019StandardCore_1909.iso'/>
      &#x3C;target dev='hda' bus='ide'/>
      &#x3C;readonly/>
      &#x3C;address type='drive' controller='0' bus='0' target='0' unit='0'/>
    &#x3C;/disk>
</code></pre>
<p>Turn it back on using <code>virsh start</code>:</p>
<pre><code class="language-bash">virsh start Server2019
</code></pre>
<h4 id="fix-the-other-drivers">Fix the other drivers</h4>
<p>The storage driver is now installed, but the network driver isn't there.
Since the VM was launched with <code>--network bridge=br0,model=virtio</code>, the VirtIO
driver is required. Not using the virtio NIC leads to some pretty terrible performance.</p>
<p>Log into the server and open up the device manager.
Navigate to "other devices". Right click the question marked network driver
and update the driver. Select your virtio disk and allow searching subdirectories,
it should find the driver and install it for you.</p>
<p>Do the same for the unidentified PCI device, that will install the VirtIO Baloon driver.</p>
<p>The server is now ready to use, or be made into a template. If you intend to use this server
with OpenStack later, consider installing cloud-init from <a href="https://cloudbase.it/cloudbase-init/">cloudbase-init</a>.</p>
<hr>
<h1 id="accessing-the-vm-through-the-console">Accessing the VM through the console</h1>
<p>When defining the VM, <code>--graphics vnc,listen=0.0.0.0</code> was used. This enabled a VNC server underneath the
VM which can be used to connect in. To find the connection details, run <code>virsh vncdisplay</code>, which will show
the port offset of that VM's VNC console.</p>
<pre><code class="language-bash">virsh vncdisplay Server2019
</code></pre>
<p>The output will be something like <code>:0</code> or <code>:1</code>. Add 5900 to that, and that's your listen port.
If you needed to use a different port, <code>--graphics</code> supports a <code>,port=</code> argument.</p>
<hr>
<h1 id="exporting-to-glance">Exporting to Glance</h1>
<p>If you want to use this file in OpenStack Glance next, you need to convert it. It's not intuitive, but the file looks like it's basically a thick-provisioned qcow2 (somehow). Use qemu-img:</p>
<pre><code class="language-bash"># thin provision the volume
qemu-img convert -f qcow2 -O qcow2 &#x3C;boot-vol> &#x3C;new-file.qcow2>
# upload it as an image
openstack image create --container-format bare --disk-format qcow2 --public --file &#x3C;new-file.qcow2> "&#x3C;image name>"
</code></pre>


<h1 id="regular-tcpdump">Regular TCPDUMP</h1>
<p>Since tcpdump is installed by default, it's the first thing to use</p>
<pre><code class="language-bash"># Only listen on one interface
tcpdump -i &#x3C;interface/any>

# Filter to a host
tcpdump 'host &#x3C;ip address/fqdn>'

# filter to a port
tcpdump 'tcp port 5000'

# combine filters
tcpdump '(tcp port 5000 or tcp port 35357) and host 192.168.0.100'

# print packets in ascii (basically always want this)
tcpdump -A

</code></pre>
<h1 id="using-tshark">Using tshark</h1>
<p>Another option is wireshark's command line, tshark:</p>
<pre><code class="language-bash">apt-get install tshark
</code></pre>
<p>Note that tshark is really slow to get started compared to tcpdump, you need to wait a good while
until it says something like "Capturing on 'eno1'"</p>
<pre><code class="language-bash"># show interfaces
tshark -D

# Capture everything from a given source and interface
tshark -i eno1 -Y "ip.src == 10.1.0.76"

# Capture HTTP to and from a given source
tshark -i eno1 -Y "ip.addr == 10.1.0.76 and http"
</code></pre>
<hr>
<p><strong>See also:</strong> <a href="/ops-cheatsheet.html">Operations Reference Page</a> for more Linux ops
recipes: vim find-and-replace, passwordless sudo, loopback volumes,
journalctl, and bash exit traps.</p>


<h1 id="install--configure-nfs-server">Install &#x26; Configure NFS Server</h1>
<p>These steps were tested on Ubuntu 18.04.</p>
<pre><code class="language-bash"># Install the NFS server package
apt-get install nfs-kernel-server

# Make a directory to publish
mkdir /nfs

# Set permissions
chown nobody:nogroup /nfs
chmod 777 /nfs
</code></pre>
<p>Configure NFS by writing <code>/etc/exports</code>. In this example I'm allowing access
from anyone on the <code>10.10.0.0/16</code> subnet. This is only secure if you trust that
entire subnet and any systems which might be routed to it.</p>
<pre><code class="language-text">/nfs 10.10.0.0/16(rw,sync,no_subtree_check)
</code></pre>
<p>Apply the changes</p>
<pre><code class="language-bash">exportfs -a
systemctl restart nfs-kernel-server
</code></pre>
<h1 id="nfs-client-setup">NFS client setup</h1>
<pre><code class="language-bash"># Install client software
apt-get install nfs-common

# Create mount point
mkdir /nfs

# Mount the NFS share manually
mount 10.10.10.9:/nfs /nfs
</code></pre>


<p><strong>Note:</strong> This is usually a bad idea. You should really get the team who runs
the CA to send you the certificate in case some <em>bad guy</em> is doing the MITM and
not the local security team. Use this procedure with caution.</p>
<hr>
<h1 id="get-the-signing-certificate">Get the signing certificate</h1>
<p>Use <code>openssl</code> to print the certificate data.</p>
<pre><code class="language-bash">openssl s_client -connect google.com:443 -showcerts
</code></pre>
<p>In the output you'll see, among other things, some certificates.
They look like this:</p>
<pre><code class="language-text">-----BEGIN CERTIFICATE-----
blaBLAbla
-----END CERTIFICATE-----
</code></pre>
<p>Find the one signed by your local CA. Copy and paste it into a new file,
such as <code>example.com.crt</code>. The <code>.crt</code> file extension is required.
Don't copy anything before or after the BEGIN and END lines.</p>
<hr>
<h1 id="trust-the-certificate">Trust the certificate</h1>
<p>As root, move the certificate file to <code>/usr/local/share/ca-certificates</code>,
then run <code>update-ca-certificates</code>.</p>
<pre><code class="language-bash">mv example.com.crt /usr/local/share/ca-certificates
update-ca-certificates
</code></pre>
<p>That's it. Now your system will trust certs signed by that CA too.</p>
<p>If you need to generate a certificate for that CA to sign, see
<a href="/openssl-csr-san.html">Creating a CSR with a SAN using openssl</a>.</p>


<p>You've deployed a server in a customer environment and you don't have remote
network access configured. The customer is OK with you being able to remote
into their network, but its not practical to set you up with a VPN. Enter the
"Dial-Home Device".</p>
<h1 id="what-is-the-dial-home-device">What is the Dial-Home Device</h1>
<p>It's just an Ubuntu server that repeatedly tries to dial an internet connected
jump box in order to open a <a href="/ssh-tunnel.html">reverse SSH tunnel</a>. As soon as
the networking is set up, it will dial home and open itself up for remote
administration.</p>
<p>Also, <a href="https://stargate.fandom.com/wiki/Dial_Home_Device">this.</a></p>
<hr>
<h1 id="deploy-the-pivot-server">Deploy the Pivot Server</h1>
<h2 id="whats-a-pivot-server">What's a Pivot Server?</h2>
<p>This is an internet-connected server such as a VPS that the DHD will SSH to.</p>
<p>You'll SSH to it, then from there you can SSH through the DHD's reverse tunnel.</p>
<h2 id="steps-to-deploy">Steps to Deploy</h2>
<h3 id="launch-a-vps">Launch a VPS</h3>
<p>Using AWS, GCP, Azure, Linode, Digital Ocean, so on, launch an
internet-connected Ubuntu server. It needs to have a static, internet routed IP
address. Its not a bad idea to make a DNS entry for it too, so you can change
the IP down the road.</p>
<p>SSH to the server to confirm that it works.</p>
<h3 id="configure-remote-user">Configure Remote User</h3>
<p>These steps will create a user named <code>remoteuser</code></p>
<pre><code class="language-bash"># Create the user
useradd remoteuser

# Set a password for the user
passwd remoteuser

# Create the home &#x26; ssh directories for the user
mkdir -p /home/remoteuser/.ssh
chown -R remoteuser:remoteuser /home/remoteuser

# Disable shell access to the user, so it can't run commands on the pivot
usermod -s /bin/false remoteuser
</code></pre>
<hr>
<h1 id="deploy-the-dial-home-device">Deploy the Dial-Home Device</h1>
<p>Install Ubuntu Server on a physical server or VM at the remote location. This
server should have network connectivity to whatever resource you want to
manage, and outbound internet access (it can ping google).</p>
<h2 id="create-an-ssh-key-on-the-dhd">Create an SSH key on the DHD</h2>
<p>On the DHD server, you may already have SSH keys. Look in <code>~/.ssh</code>. If not,
run the following to create one. Don't use a passphrase, just hit enter when
prompted.</p>
<pre><code class="language-bash">ssh-keygen
</code></pre>
<p>Collect the public key for the next step:</p>
<pre><code class="language-bash">cat ~/.ssh/id_rsa.pub
</code></pre>
<p>For convenience, I like to copy the key to the root user too. Optional.</p>
<h2 id="authorize-the-ssh-public-key-on-the-pivot-serer">Authorize the SSH Public Key on the Pivot Serer</h2>
<p>Back on the Pivot server, edit the remote user's <code>authorized_keys</code> file to
grant this SSH key-pair access. Paste the SSH key at the end of the file</p>
<pre><code class="language-bash">vi /home/remoteuser/.ssh/authorized_keys
</code></pre>
<h2 id="test-a-remote-connection">Test a remote connection</h2>
<h3 id="initiate-the-connection">Initiate the connection</h3>
<p>From the DHD server, open the reverse tunnel manually to make sure it works.</p>
<ul>
<li>Replace the username as needed</li>
<li>The <code>$pivot</code> variable should be the IP or FQDN of the pivot server</li>
<li>Pick a random port in the range 1024-49151 for the listen port.</li>
</ul>
<pre><code class="language-bash">user="remoteuser"
pivot="support.example.com"
port=22222
ssh -fN -R 0.0.0.0:$port:127.0.0.1:22 $user@$pivot
</code></pre>
<h3 id="connect-through-the-pivot">Connect through the pivot</h3>
<p>From the pivot server, SSH to the DHD. This example assumes your username
is <code>myuser</code>.</p>
<pre><code class="language-bash">ssh -p 22222 myuser@127.0.0.1
</code></pre>
<p>Consider adding your public key to the <code>~/.ssh/authorized_keys</code> file now so
passwords aren't needed.</p>
<p>Now you can jump through the pivot into the DHD.</p>
<hr>
<h1 id="persist-the-connection">Persist the Connection</h1>
<h2 id="build-the-dial-home-command">Build the dial-home command</h2>
<p>From any directory, create the script:</p>
<p><code>vi dial-home</code></p>
<pre><code class="language-bash">#!/usr/bin/env bash

# Check if the reverse tunnel  is already open
port=22222
if [[ $(ps aux | grep ssh | grep $port) ]]; then
  # reverse tunnel is open
  exit 0
fi

# tunnel is not open, open it
user="remoteuser"
pivot="support.example.com"
ssh -fN -R 0.0.0.0:$port:127.0.0.1:22 $user@$pivot
</code></pre>
<p>Make the script executable</p>
<pre><code class="language-bash">chmod +x dial-home
</code></pre>
<p>Move the script to somewhere in the <code>$PATH</code></p>
<pre><code class="language-bash">sudo mv dial-home /usr/local/bin/
</code></pre>
<p>Now the <code>dial-home</code> command will open the tunnel.</p>
<h2 id="make-cron-dial-home-every-minute">Make cron dial-home every minute</h2>
<p>The connection will die for lots of reasons. Make it bring itself back online
automatically using cron.</p>
<p>From the DHD, make sure cron is running (by default, it is)</p>
<pre><code class="language-bash">systemctl status cron
</code></pre>
<p>Edit the crontab</p>
<p><code>sudo vi /etc/crontab</code></p>
<p>and insert this line, where <code>myuser</code> is your username on the DHD.
If you use root, you have to also give root the ssh private key.</p>
<pre><code class="language-text">* * * * *   myuser dial-home
</code></pre>
<p>Reload cron</p>
<pre><code class="language-bash">service cron reload
</code></pre>
<p>Then kill the session (using <code>ps aux</code> and <code>kill</code>) then watch it come back.</p>
<p>Done. Now the server is a dial-home device. It will constantly dial your pivot
server and re-open the session whenever its closed. To lock the server out,
just remove its public key from <code>remoteuser</code>'s <code>authorized_keys</code> file.</p>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page contains operations-related mini-guides and minor posts.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</a></li>
<li><a href="#convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</a></li>
<li><a href="#journalctl-commands">Journalctl commands</a></li>
<li><a href="#get-real-ram-use-of-processes">Get real RAM use of processes</a>
<ul>
<li><a href="#of-ceph">of ceph</a></li>
<li><a href="#of-vms">of VMs</a></li>
</ul>
</li>
<li><a href="#bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</a></li>
<li><a href="#create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</a>
<ul>
<li><a href="#use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</a></li>
</ul>
</li>
<li><a href="#test-mtu-from-ubuntu">Test MTU from Ubuntu</a></li>
<li><a href="#find-big-files-on-mac-os">Find big files on Mac OS</a></li>
<li><a href="#renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</a></li>
<li><a href="#create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</a></li>
<li><a href="#passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</a></li>
<li><a href="#vim-tasks">VIM Tasks</a>
<ul>
<li><a href="#find--replace">Find &#x26; Replace</a></li>
<li><a href="#spell-check-commands">Spell-Check commands</a></li>
</ul>
</li>
<li><a href="#use-an-insecure-docker-registry">Use an Insecure Docker Registry</a></li>
<li><a href="#ansible-ignore-errors">Ansible ignore errors</a></li>
<li><a href="#how-to-tail--f-on-dmesg">How to tail -f on dmesg</a></li>
<li><a href="#ubuntu-downgrade-a-package">Ubuntu Downgrade a package</a></li>
<li><a href="#make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</a></li>
<li><a href="#bash-print-all-but-first-line">Bash: Print all but first line</a></li>
<li><a href="#bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</a></li>
</ul>
<h1 id="find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</h1>
<pre><code>cat /proc/net/bonding/bond0
</code></pre>
<hr>
<h1 id="convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</h1>
<pre><code class="language-bash">openssl pkcs7 -print_certs -in certnew.p7b -out cert.pem
</code></pre>
<p>To generate a CSR with Subject Alternative Names (required to fix
<code>ERR_CERT_COMMON_NAME_INVALID</code>), see
<a href="/openssl-csr-san.html">Creating a CSR with a SAN</a>.</p>
<hr>
<h1 id="journalctl-commands">Journalctl commands</h1>
<p>Watch the logs as they come in:</p>
<pre><code class="language-bash">journalctl -f
</code></pre>
<p>When journalctl logs too much and you need to pair it down:</p>
<pre><code class="language-bash"># to 30 days
journalctl --vacuum-time=30d

# to 500MB
journalctl --vacuum-size=500M
</code></pre>
<p>Show for a specific service</p>
<pre><code class="language-bash">journalctl ceph*
</code></pre>
<hr>
<h1 id="get-real-ram-use-of-processes">Get real RAM use of processes</h1>
<p>You can always use top or htop or whatever. I use this to specifically get the
info about those processes.</p>
<h2 id="of-ceph">of ceph</h2>
<pre><code>ps aux | grep ceph | grep -v -e grep -e qemu -e "\[" | while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); id=$(echo $line | awk '{print $16}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tid: $id,    \tRAM: ($size)"; done
</code></pre>
<h2 id="of-vms">of VMs</h2>
<p>for KVM</p>
<pre><code>ps aux | grep qemu | grep -v -e grep| while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); guest=$(echo $line | awk '{print $13}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tguest: $guest,   \t\tRAM: ($size)" ; done
</code></pre>
<h1 id="bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</h1>
<p>As of Mac Catalina, developing a site using self-signed certs is a bigger pain
than before. Yeah you can go get the key and add it, but that's a nuisance.
It looks like the Chrome team built something they call an
"interstitial bypass keyword" to deal with that. That's one cool feature name.</p>
<p><strong>Obvious disclaimer</strong> If you're not a developer doing this to access your own
site, don't do this.</p>
<p>To bypass the warning, type</p>
<pre><code>thisisunsafe
</code></pre>
<p>Apparently it used to be <code>badidea</code> but they rotated it since it got too well
known. They'll probably do it again someday. I suspect people are using this
on sites they "trust" without really knowing any better and getting hurt by
it.</p>
<hr>
<h1 id="create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</h1>
<p>Useful for simulating a real drive when building a test environment. I use this to test OpenStack in a VM as a Ceph or LVM back-end.</p>
<pre><code class="language-bash"># create empty file of all zeros
sudo dd if=/dev/zero of=/root/virtual-disk bs=1M count=512
# show list of loopback devices. Note its not there
losetup --list
# create the loopback device
losetup /dev/loop0 /root/virtual-disk
# confirm it worked
losetup --list
</code></pre>
<p>You can now treat it like a normal volume. Mount it, put a filesystem on it, whatever. Example:</p>
<pre><code class="language-bash">pvcreate /dev/loop0
vgcreate cinder-volumes /dev/loop0
</code></pre>
<h2 id="use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</h2>
<p>I found <a href="https://unix.stackexchange.com/questions/418322/persistent-lvm-device-with-loopback-devices-by-fstab">this useful systemd unit example</a> that didn't quite work for me, but was close. Here's mine:</p>
<p><code>vi /etc/systemd/system/loops-setup.service</code></p>
<pre><code class="language-text">[Unit]
Description=Setup loopback devices

DefaultDependencies=no
Conflicts=umount.target

Requires=lvm2-lvmetad.service mnt-host.mount
Before=local-fs.target umount.target
After=lvm2-lvmetad.service mnt-host.mount

[Service]
ExecStart=/sbin/losetup /dev/loop0 /root/virtual-disk
ExecStop=/sbin/losetup -d /dev/loop0

RemainAfterExit=yes
Type=oneshot

[Install]
WantedBy=local-fs-pre.target
</code></pre>
<p>Then enable it</p>
<pre><code class="language-bash">systemctl enable loops-setup
</code></pre>
<p>Reboot, then check <code>vgdisplay</code> (if that's your use case). It'll be there.</p>
<hr>
<h1 id="test-mtu-from-ubuntu">Test MTU from Ubuntu</h1>
<p>You would not believe the strange problems that an MTU issue can be the root cause of.</p>
<pre><code class="language-bash">ping -M do -s 1472 &#x3C;remote server>
ping -M do -s 8900 &#x3C;remote server>
</code></pre>
<hr>
<h1 id="find-big-files-on-mac-os">Find big files on Mac OS</h1>
<p>My SSD is always almost full. Here are useful commands for cleaning up:</p>
<pre><code class="language-bash"># Find all the files over 1G
sudo find / -type f -size +1G -exec ls -lh {} \; | awk '{ print $9 ": " $5 }' 2>/dev/null

# List files in Downloads directory by size
ls -lhS ~/Downloads
</code></pre>
<h1 id="renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</h1>
<pre><code class="language-bash">service ntp stop
ntpd -gq
service ntp start
</code></pre>
<hr>
<h1 id="create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</h1>
<pre><code class="language-bash">aws ecr describe-repositories --repository-names $repo_name \
    || aws ecr create-repository --repository-name $repo_name
</code></pre>
<hr>
<h1 id="passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</h1>
<p>Run <code>visudo</code> as root.</p>
<pre><code class="language-bash"># as root
export EDITOR=vim
visudo
</code></pre>
<p>Add the user (replace <code>exampleUser</code>) at the end.</p>
<pre><code class="language-text">exampleUser ALL=(ALL) NOPASSWD: ALL
</code></pre>
<p>For those who don't use vim, a quick reminder:
<code>i</code> enter to insert mode , <code>[esc]</code> to exit insert mode, <code>:wq</code> to save and quit.</p>
<hr>
<h1 id="vim-tasks">VIM Tasks</h1>
<h2 id="find--replace">Find &#x26; Replace</h2>
<pre><code class="language-bash"># replace findThis with ReplaceWithThis
:%s/findThis/ReplaceWithThis/g
</code></pre>
<h2 id="spell-check-commands">Spell-Check commands</h2>
<pre><code class="language-text"># enable/disable
:set spell spelllang=en_ca
:setlocal spell spelllang=en_ca
:set nospell

# next/last
]s
[s

# suggest change
z=

# add/remove word from dictionary
zg
zw`
</code></pre>
<hr>
<h1 id="use-an-insecure-docker-registry">Use an Insecure Docker Registry</h1>
<p>To use a registry with no HTTP cert:</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
    "insecure-registries" : ["myregistrydomain.com:5000"]
}
</code></pre>
<hr>
<h1 id="ansible-ignore-errors">Ansible ignore errors</h1>
<p>The command is <code>ignore_errors: yes</code></p>
<p>Just throw this into the task that's failing. It will still fail, but the play
won't stop.</p>
<p>Example:</p>
<pre><code class="language-yml">- name: Ensure the docker service is running
  script: ../tools/validate-docker-execute.sh
    ignore_errors: yes
</code></pre>
<hr>
<h1 id="how-to-tail--f-on-dmesg">How to tail -f on dmesg</h1>
<pre><code class="language-bash">dmesg -wH
</code></pre>
<hr>
<h1 id="ubuntu-downgrade-a-package">Ubuntu Downgrade a package</h1>
<pre><code class="language-bash"># Find the available versions under Versions:
apt-cache showpkg &#x3C;package>

# Install one of those
apt-get install &#x3C;package>=&#x3C;version>
</code></pre>
<hr>
<h1 id="make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</h1>
<p>Useful for OS that ship with python3 but not python2.
Edit the inventory, find a group of the groups and apply this var.</p>
<pre><code class="language-ini">[baremetal:vars]
ansible_python_interpreter=/usr/bin/python3
</code></pre>
<hr>
<h1 id="bash-print-all-but-first-line">Bash: Print all but first line</h1>
<p>Use <code>tail -n +2</code>.
This was unintuitive to me because +1 seems like it should do it.</p>
<pre><code class="language-bash"># Example
docker images | awk '{print $3}' | tail -n +2
</code></pre>
<hr>
<h1 id="bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</h1>
<p>Use trap.</p>
<ul>
<li><a href="http://redsymbol.net/articles/bash-exit-traps/">This is a nice post covering lots of examples</a>.</li>
<li><a href="https://github.com/GoogleCloudPlatform/endpoints-quickstart/blob/master/scripts/deploy_api.sh">Here's where I first saw it used</a></li>
</ul>
<pre><code class="language-bash">temp_file=$(mktemp)

cleanup() {
  rm "$temp_file"
}

trap cleanup EXIT
</code></pre>


<h1 id="setup">Setup</h1>
<p>This setup was done on an <a href="https://www.intel.ca/content/www/ca/en/products/boards-kits/nuc.html">Intel Nuc</a>.</p>
<p>The Nuc is running Ubuntu Server 18.04 Bionic, CLI only. It has a wired
connection to a rack switch on an access port, and a wireless connection to the
office WiFi.</p>
<p>The server will host 192.168.0.1 and act as a gateway for the wired servers.
It will do NAT just like a home router does to ensure there are no routing
problems with the upstream layer 3 devices.</p>
<p>This setup begins where my <a href="ubuntu-bionic-wifi">Ubuntu WiFi Guide</a> left off.
Check it out if you're not sure how to connect to the WiFi from an Ubuntu
server.</p>
<h1 id="enable-ip-routing">Enable IP Routing</h1>
<p>Check if it's enabled in your config file. This command will just print the
file and ignore any comments or blank lines.</p>
<pre><code>cat /etc/sysctl.conf | grep -v -e "#" -v -e "^$"
</code></pre>
<p>If you don't have this entry, edit the file and insert it:</p>
<pre><code>net.ipv4.ip_forward=1
</code></pre>
<p>If you changed the file, apply your changes:</p>
<pre><code>sysctl -p
</code></pre>
<h1 id="configure-interfaces">Configure Interfaces</h1>
<p>Here's my netplan, with the wireless details changed:</p>
<pre><code>network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s25:
      dhcp4: false
      addresses:
        - 192.168.0.1/24
  wifis:
    wlp2s0:
      dhcp4: true
      access-points:
        "My SSID":
          password: "My Password"
</code></pre>
<p>If you changed your netplan file, apply the changes with</p>
<pre><code>netplan try
</code></pre>
<h1 id="configure-nat">Configure NAT</h1>
<p>We'll use <code>iptables</code> for NAT.</p>
<p>First, make a file that defines the rules you want:</p>
<h2 id="define-iptables-rules-file">Define iptables rules file</h2>
<p>These rules will take any traffic coming from 192.168.0.0/24 that are being
routed out the WiFi interface <code>wlp2s0</code> (the default gateway / default route),
and apply the MASQUERADE rule which will run port address translation, allowing
internet.</p>
<p>There are two tables, <code>nat</code> and <code>filter</code>. The NAT table does the translation,
the filter table defines what's allowed in and out. We need to specify that
traffic is allowed to go:</p>
<ul>
<li>From the Ethernet port to the WiFi port</li>
<li>Back from WiFi to Ethernet, when it is <code>RELATED</code> or <code>ESTABLISHED</code></li>
</ul>
<p><code>vi /etc/iptables_rules.sh</code></p>
<pre><code>#!/usr/bin/env bash
echo "Loading rules..."
iptables -t filter -A FORWARD -i enp0s25 -o wlp2s0 -j ACCEPT
iptables -t filter -A FORWARD -i wlp2s0 -o enp0s25 -m state \
  --state RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o wlp2s0 -j MASQUERADE
echo "Done"
</code></pre>
<p>Make the script executable</p>
<pre><code class="language-bash">chmod +x /etc/iptables_rules.sh
</code></pre>
<h2 id="load-iptables-rules-now">Load iptables rules now</h2>
<p>Run the new file to apply the rules.</p>
<pre><code class="language-bash">/etc/iptables_rules.sh
</code></pre>
<p>You can now use this device as a router without having to configure any special
routes.</p>
<h2 id="configure-iptables-rules-to-load-on-boot">Configure iptables rules to load on boot</h2>
<p>Add the following line to your rc.local file. If it doesn't exist, create it.</p>
<p><code>vi /etc/rc.local</code></p>
<pre><code>/etc/iptables_rules.sh
</code></pre>
<p>If the file didn't exist, make sure that it starts with a shebang:</p>
<pre><code>#!/usr/bin/env bash
</code></pre>
<p>Ensure that it's executable:</p>
<pre><code>chmod +x /etc/rc.local
</code></pre>


<h1 id="setup">Setup</h1>
<p>This setup was done on an <a href="https://www.intel.ca/content/www/ca/en/products/boards-kits/nuc.html">Intel Nuc</a>.</p>
<p>The Nuc is running Ubuntu Server 18.04 Bionic, CLI only. It has a wired
connection to a DHCP-enabled home router initially, but will use wireless
internet after this in finished</p>
<h1 id="find-and-start-the-wireless-device">Find and Start the Wireless Device</h1>
<pre><code class="language-bash">ip link
</code></pre>
<p>Mine was called <code>wlp2s0</code>. Start it:</p>
<pre><code class="language-bash">ip link set dev wlp2s0 up
</code></pre>
<hr>
<h1 id="download--install-wireless-software">Download &#x26; Install wireless software</h1>
<p>Ubuntu server doesn't ship with the required packages to connect to WiFi.</p>
<p>From another Ubuntu server that <strong>does</strong> have internet, plug in a USB stick
and mount it. This example assumes it's mounted to /mnt.</p>
<p>I'm pretty sure that it can connect to WEP without this, and its just the
newer WPA2 networks that basically everyone is using that need this.</p>
<pre><code class="language-bash"># Delete anything currently in the apt cache
apt-get clean

# building some temp package lists, drop them in ~ or wherever
cd ~

# use apt-rdepends to get the dependencies
apt-get install apt-rdepends

# find deps for wireless-tools
apt-rdepends wireless-tools 2>/dev/null | grep "Depends: " | awk '{print $2}' | sort | uniq > packages1
echo wireless-tools >> packages1

# find deps for wpasupplicant
apt-rdepends wpasupplicant 2>/dev/null | grep "Depends: " | awk '{print $2}' | sort | uniq > packages2
echo wpasupplicant >> packages2
cat packages1 packages2 | sort | uniq > packages

# download the debs
cd /var/cache/apt/archives
cat ~/packages | while read p; do echo $p; apt-get download $p; done

# Copy the files
mkdir -p /mnt/debs
cp /var/cache/apt/archives/*.deb /mnt/debs
umount /mnt
</code></pre>
<p>Now remove the USB drive from that system and plug it into the offiline one.</p>
<pre><code class="language-bash"># mount the usb drive
mount /dev/sdb1 /mnt

# move the files to your cache directory
cp /mnt/debs/* /var/cache/apt/archives/

# Install the packages
apt-get install wireless-tools wpasupplicant
</code></pre>
<h1 id="scan-for-ssids">Scan for SSIDs</h1>
<p>The <code>iwlist</code> command can also give you other information about the network if
you don't pipe its output to grep, but it's pretty verbose.</p>
<pre><code class="language-bash">iwlist wlp2s0 scanning | grep -ie ssid
</code></pre>
<h1 id="join-the-wifi-network-with-netplan">Join the WiFi Network with Netplan</h1>
<p>You can use various commands to interactively connect to the network, but I
want this connection to come up with the NUC. Netplan is the new tool used to
configure networks since Ubuntu's Bionic stable release. Here's the netplan
config I used. You can see the wired connection in there too, it doesn't need
to be there once this is finished.</p>
<p>Be sure to replace "<code>MySSID</code>" and "<code>My Password</code>" with your own.</p>
<p><code>vi /etc/netplan/01-netcfg.yaml</code></p>
<pre><code>network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s25:
      dhcp4: yes
  wifis:
    wlp2s0:
      dhcp4: yes
      access-points:
        "MySSID":
          password: "My Password"
</code></pre>
<p>Then apply the changes.</p>
<pre><code>netplan try
</code></pre>
<p>You should have an IP address now in <code>ifconfig</code>. For some reason I didn't get
one right away, so I forced the DHCP request like this:</p>
<pre><code>dhclient wlp2s0 -v
</code></pre>
<p>After that, you can connect to the wireless IP and don't need the wired one.</p>
<h1 id="reference-links">Reference Links</h1>
<ul>
<li><a href="https://askubuntu.com/questions/138472/how-do-i-connect-to-a-wpa-wifi-network-using-the-command-line">AskUbuntu post about wpasupplicant</a></li>
<li><a href="https://ubuntuforums.org/showthread.php?t=2392154">Ubuntuforums.org netplan post</a></li>
</ul>


<p>One pain point I often have is SSH forwarding the ports for a bunch of web
services and IPMI/IDRAC/ILO/Whatever connections through my pivot server to my
workstation. Its really tedious, and having a web UI right on the jump server's
subnet can be really helpful.</p>
<p>In this guide, I show how I launch Chrome on a remote Ubuntu server without
ever installing any graphics tools like Ubuntu Desktop on that server, and
without needing VNC/RDP/PCOIP/etc. X11 forwarding lets the XQuarts process on
a MacBook render the browser, while it still uses the networking on the Ubuntu
server it launched from.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#configure-workstation-macbook-ssh-settings-for-x11-forwarding">Configure Workstation (MacBook) SSH Settings for X11 Forwarding</a>
<ul>
<li><a href="#set-xauthlocation">Set XAuthLocation</a></li>
<li><a href="#enable-forwardagent-and-forwardx11">Enable ForwardAgent and ForwardX11</a></li>
</ul>
</li>
<li><a href="#remotely-launch-graphical-apps-with-xwindows">Remotely Launch Graphical Apps with XWindows</a>
<ul>
<li><a href="#install-chrome">Install Chrome</a></li>
<li><a href="#install-x11-apps-for-xeyes">Install X11-apps for xeyes</a></li>
</ul>
</li>
<li><a href="#launch-forwarded-xwindows-session">Launch Forwarded XWindows Session</a>
<ul>
<li><a href="#open-ssh-session-with-x11-forwarding-enabled">Open SSH session with X11 Forwarding Enabled.</a></li>
</ul>
</li>
<li><a href="#troubleshooting">Troubleshooting</a>
<ul>
<li><a href="#xauth--timeout-in-locking-authority-file-xauthority">xauth:  timeout in locking authority file ~/.Xauthority</a></li>
<li><a href="#display-is-empty-no-xauth-program">$DISPLAY is Empty, No xauth program</a></li>
<li><a href="#x11-connection-rejected-because-of-wrong-authentication">X11 connection rejected because of wrong authentication.</a></li>
</ul>
</li>
<li><a href="#reference-links">Reference Links</a></li>
</ul>
<h1 id="configure-workstation-macbook-ssh-settings-for-x11-forwarding">Configure Workstation (MacBook) SSH Settings for X11 Forwarding</h1>
<p>If you're not using a MacBook, this won't apply.</p>
<h2 id="set-xauthlocation">Set XAuthLocation</h2>
<p>The workstation needs to allow X11 forwarding. Some brilliant update broke it
in my version of Mac OS I had to set the XAuthLocation in <code>ssh_config</code>.</p>
<p><code>vi /etc/ssh/ssh_config</code></p>
<pre><code class="language-bash">Host *
  SendEnv LANG LC_*
  XAuthLocation /usr/X11/bin/xauth
  ServerAliveInterval 60
  ForwardX11Timeout 596h
</code></pre>
<h2 id="enable-forwardagent-and-forwardx11">Enable ForwardAgent and ForwardX11</h2>
<p>My hostname was <code>adc-bmc</code> and its IP was 192.168.2.2. Alter this as needed.
On your workstation:</p>
<p><code>vi ~/.ssh/config</code></p>
<pre><code class="language-bash">Host adc-bmc
  HostName 192.168.2.2
  ForwardAgent yes
  ForwardX11 yes
</code></pre>
<h1 id="remotely-launch-graphical-apps-with-xwindows">Remotely Launch Graphical Apps with XWindows</h1>
<h2 id="install-chrome">Install Chrome</h2>
<p>You can use Firefox or whatever instead. Chrome's install will fail due to
missing dependencies, but apt can sort it out.</p>
<pre><code class="language-bash">wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
apt-get install -f
dpkg -i google-chrome-stable_current_amd64.deb
</code></pre>
<h2 id="install-x11-apps-for-xeyes">Install X11-apps for xeyes</h2>
<p>This is just for fun and testing. Xeyes is kind of neat.</p>
<pre><code>apt-get install -y x11-apps
</code></pre>
<hr>
<h1 id="launch-forwarded-xwindows-session">Launch Forwarded XWindows Session</h1>
<h2 id="open-ssh-session-with-x11-forwarding-enabled">Open SSH session with X11 Forwarding Enabled.</h2>
<p>XWindows can be passed through SSH. Open a new terminal window and connect to
the server using the <code>-X</code> flag to enable X forwarding on the client.</p>
<p>The xeyes application is a fun test.</p>
<pre><code class="language-bash">ssh -X myuser@192.168.2.2 xeyes
</code></pre>
<p>If that works, try Chrome. The extra flags enable compression and use a faster
encryption algorithm. If they cause problems, just use <code>-X</code>.</p>
<pre><code class="language-bash">ssh -XC -c chacha20-poly1305@openssh.com myuser@192.168.2.2 google-chrome
</code></pre>
<h1 id="troubleshooting">Troubleshooting</h1>
<h2 id="xauth--timeout-in-locking-authority-file-xauthority">xauth:  timeout in locking authority file ~/.Xauthority</h2>
<p>As your not-root user, remove and <code>chmod</code> some files.</p>
<pre><code class="language-bash">sudo rm ~/.Xauthority-c
sudo rm ~/.Xauthority-l
sudo rm -r ~/.Xauthority
mkdir ~/.Xauthority
chmod 0600 ~/.Xauthority
</code></pre>
<h2 id="display-is-empty-no-xauth-program">$DISPLAY is Empty, No xauth program</h2>
<p>On your mac, if you connect with <code>ssh -v -X ...</code> and you get an error line
saying <code>No xauth program</code>, then you need to edit your SSH config to specify
the xauth path. Mine was <code>/usr/X11/bin/xauth</code>.</p>
<h2 id="x11-connection-rejected-because-of-wrong-authentication">X11 connection rejected because of wrong authentication.</h2>
<p>If you're on Mac, this can happen when you didn't set the <code>XAuthLocation</code> ssh
configuration.</p>
<h1 id="reference-links">Reference Links</h1>
<ul>
<li><a href="https://linuxize.com/post/how-to-install-google-chrome-web-browser-on-ubuntu-18-04/">linuxize.com - Install chrome</a></li>
<li><a href="https://github.com/dnschneid/crouton/issues/2676">github.com - x11 forwarding tshooting</a></li>
</ul>


Kyle Pericak

Create Windows KVM VM from Command Line

Packet Tracing Reference

Ubuntu NFS Server Setup

Ubuntu: Blindly Trusting the Corporate CA

Remote Access: Build a Dial-Home Device

Operations Reference Page

Creating a Wireless Router for Wired Servers With Ubuntu Server

Connecting Ubuntu 18.04 to WPA WiFi from CLI

Launching Chrome from a Remote Ubuntu Server over SSH