<p>Sometimes you want to test a switch's trunk port without having to carve our an access port to plug your MacBook into.</p>
<p>Here's how:</p>
<ol>
<li>Open System Preferences</li>
<li>Click Network</li>
<li>Select your Thunderbolt Ethernet adapter</li>
<li>Click the gear below the sidebar listing the interfaces</li>
<li>Click Manage Virtual Interfaces</li>
<li>Click the + button</li>
<li>New VLAN</li>
<li>Enter a name, such as "VLAN-200"</li>
<li>Enter a VLAN ID in the Tag field, such as 200</li>
<li>Choose Tunderbolt Ethernet for your Interface</li>
<li>Click Create</li>
<li>On the sidebar, your new vlan ("VLAN-200") is now an option. Click it.</li>
<li>Configure as needed.</li>
</ol>


<p>When you use a tool like <code>dd</code> or <a href="https://www.balena.io/etcher/">Etcher</a> to
turn a USB
stick into boot media, usually for installing an operating system, any attempt
at formatting that disk later will be met with a drive that only has a few GB
total. This is particularly annoying when you use something like a 100+GB USB
stick for that.</p>
<p>The cause of this problem is that a smaller partition is created on the drive
than the drive could potentially hold. To fix the drive you need to
removing all the partitions.</p>
<p>List your devices. Note your USB stick (example: <code>/dev/disk2</code>)</p>
<pre><code class="language-bash">diskutil list
</code></pre>
<p>Try to fix it using diskutil:</p>
<pre><code class="language-bash">diskutil eraseDisk free EMPTY /dev/disk2
</code></pre>
<p>Often that's good enough, and the drive now shows up in Disk Utility with its full capacity.</p>
<hr>
<p>If that fails, you can overwrite the partition/filesystem's metadata at the start of the disk with zeros.</p>
<p>If the drive is mounted, unmount it.</p>
<pre><code class="language-bash">diskutil umountDisk /dev/disk2
</code></pre>
<p>Delete the partition. Note that I tried <code>gpt</code> first but it threw an error
saying <code>gpt show: error: bogus map</code>. Also, as far as I can tell, <code>diskutil</code>
won't actually erase a partition and instead will only reformat it. Strangely,
even <code>fdisk</code> seems to have hidden the option to delete a partition. Guess Apple
<strong>REALLY</strong> doesn't trust its users. That's too bad, since I ended up doing
something even more dangerous.</p>
<p>Overwrite the partition and filesystem data in the first 1GB of the USB drive
with zeros. Be <strong>really really sure</strong> that you select the right disk here.
You can 100% toast your workstation's boot disk if you accidentally specify it.</p>
<pre><code class="language-bash">sudo dd if=/dev/zero of=/dev/disk2 count=1 bs=1GB
</code></pre>
<p>If that doesn't work, set <code>bs=1MB</code> and remove the <code>count=1</code>. You might need to
delete the end of it instead. That'll take a while.</p>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page lists some of the errors that I've personally encountered,
and how I fixed them. I might also include errors I never fixed in case anyone
wants to talk about it.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#docker-login-fails">Docker Login Fails</a></li>
<li><a href="#runaway-memory-usage-in-ceph-osd-process">Runaway memory usage in Ceph OSD process</a></li>
<li><a href="#chrome-blocks-https127001-with-neterr_cert_revoked">Chrome blocks https://127.0.0.1 with NET::ERR_CERT_REVOKED</a></li>
<li><a href="#failed-command-aws-ecr-get-login">Failed command: aws ecr get-login</a></li>
<li><a href="#pure-storage-iscsi-with-kolla-openstack---fdisk-freezes">Pure Storage iSCSI with Kolla OpenStack - fdisk freezes</a></li>
<li><a href="#weird-chars-when-pasting-on-mac-os">Weird chars when pasting on Mac OS</a></li>
</ul>
<h1 id="docker-login-fails">Docker Login Fails</h1>
<pre><code>Error saving credentials: error storing credentials - err: exit status 1, out: `Cannot autolaunch D-Bus without X11 $DISPLAY`
</code></pre>
<p>I had this happen after running both Ceph-Ansible then Kolla-Ansible on an Ubuntu server, so it could be either of those that set up the conditions for this to happen. <a href="https://github.com/docker/cli/issues/1136">this solution</a> worked for me:</p>
<pre><code class="language-bash">apt-get install -y gnupg2 pass
</code></pre>
<hr>
<h1 id="runaway-memory-usage-in-ceph-osd-process">Runaway memory usage in Ceph OSD process</h1>
<p>It looked like a memory leak. Each OSD process was using around 30GB of RAM.</p>
<p>It turns out that Ceph-Ansible has the following logic when generating its
<code>ceph.conf</code> file:</p>
<pre><code># this math
roles/ceph-config/templates/ceph.conf.j2:{% set _osd_memory_target = (ansible_memtotal_mb * 1048576 * hci_safety_factor / _num_osds) | int %}
# sets osd memory target
roles/ceph-config/templates/ceph.conf.j2:osd memory target = {{ _osd_memory_target | default(osd_memory_target) }}
</code></pre>
<p>In my case, this set the value WAY too high. In one of my deployments, it was
set to <code>37853425827</code>, which is like 35GB (PER OSD SERVICE!).</p>
<p>The 4GB default is much more sane, <code>4294967296</code>.</p>
<pre><code># ceph.conf
[osd]
osd memory target = 4294967296
</code></pre>
<hr>
<h1 id="chrome-blocks-https127001-with-neterr_cert_revoked">Chrome blocks <a href="https://127.0.0.1">https://127.0.0.1</a> with NET::ERR_CERT_REVOKED</h1>
<p>This comes up when I reverse tunnel to a remove web server, forwarding it to my workstation. Chrome has decided to not let you do that. They give this errror:</p>
<pre><code class="language-text">Your connection is not private
Attackers might be trying to steal your information from 127.0.0.1 (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_REVOKED
</code></pre>
<p>To bypass that, open <code>chrome://flags/#allow-insecure-localhost</code> and set allow-insecure-localhost to enabled. Click relaunch. This will nuke your current browser…</p>
<p>Once it relaunches you'll be able to open the site.</p>
<hr>
<h1 id="failed-command-aws-ecr-get-login">Failed command: aws ecr get-login</h1>
<p>This error was caused by a bad NTP config. Renew the NTP lease to fix it.</p>
<pre><code>An error occurred (InvalidSignatureException) when calling the GetAuthorizationToken operation: Signature expired: 20190911T174538Z is now earlier than 20190912T004835Z (20190912T010335Z - 15 min.)
</code></pre>
<hr>
<h1 id="pure-storage-iscsi-with-kolla-openstack---fdisk-freezes">Pure Storage iSCSI with Kolla OpenStack - fdisk freezes</h1>
<p>This was a weird one. When using the Kolla containers project to make a
Cinder-volume container that has the <code>purecinder</code> plugin, I was able to get
OpenStack to create the volumes in Pure but it couldn't mount them to the hosts
for things like creating a volume with the <code>--image</code> argument. That made the
volumes I could create basically useless.</p>
<p>These errors were in the logs:</p>
<pre><code class="language-text"># Lots of these in cinder's log
Trying to connect to iSCSI portal

# also from cinder's log, also recurring:
iscsiadm stderr output when getting sessions: iscsiadm: No active sessions

# in DMESG
FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
</code></pre>
<p>The weirdest part was that the <code>fdisk -l</code> command would totally lock up and
the process running it couldn't be killed even with <code>kill -9</code>. The iSCSI volume
never got properly mounted but Pure support said they could see some data
coming into it.</p>
<p><strong>Root cause</strong>: MTU mismatch. Of course. I had set MTU 9000 on the iSCSI ports,
my host ports, and the switchports heading to the Pure Storage iSCSI
interfaces. I had forgotten to set it on the switch heading to the Dell servers
being used for OpenStack.</p>
<hr>
<h1 id="weird-chars-when-pasting-on-mac-os">Weird chars when pasting on Mac OS</h1>
<p>When you paste sometimes you get <code>^[[200~</code> at the start and <code>^[[201~</code> at the
end of the pasted content. Or maybe <code>0~</code> and <code>1~</code>.</p>
<p>To resolve, run:</p>
<pre><code class="language-bash">printf "\e[?2004l"^C
</code></pre>


<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page contains operations-related mini-guides and minor posts.</p>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</a></li>
<li><a href="#convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</a></li>
<li><a href="#journalctl-commands">Journalctl commands</a></li>
<li><a href="#get-real-ram-use-of-processes">Get real RAM use of processes</a>
<ul>
<li><a href="#of-ceph">of ceph</a></li>
<li><a href="#of-vms">of VMs</a></li>
</ul>
</li>
<li><a href="#bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</a></li>
<li><a href="#create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</a>
<ul>
<li><a href="#use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</a></li>
</ul>
</li>
<li><a href="#test-mtu-from-ubuntu">Test MTU from Ubuntu</a></li>
<li><a href="#find-big-files-on-mac-os">Find big files on Mac OS</a></li>
<li><a href="#renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</a></li>
<li><a href="#create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</a></li>
<li><a href="#passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</a></li>
<li><a href="#vim-tasks">VIM Tasks</a>
<ul>
<li><a href="#find--replace">Find &#x26; Replace</a></li>
<li><a href="#spell-check-commands">Spell-Check commands</a></li>
</ul>
</li>
<li><a href="#use-an-insecure-docker-registry">Use an Insecure Docker Registry</a></li>
<li><a href="#ansible-ignore-errors">Ansible ignore errors</a></li>
<li><a href="#how-to-tail--f-on-dmesg">How to tail -f on dmesg</a></li>
<li><a href="#ubuntu-downgrade-a-package">Ubuntu Downgrade a package</a></li>
<li><a href="#make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</a></li>
<li><a href="#bash-print-all-but-first-line">Bash: Print all but first line</a></li>
<li><a href="#bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</a></li>
</ul>
<h1 id="find-which-active-passive-bond-member-is-active">Find which active-passive bond member is active</h1>
<pre><code>cat /proc/net/bonding/bond0
</code></pre>
<hr>
<h1 id="convert-p7b-certificate-to-pem-format">Convert p7b certificate to pem format</h1>
<pre><code class="language-bash">openssl pkcs7 -print_certs -in certnew.p7b -out cert.pem
</code></pre>
<hr>
<h1 id="journalctl-commands">Journalctl commands</h1>
<p>Watch the logs as they come in:</p>
<pre><code class="language-bash">journalctl -f
</code></pre>
<p>When journalctl logs too much and you need to pair it down:</p>
<pre><code class="language-bash"># to 30 days
journalctl --vacuum-time=30d

# to 500MB
journalctl --vacuum-size=500M
</code></pre>
<p>Show for a specific service</p>
<pre><code class="language-bash">journalctl ceph*
</code></pre>
<hr>
<h1 id="get-real-ram-use-of-processes">Get real RAM use of processes</h1>
<p>You can always use top or htop or whatever. I use this to specifically get the
info about those processes.</p>
<h2 id="of-ceph">of ceph</h2>
<pre><code>ps aux | grep ceph | grep -v -e grep -e qemu -e "\[" | while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); id=$(echo $line | awk '{print $16}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tid: $id,    \tRAM: ($size)"; done
</code></pre>
<h2 id="of-vms">of VMs</h2>
<p>for KVM</p>
<pre><code>ps aux | grep qemu | grep -v -e grep| while read line; do bin=$(echo $line | awk '{print $11}'); pid=$(echo $line | awk '{print $2}'); guest=$(echo $line | awk '{print $13}'); size=$(grep 'VmSize' /proc/$pid/status); echo -e "pid: $pid, \tbin: $bin, \tguest: $guest,   \t\tRAM: ($size)" ; done
</code></pre>
<h1 id="bypass-chromes-self-signed-cert-warning">Bypass Chrome's self-signed cert warning</h1>
<p>As of Mac Catalina, developing a site using self-signed certs is a bigger pain
than before. Yeah you can go get the key and add it, but that's a nuisance.
It looks like the Chrome team built something they call an
"interstitial bypass keyword" to deal with that. That's one cool feature name.</p>
<p><strong>Obvious disclaimer</strong> If you're not a developer doing this to access your own
site, don't do this.</p>
<p>To bypass the warning, type</p>
<pre><code>thisisunsafe
</code></pre>
<p>Apparently it used to be <code>badidea</code> but they rotated it since it got too well
known. They'll probably do it again someday. I suspect people are using this
on sites they "trust" without really knowing any better and getting hurt by
it.</p>
<hr>
<h1 id="create-a-virtual-loopback-volume-on-ubuntu">Create a virtual loopback volume on Ubuntu</h1>
<p>Useful for simulating a real drive when building a test environment. I use this to test OpenStack in a VM as a Ceph or LVM back-end.</p>
<pre><code class="language-bash"># create empty file of all zeros
sudo dd if=/dev/zero of=/root/virtual-disk bs=1M count=512
# show list of loopback devices. Note its not there
losetup --list
# create the loopback device
losetup /dev/loop0 /root/virtual-disk
# confirm it worked
losetup --list
</code></pre>
<p>You can now treat it like a normal volume. Mount it, put a filesystem on it, whatever. Example:</p>
<pre><code class="language-bash">pvcreate /dev/loop0
vgcreate cinder-volumes /dev/loop0
</code></pre>
<h2 id="use-systemd-to-re-map-loopback-on-reboot">Use Systemd to re-map loopback on reboot</h2>
<p>I found <a href="https://unix.stackexchange.com/questions/418322/persistent-lvm-device-with-loopback-devices-by-fstab">this useful systemd unit example</a> that didn't quite work for me, but was close. Here's mine:</p>
<p><code>vi /etc/systemd/system/loops-setup.service</code></p>
<pre><code class="language-text">[Unit]
Description=Setup loopback devices

DefaultDependencies=no
Conflicts=umount.target

Requires=lvm2-lvmetad.service mnt-host.mount
Before=local-fs.target umount.target
After=lvm2-lvmetad.service mnt-host.mount

[Service]
ExecStart=/sbin/losetup /dev/loop0 /root/virtual-disk
ExecStop=/sbin/losetup -d /dev/loop0

RemainAfterExit=yes
Type=oneshot

[Install]
WantedBy=local-fs-pre.target
</code></pre>
<p>Then enable it</p>
<pre><code class="language-bash">systemctl enable loops-setup
</code></pre>
<p>Reboot, then check <code>vgdisplay</code> (if that's your use case). It'll be there.</p>
<hr>
<h1 id="test-mtu-from-ubuntu">Test MTU from Ubuntu</h1>
<p>You would not believe the strange problems that an MTU issue can be the root cause of.</p>
<pre><code class="language-bash">ping -M do -s 1472 &#x3C;remote server>
ping -M do -s 8900 &#x3C;remote server>
</code></pre>
<hr>
<h1 id="find-big-files-on-mac-os">Find big files on Mac OS</h1>
<p>My SSD is always almost full. Here are useful commands for cleaning up:</p>
<pre><code class="language-bash"># Find all the files over 1G
sudo find / -type f -size +1G -exec ls -lh {} \; | awk '{ print $9 ": " $5 }' 2>/dev/null

# List files in Downloads directory by size
ls -lhS ~/Downloads
</code></pre>
<h1 id="renew-ntp-lease-ubuntu">Renew NTP Lease Ubuntu</h1>
<pre><code class="language-bash">service ntp stop
ntpd -gq
service ntp start
</code></pre>
<hr>
<h1 id="create-aws-ecr-registry-if-not-found">Create AWS ECR Registry If Not Found</h1>
<pre><code class="language-bash">aws ecr describe-repositories --repository-names $repo_name \
    || aws ecr create-repository --repository-name $repo_name
</code></pre>
<hr>
<h1 id="passwordless-sudo-on-ubuntu-server">Passwordless sudo on Ubuntu Server</h1>
<p>Run <code>visudo</code> as root.</p>
<pre><code class="language-bash"># as root
export EDITOR=vim
visudo
</code></pre>
<p>Add the user (replace <code>exampleUser</code>) at the end.</p>
<pre><code class="language-text">exampleUser ALL=(ALL) NOPASSWD: ALL
</code></pre>
<p>For those who don't use vim, a quick reminder:
<code>i</code> enter to insert mode , <code>[esc]</code> to exit insert mode, <code>:wq</code> to save and quit.</p>
<hr>
<h1 id="vim-tasks">VIM Tasks</h1>
<h2 id="find--replace">Find &#x26; Replace</h2>
<pre><code class="language-bash"># replace findThis with ReplaceWithThis
:%s/findThis/ReplaceWithThis/g
</code></pre>
<h2 id="spell-check-commands">Spell-Check commands</h2>
<pre><code class="language-text"># enable/disable
:set spell spelllang=en_ca
:setlocal spell spelllang=en_ca
:set nospell

# next/last
]s
[s

# suggest change
z=

# add/remove word from dictionary
zg
zw`
</code></pre>
<hr>
<h1 id="use-an-insecure-docker-registry">Use an Insecure Docker Registry</h1>
<p>To use a registry with no HTTP cert:</p>
<p><code>vi /etc/docker/daemon.json</code></p>
<pre><code class="language-json">{
    "insecure-registries" : ["myregistrydomain.com:5000"]
}
</code></pre>
<hr>
<h1 id="ansible-ignore-errors">Ansible ignore errors</h1>
<p>The command is <code>ignore_errors: yes</code></p>
<p>Just throw this into the task that's failing. It will still fail, but the play
won't stop.</p>
<p>Example:</p>
<pre><code class="language-yml">- name: Ensure the docker service is running
  script: ../tools/validate-docker-execute.sh
    ignore_errors: yes
</code></pre>
<hr>
<h1 id="how-to-tail--f-on-dmesg">How to tail -f on dmesg</h1>
<pre><code class="language-bash">dmesg -wH
</code></pre>
<hr>
<h1 id="ubuntu-downgrade-a-package">Ubuntu Downgrade a package</h1>
<pre><code class="language-bash"># Find the available versions under Versions:
apt-cache showpkg &#x3C;package>

# Install one of those
apt-get install &#x3C;package>=&#x3C;version>
</code></pre>
<hr>
<h1 id="make-ansible-run-python-3-on-target-nodes">Make Ansible run Python 3 on target nodes</h1>
<p>Useful for OS that ship with python3 but not python2.
Edit the inventory, find a group of the groups and apply this var.</p>
<pre><code class="language-ini">[baremetal:vars]
ansible_python_interpreter=/usr/bin/python3
</code></pre>
<hr>
<h1 id="bash-print-all-but-first-line">Bash: Print all but first line</h1>
<p>Use <code>tail -n +2</code>.
This was unintuitive to me because +1 seems like it should do it.</p>
<pre><code class="language-bash"># Example
docker images | awk '{print $3}' | tail -n +2
</code></pre>
<hr>
<h1 id="bash-run-code-whenever-a-script-exits">Bash: Run code whenever a script exits</h1>
<p>Use trap.</p>
<ul>
<li><a href="http://redsymbol.net/articles/bash-exit-traps/">This is a nice post covering lots of examples</a>.</li>
<li><a href="https://github.com/GoogleCloudPlatform/endpoints-quickstart/blob/master/scripts/deploy_api.sh">Here's where I first saw it used</a></li>
</ul>
<pre><code class="language-bash">temp_file=$(mktemp)

cleanup() {
  rm "$temp_file"
}

trap cleanup EXIT
</code></pre>


Kyle Pericak

"It works in my environment"

Categories

Tags