<p>When you import a RedHat VM, say from VMWare, to a KVM environment like OpenStack it probably won't boot and instead goes straight to the dracut prompt.</p>
<p>This happens because the ramdisk doesn't have virtio drivers. You need to inject them. Import the volume and attach it to a running Linux server.
Personally I use Ubuntu, but it doesn't matter since you'll <code>chroot</code> in.</p>
<h1 id="steps">Steps</h1>
<h2 id="mount-the-volume">Mount the volume</h2>
<ol>
<li>In your cloud environment, mount the RedHat boot volume to your working Ubuntu (or whatever) machine.</li>
<li>Find the boot and data volumes, and mount them too.</li>
<li>bind-mount your system paths into the root volume's mount point</li>
</ol>
<pre><code class="language-bash">mkdir /mnt/boot /mnt/root
mount &#x3C;boot device> /mnt/boot
mount &#x3C;root device> /mnt/root

for x in sys proc run dev tmp; do mount --bind /$x /mnt/root/$x; done
</code></pre>
<h2 id="use-dracut-to-update-the-initrd-file">Use Dracut to update the initrd file</h2>
<ol>
<li>chroot into the volume</li>
<li>Check <code>/etc/grub/grub.conf</code> to see which initrd file is being used. Something like <code>3.10.0-1062.1.2.el7.x86_64</code>.</li>
<li>check if the drivers are present</li>
<li>install the drivers</li>
<li>confirm they're present</li>
</ol>
<pre><code class="language-bash"># chroot in
cd /mnt
chroot root bash

# set the version
cat /etc/grub/grub.cnf
version=
# example: version=3.10.0-1062.1.2.el7.x86_64

# check for virtio drivers
lsinitrd /boot/initramfs-$version.img | grep virtio

# if not found, add them
cd /boot
dracut -f \
  /boot/initramfs-$version.img \
  $version \
  --add-drivers "virtio virtio_pci virtio_blk virtio_net"

# prove it worked
lsinitrd /boot/initramfs-$version.img | grep virtio

# leave the chroot
exit
</code></pre>
<p>This is a good time to do any other modifications you like, such as installing cloud-init or enabling DHCP.</p>
<h2 id="clean-up">Clean up</h2>
<p>Remove the mounts</p>
<pre><code class="language-bash">for x in sys proc run dev tmp; do umount /mnt/root/$x; done
umount /mnt/root
umount /mnt/boot
</code></pre>
<h2 id="validate">Validate</h2>
<p>Now you should be able to boot your RedHat VM without dracut coming up.</p>


<h1 id="tldr">TL;DR</h1>
<pre><code># replace "import queue" with:
from multiprocessing import Queue as queue
</code></pre>
<hr>
<h1 id="steps-to-fix">Steps to fix</h1>
<p>In Ubuntu 18.04 you'd typically install the openstack CLI client like this:</p>
<pre><code class="language-bash">pip install python-openstackclient
</code></pre>
<p>Sadly, this installs a broken version of the client. If you try and run any command you get a stack trace:</p>
<pre><code class="language-bash">Traceback (most recent call last):
  File "/usr/local/bin/openstack", line 6, in &#x3C;module>
    from openstackclient.shell import main
  File "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", line 24, in &#x3C;module>
    from osc_lib import shell
  File "/usr/local/lib/python2.7/dist-packages/osc_lib/shell.py", line 33, in &#x3C;module>
    from osc_lib.cli import client_config as cloud_config
  File "/usr/local/lib/python2.7/dist-packages/osc_lib/cli/client_config.py", line 18, in &#x3C;module>
    from openstack.config import exceptions as sdk_exceptions
  File "/usr/local/lib/python2.7/dist-packages/openstack/__init__.py", line 16, in &#x3C;module>
    import openstack.config
  File "/usr/local/lib/python2.7/dist-packages/openstack/config/__init__.py", line 17, in &#x3C;module>
    from openstack.config.loader import OpenStackConfig  # noqa
  File "/usr/local/lib/python2.7/dist-packages/openstack/config/loader.py", line 33, in &#x3C;module>
    from openstack.config import cloud_region
  File "/usr/local/lib/python2.7/dist-packages/openstack/config/cloud_region.py", line 44, in &#x3C;module>
    from openstack import proxy
  File "/usr/local/lib/python2.7/dist-packages/openstack/proxy.py", line 24, in &#x3C;module>
    from openstack import resource
  File "/usr/local/lib/python2.7/dist-packages/openstack/resource.py", line 49, in &#x3C;module>
    from openstack import utils
  File "/usr/local/lib/python2.7/dist-packages/openstack/utils.py", line 13, in &#x3C;module>
    import queue
ImportError: No module named queue
</code></pre>
<p>To fix it, replace <code>import queue</code> with <code>from multiprocessing import Queue as queue</code> everywhere that it's called.</p>
<p>In my case:</p>
<ul>
<li><code>usr/local/lib/python2.7/dist-packages/openstack/utils.py</code> - line 13</li>
<li><code>/usr/local/lib/python2.7/dist-packages/openstack/cloud/openstackcloud.py</code> - line 14</li>
</ul>


<p><strong>This post is linked to from the <a href="/openstack.html">OpenStack Deep Dive Project</a></strong></p>
<hr>
<p>When you launch an instance in OpenStack, you can provide a script to
cloud-init that will be executed at startup time. The glance template needs to
have <a href="https://cloudinit.readthedocs.io/en/latest/">cloud-init</a> installed
(linux) or <a href="https://cloudbase.it/cloudbase-init/">Cloudbase-init</a> (windows).</p>
<h1 id="write-your-powershell-script">Write your Powershell Script</h1>
<p>You need to define the script ahead of time. Here's a super simple script
to add a local administrator:</p>
<pre><code class="language-powershell">#ps1
$name = "MyUser"
$password = "MyPassword"
$password_secure_string = ConvertTo-SecureString -AsPlainText -Force $password
$new_user = New-LocalUser -Name $name -Password $password_secure_string -AccountNeverExpires
Add-LocalGroupMember -Group "Administrators" -Member $new_user
</code></pre>
<hr>
<h1 id="create-the-instance">Create the Instance</h1>
<p>There's more than one way to do just about everything, but here's how I do it.</p>
<pre><code class="language-bash"># source openrc file
source my-openrc.sh

# Collect environment details
openstack flavor list
openstack network list
openstack image list

# Set the VM specs
flavor=""
network=""
image=""
name="CloudInitDemo"
size=60
vol_name="$name-boot"

# Create a boot volume
openstack volume create --image $image --bootable --size $size $vol_name

# create server and specify cloud-init script
script_file="/home/kyle/localAdmin.ps1"
openstack server create \
  --volume $vol_name \
  --flavor $flavor \
  --network $network \
  --user-data $script_file \
  $name
</code></pre>


<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#prepare-the-image-in-guest">Prepare the image in-guest</a>
<ul>
<li><a href="#remove-vmware-tools">Remove VMWare Tools</a></li>
<li><a href="#install-cloudbase-init--sysprep-if-needed">Install cloudbase-init &#x26; sysprep (if needed)</a></li>
<li><a href="#shut-down-the-vm--export-the-drive">Shut down the VM &#x26; Export the drive</a></li>
</ul>
</li>
<li><a href="#inject-the-kvm-drivers">Inject the KVM Drivers</a></li>
</ul>
<h1 id="prepare-the-image-in-guest">Prepare the image in-guest</h1>
<p>From VMWare or HyperV, launch your template Windows VM and log in as an
administrative user. Make any firewall changes you need, install any packages,
and ensure that RDP is enabled.</p>
<h2 id="remove-vmware-tools">Remove VMWare Tools</h2>
<p>If the guest has VMWare tools installed, remove them. They won't be needed
where we're going.</p>
<h2 id="install-cloudbase-init--sysprep-if-needed">Install cloudbase-init &#x26; sysprep (if needed)</h2>
<p>This isn't so much for KVM as it is for OpenStack, but cloudbase-init is the
Windows version of cloud-init. This will set your hostname and such when the VM
is created.</p>
<ol>
<li><a href="https://cloudbase.it/cloudbase-init/">Download the Cloudbase-init installer</a></li>
<li>Run through the installer.
<ol>
<li>Don't run as Local System.</li>
<li>Don't name your user Administrator, the named user is made by this
service to run the startup tasks.</li>
</ol>
</li>
<li>At the end of the installation, you'll be prompted to sysprep. If this VM
will be re-used as a template and not just a one-off VM, then sysprep it.</li>
</ol>
<h2 id="shut-down-the-vm--export-the-drive">Shut down the VM &#x26; Export the drive</h2>
<p>Shut down the VM and grab the VMDK or VHDX file it booted from. Copy that file
to a directory on your computer, such as <code>C:\Temp</code>.</p>
<hr>
<h1 id="inject-the-kvm-drivers">Inject the KVM Drivers</h1>
<ol>
<li><a href="https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/virtio-win.iso">Download the VirtIO ISO</a></li>
<li>Mount the ISO to your workstation. In these examples, it's E:</li>
<li>Open PowerShell as admin and verify that the file is where you put it. You
can show some basic info about it like this: <code>get-vhd C:\Temp\Win10.vhdx</code></li>
<li>Create a mount-point for Dism: <code>mkdir C:\mount</code></li>
<li>Mount the volume with Dism:
<code>Dism /mount-image /ImageFile:C:\Temp\Win10.vhdx /Index:1 /MountDir:C:\mount</code></li>
<li>Inject the <code>viostor</code> driver:
<code>Dism /image:C:\mount /Add-Driver /Driver:E:\viostor\w10\amd64 /Recurse</code></li>
<li>Inject the <code>Baloon</code> driver:
<code>Dism /image:C:\mount /Add-Driver /Driver:E:\Balloon\w10\amd64 /Recurse</code></li>
<li>Inject the <code>NetKVM</code> driver:
<code>Dism /image:C:\mount /Add-Driver /Driver:E:\NetKVM\w10\amd64 /Recurse</code></li>
<li>Confirm the drivers injected: <code>Dism /image:C:\mount /Get-Drivers</code></li>
<li>Unmount the image: <code>Dism /Unmount-image /MountDir:c:\mount /commit</code></li>
</ol>
<p>Now the image is ready to be converted to raw/qcow2 and uploaded to OpenStack.</p>
<p><strong>Note</strong>: CloudBase has released a Windows version of <code>qemu-img</code> so you can do
the conversions on Windows now without VirtualBox tools. You can download it
<a href="https://cloudbase.it/qemu-img-windows/">here</a></p>
<p>If you want to create a brand-new Windows VM on KVM from scratch rather than
importing an existing image, see
<a href="/windows-kvm-command-line.html">Create Windows KVM VM from Command Line</a>.</p>


<p><strong>This project is a "Deep Dive":</strong> In a Deep Dive project I'll be investigating
as much about the subject as time allows and assembling the posts in some
orderly fashion. I'll also be writing high-level notes and opinions in the
project page itself.</p>
<h4 id="table-of-contents">Table of contents</h4>
<ul>
<li><a href="#what-is-openstack">What is OpenStack?</a></li>
<li><a href="#project-posts--progress">Project Posts &#x26; Progress</a></li>
<li><a href="#why-use-an-openstack-private-cloud">Why Use an OpenStack Private Cloud?</a></li>
<li><a href="#complexity-and-openstack-ad">Complexity and OpenStack (Ad)</a></li>
<li><a href="#installing-openstack">Installing OpenStack</a>
<ul>
<li><a href="#non-production-installation-options">Non-Production Installation Options</a>
<ul>
<li><a href="#devstack">DevStack</a></li>
<li><a href="#small-physical-server--kolla-ansible">Small Physical Server + Kolla-Ansible</a></li>
<li><a href="#virtual-server--kolla-ansible">Virtual Server + Kolla-Ansible</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#containerized-openstack">Containerized OpenStack</a></li>
</ul>
<h1 id="what-is-openstack">What is OpenStack?</h1>
<p><a href="https://www.openstack.org/">OpenStack</a> is a collection of open-source projects
that work together to form a private cloud solution. It's by far the
biggest open source private cloud option available. The <a href="https://www.openstack.org/foundation/">OpenStack Foundation</a>
has over 105,000 community members from 187 countries around the world.</p>
<p>At a high level, you can think of it as an abstraction layer that sits on top
of other popular cloud technologies such as storage clusters, hypervisors, and
network solutions. For example, KVM (the popular Linux hypervisor) is not part
of OpenStack, but it's used by <strong>Nova</strong>, a component of OpenStack which can
orchestrate against KVM, VMWare, HyperV, and Xen.</p>
<hr>
<h1 id="project-posts--progress">Project Posts &#x26; Progress</h1>
<p>This project includes the following posts.</p>
<table>
<thead>
<tr>
<th align="left">Date</th>
<th align="left">Article</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">2019-08-13</td>
<td align="left"><a href="/openstack-aio-ka-metal.html">Install OpenStack on Metal - Intel NUC</a></td>
</tr>
<tr>
<td align="left">2019-08-11</td>
<td align="left"><a href="/openstack-aio-ka-vm.html">Install OpenStack inside a VM</a></td>
</tr>
<tr>
<td align="left">2019-08-26</td>
<td align="left"><a href="/openstack-kolla-custom-plugin.html">Modifying OpenStack Kolla Docker Images</a></td>
</tr>
<tr>
<td align="left">2019-12-03</td>
<td align="left"><a href="/openstack-ansible.html">Operating OpenStack from Ansible</a></td>
</tr>
<tr>
<td align="left">2020-01-25</td>
<td align="left"><a href="/openstack-cloudinit-powershell.html">OpenStack Cloud-Init Powershell Example</a></td>
</tr>
<tr>
<td align="left">2019-12-17</td>
<td align="left"><a href="/windows-kvm-drivers.html">Injecting KVM Drivers into Windows</a></td>
</tr>
<tr>
<td align="left">2020-10-21</td>
<td align="left"><a href="/rhel-virtio-initrd.html">Adding Virtio Drivers to RHEL's initrd for KVM</a></td>
</tr>
</tbody>
</table>
<hr>
<h1 id="why-use-an-openstack-private-cloud">Why Use an OpenStack Private Cloud?</h1>
<ol>
<li><strong>Price at scale:</strong> At small scales, private clouds lose on price
against the public clouds. As you start to ramp up though, the public clouds
start to really nickle and dime you. Private clouds have higher up-front
costs but they win out in the long-run.</li>
<li><strong>Predictable Cost:</strong> Spin up a few services on AWS or Azure. How much will
they cost you each month? As it stands, you're guess is going to be wrong.
After the free trial periods and special offers stop muddying the water,
you'll usually discover you're spending more than you thought.</li>
<li><strong>Hardware Re-Use:</strong> Many IT organizations already have hardware sitting
around running expensive hypervisor software. OpenStack is free and can use
that existing hardware, giving you more for less.</li>
<li><strong>Data Sovereignty:</strong> For legal and security reasons, many organizations
must know exactly where their data is sitting at rest. With private clouds,
there's no ambiguity.</li>
<li><strong>Behind Your Firewall:</strong> So you spent a ton of money on some next
generation firewall technologies with intrusion detection/prevention, and
trained the staff up to use it. You run an arguably well secured
data-center. Running OpenStack keeps the cloud behind your existing security
measures.</li>
<li><strong>Latency:</strong> Particularly for edge clouds and VDI solutions, having the
cloud located right on-premises can hugely improve user experience.</li>
<li><strong>Shadow IT:</strong> Developers are going to develop, even if your organization
can't get them the tools they need in time. By offering your own in-house
cloud, you can help prevent
<a href="https://en.wikipedia.org/wiki/Shadow_IT">shadow IT</a> from emerging.</li>
<li><strong>Control:</strong> This one is a double-edged sword. Having more control over your
cloud can be its own benefit, but there's also value in letting the pros
handle it.</li>
<li><strong>Business as usual, without VMWare:</strong> Don't get me wrong, VMWare make a
great product... I've never heard anyone say they were happy with their
VMWare bill. It's insanely expensive when there's an open source alternative
that can check all the same boxes.</li>
</ol>
<hr>
<h1 id="complexity-and-openstack-ad">Complexity and OpenStack (Ad)</h1>
<p>OpenStack is significantly more difficult to get started with than the public
clouds. You can't really install OpenStack without first understanding what
each of the main components does, and even then each one has its own quirks for
their installation. If you want to install in production, you also need to
figure out how to get it all highly available.</p>
<p>Unless you've got a huge team that's eager to tackle this complexity, I'm going
to use this opportunity to plug the software &#x26; solution I've helped build for
<a href="https://breqwatr.com">Breqwatr</a>. They'll stand up your cloud in a couple days
flat, teach your team how to use it, and keep the lights on - all while
charging significantly less than VMWare would cost for the same hardware.</p>
<p>Breqwatr, and companies like it, can be your trusted cloud partner and abstract
away that complexity by handling the installation, upgrades, monitoring, and
day-to-day operation of the cloud.</p>
<hr>
<h1 id="installing-openstack">Installing OpenStack</h1>
<h2 id="non-production-installation-options">Non-Production Installation Options</h2>
<p>If you're not looking for a production deployment and instead want to try
OpenStack out to learn it, there are a few good options.</p>
<h3 id="devstack">DevStack</h3>
<p>The standard developer option is probably <a href="https://docs.openstack.org/devstack/latest/">DevStack</a>.</p>
<p>In truth I haven't used this since I've always needed my OpenStack installs to
mirror a production-ready environment. If I try it out, I'll replace the above
link with one to a guide I write myself.</p>
<h3 id="small-physical-server--kolla-ansible">Small Physical Server + Kolla-Ansible</h3>
<p>Another option is to install OpenStack on a physical server. I've documented
my experience doing this using images from the <a href="https://github.com/openstack/kolla">Kolla project</a>,
and Ansible automation from the <a href="https://github.com/openstack/kolla-ansible">Kolla-Ansible project</a>.</p>
<p>The nice thing about this approach is you can use a server at home, and once
you're done it can be a VM host as your private house cloud.</p>
<p><strong><a href="/openstack-aio-ka-metal.html">Installing OpenStack on a physical server</a></strong></p>
<h3 id="virtual-server--kolla-ansible">Virtual Server + Kolla-Ansible</h3>
<p>This option is great for development use if you already have a cloud available
or want to use a public cloud for testing OpenStack. The VMs you launch on it
will perform terribly though, and there are some networking complications.</p>
<p>Don't bother trying to launch Windows VMs at all in a nested virtualisation
scenario, you'll be lucky if they even boot.</p>
<p><strong><a href="/openstack-aio-ka-vm.html">Installing OpenStack on a Virtual Server</a></strong></p>
<hr>
<h1 id="containerized-openstack">Containerized OpenStack</h1>
<p>Managing OpenStack clusters is a pain. It has so many services, each with their
own dependencies and operating instructions, there's a lot that can go wrong.
One thing that's really helped with that is to containerize each service and
deploy them all using Docker.</p>
<p>The <a href="https://github.com/openstack/kolla">Kolla project</a> does such a great job
of that, that there's no need to do it yourself. That is, until you need to
change an image, for instance to install a Cinder plugin.</p>
<p><strong><a href="/openstack-kolla-custom-plugin.html">Modifying OpenStack Kolla Docker Images</a></strong></p>


<p><strong>This post is linked to from the <a href="/openstack.html">OpenStack Deep Dive Project</a></strong></p>
<hr>
<p><a href="https://docs.openstack.org/cinder/latest/">OpenStack Cinder</a> needs some
special software installed to work with certain storage backends.</p>
<p><a href="https://docs.openstack.org/kolla/latest/">Kolla</a> builds Docker containers for
OpenStack.</p>
<p>This post documents the steps to customize how Kolla builds containers. In
particular, these steps add the Pure Storage plugin to the Cinder-Volume
service's Docker image.</p>
<hr>
<h1 id="install-kolla">Install Kolla</h1>
<p>This example uses the Rocky checkout to make Rocky based images. Use the stable
branch matching your OpenStack version else things get weird.</p>
<pre><code class="language-bash">apt-get install -y git python python-pip
git clone https://github.com/openstack/kolla.git
get checkout stable/rocky
cd kolla
pip install .
</code></pre>
<h2 id="configure-kolla">Configure Kolla</h2>
<p>Use tox's <code>genconfig envlist</code> to generate a commented config file:</p>
<pre><code class="language-bash">tox -e genconfig
mkdir -p /etc/kolla
mv etc/kolla/kolla-build.conf /etc/kolla/
</code></pre>
<p>Then make any changes to the new config file as needed.</p>
<hr>
<h1 id="add-the-modifications">Add the Modifications</h1>
<p>Here's an example of adding the Pure Storage's plugin to the cinder_volume
image.</p>
<p><code>vi /etc/kolla/template-overrides.j2</code></p>
<pre><code class="language-jinja2">{% extends parent_template %}

# Cinder Volume
{% block cinder_volume_ubuntu_setup %}
RUN pip install purestorage
{% endblock %}
</code></pre>
<hr>
<h1 id="build-the-image-w-modifications">Build the Image w/ Modifications</h1>
<pre><code class="language-bash">kolla-build \
  --tag rocky-test-20190826-01 \
  --template-override /etc/kolla/template-overrides.j2  \
  cinder-volume
</code></pre>
<p>That's it. You can now use this image with the included plugin.</p>


<p>This guide installs OpenStack on a single metal server.</p>
<p>To install OpenStack, I use <a href="https://github.com/openstack/kolla-ansible">Kolla-Ansible</a>,
an open source Ansible project that deploys Kolla OpenStack images.</p>
<p>OpenStack <a href="https://www.openstack.org/software/rocky/">Rocky</a> will be installed,
configured with a
<a href="https://docs.openstack.org/ocata/networking-guide/intro-os-networking.html#intro-os-networking-provider">VLAN provider network</a>
for overcloud networking, and include the following OpenStack APIs:</p>
<ul>
<li>Keystone</li>
<li>Cinder</li>
<li>Nova</li>
<li>Glance</li>
<li>Neutron</li>
<li>Heat</li>
<li>Magnum</li>
</ul>
<hr>
<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#environment">Environment</a></li>
<li><a href="#update--prepare-system">Update &#x26; Prepare System</a></li>
<li><a href="#configure-networking">Configure Networking</a></li>
<li><a href="#enable-vlans-for-neutron">Enable VLANs for Neutron</a></li>
<li><a href="#globals-file">Globals file</a></li>
<li><a href="#install-openstack-using-kolla-ansible">Install OpenStack using Kolla-Ansible</a></li>
<li><a href="#overcloud-cluster-configuration">Overcloud Cluster Configuration</a></li>
<li><a href="#create-a-test-vm">Create a test VM</a></li>
</ul>
<h1 id="environment">Environment</h1>
<p>This was the hardware used while writing this guide. It's neither the minimum
nor recommended hardware setup for any sort of cloud.</p>
<p><strong>Hardware:</strong></p>
<p>Intel NUC with two drives, one interface, 32G RAM, running Ubuntu 18.04.</p>
<p><strong>Drive Labels:</strong></p>
<ul>
<li>Boot drive: <code>/dev/nvme0n1</code></li>
<li>Cinder LVM backup: <code>dev/sda</code></li>
</ul>
<p><strong>Networking</strong></p>
<ul>
<li>Physical interface <code>eno1</code></li>
<li>VLAN subinterface <code>vlan.2@eno1</code></li>
<li>Host IP: <code>10.254.2.2</code></li>
<li>Cloud VIP: <code>10.254.2.254</code></li>
</ul>
<hr>
<h1 id="update--prepare-system">Update &#x26; Prepare System</h1>
<p>Configure the VG for cinder and install Kolla-Ansible</p>
<pre><code class="language-bash"># Update and install deps
apt-get update &#x26;&#x26; apt-get upgrade -y
apt-get -y install python python-pip
pip install \
  ansible \
  python-openstackclient \
  python-neutronclient \
  python-magnumclient

# Create VG for cinder LVM
pvcreate /dev/sda
vgcreate cinder-volumes /dev/sda

# Install Kolla-Ansible
cd ~
git clone https://github.com/openstack/kolla-ansible.git
cd ~/kolla-ansible
git checkout stable/rocky
pip install .

# Deploy initial KA config files
mkdir -p /etc/kolla/config
cp ~/kolla-ansible/etc/kolla/passwords.yml /etc/kolla/
cp ~/kolla-ansible/ansible/inventory/all-in-one /etc/kolla/inventory
kolla-genpwd
</code></pre>
<h1 id="configure-networking">Configure Networking</h1>
<p>The Intel NUC only has one Ethernet port, but Kolla-Ansible requires two ports.
One port is for the APIs, and another can't have any addresses on it as it ends
up being controlled by OpenVSwitch and Neutron.</p>
<p>To get around this, I use VLAN subinterfaces. This works well, but it does
require that you add a layer 3 route from the overcloud subnet to the
undercloud API network the guests need to connect to the host.</p>
<p>OVS creates a bridge bound to the physical port, where the VLAN modules
work alongside it. The only drawback is you can't use those VLANs in the
overcloud.</p>
<p><code>vi /etc/netplan/01-netcfg.yaml</code></p>
<pre><code class="language-text">network:
  version: 2
  renderer: networkd
  ethernets:
      eno1:
        dhcp4: false
        dhcp6: false
  vlans:
    vlan.2:
      id: 2
      link: eno1
      addresses: [ 10.254.2.2/24 ]
      gateway4: 10.254.2.1
      nameservers:
        addresses: ["8.8.8.8", "8.8.4.4"]
</code></pre>
<p>Apply the changes. This might kick you.</p>
<pre><code class="language-bash">netplan try
</code></pre>
<h1 id="enable-vlans-for-neutron">Enable VLANs for Neutron</h1>
<p>Create this config file to enable VLAN support</p>
<pre><code class="language-bash">mkdir -p /etc/kolla/config/neutron
</code></pre>
<p><code>vi /etc/kolla/config/neutron/ml2_conf.ini</code></p>
<pre><code class="language-ini">[ml2_type_vlan]
network_vlan_ranges = physnet1:1:4094
</code></pre>
<h1 id="globals-file">Globals file</h1>
<p><code>vi /etc/kolla/globals.yml</code></p>
<pre><code class="language-yaml">---

kolla_internal_vip_address: "10.254.2.254"
neutron_external_interface: "eno1"
network_interface: "vlan.2"
keepalived_virtual_router_id: "66"

kolla_base_distro: "ubuntu"
kolla_install_type: "source"
openstack_release: "rocky"

enable_keepalived: "yes"
enable_mariadb: "yes"
enable_memcached: "yes"
enable_rabbitmq: "yes"
enable_chrony: "yes"
enable_fluentd: "yes"
enable_nova_ssh: "yes"
enable_ceph: "no"
enable_horizon: "yes"

enable_haproxy: "yes"
kolla_enable_tls_external: "yes"

enable_nova: "yes"
nova_compute_virt_type: "kvm"

enable_keystone: "yes"
keystone_admin_user: "admin"
keystone_admin_project: "admin"

enable_glance: "yes"
glance_backend_file: "yes"

enable_cinder: "yes"
enable_cinder_backup: "no"
enable_cinder_backend_lvm: "yes"
cinder_volume_group: "cinder-volumes"
cinder_backend_ceph: "no"

enable_neutron: "yes"
enable_neutron_lbaas: "yes"
neutron_extension_drivers:
  - name: "port_security"
    enabled: true
  - name: "dns"
    enabled: true

enable_magnum: "yes"
enable_octavia: "no"
default_docker_volume_type: "local-lvm"
</code></pre>
<hr>
<h1 id="install-openstack-using-kolla-ansible">Install OpenStack using Kolla-Ansible</h1>
<pre><code class="language-bash">kolla-ansible certificates
kolla-ansible -i /etc/kolla/inventory bootstrap-servers
kolla-ansible -i /etc/kolla/inventory deploy
kolla-ansible -i /etc/kolla/inventory post-deploy
</code></pre>
<hr>
<h1 id="overcloud-cluster-configuration">Overcloud Cluster Configuration</h1>
<p>The OpenStack services are deployed, now to configure the cloud to be usable.</p>
<p>For details about these commands, see my previous VM guide <a href="/openstack-aio-ka-vm.html">here</a>.</p>
<p>These commands are ran right on the physical OpenStack host.</p>
<pre><code class="language-bash"># Authenticate to the cloud
source /etc/kolla/admin-openrc.sh

# Define the volume type in cinder
openstack volume type create --public local-lvm

# Create a provider network on VLAN 3
openstack network create --external --share \
  --provider-physical-network physnet1 --provider-network-type vlan \
  --provider-segment 3 --disable-port-security vlan3-net

# Assign a subnet to the vlan3-net network
openstack subnet create \
  --dhcp --network vlan3-net \
  --subnet-range 10.254.3.0/24 \
  vlan3-subnet

# Create a small cirros-sized flavor
openstack flavor create tiny --disk 1 --vcpus 1 --ram 256

# Download and import Cirros as a glance image
wget http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
apt-get install -y qemu-utils
qemu-img convert -f qcow2 -O raw cirros-0.4.0-x86_64-disk.img cirros.raw
openstack image create --container-format bare --disk-format raw \
  --file cirros.raw --public "cirros"

# Open up the firewall rules for the admin project
proj_id=$(openstack project list | grep admin | awk '{print $2}')
group_id=$(openstack security group list | grep $proj_id | awk '{print $2}')

openstack security group rule create --proto icmp $group_id
openstack security group rule create --proto tcp --dst-port 1:65535 $group_id
openstack security group rule create --proto udp --dst-port 1:65535 $group_id
</code></pre>
<hr>
<h1 id="create-a-test-vm">Create a test VM</h1>
<pre><code class="language-bash">openstack server create --image cirros --flavor tiny --network vlan3-net test
</code></pre>
<p>The cloud is now ready for use. If layer 3 routing from VLAN 2 to VLAN 3 is
configured, then the VM will be accessible.</p>


<h3 id="table-of-contents">Table of Contents</h3>
<ul>
<li><a href="#environment">Environment</a>
<ul>
<li><a href="#openstack-on-openstack-considerations">OpenStack-on-OpenStack Considerations</a>
<ul>
<li><a href="#disable-port-security">Disable Port Security</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#update-the-os">Update the OS</a></li>
<li><a href="#create-volume-group-for-cinder">Create Volume Group for Cinder</a></li>
<li><a href="#install-ansible">Install Ansible</a></li>
<li><a href="#install-kolla-ansible">Install Kolla-Ansible</a>
<ul>
<li><a href="#check-out-the-kolla-ansible-code">Check out the Kolla-Ansible code</a></li>
</ul>
</li>
<li><a href="#generate-openstack-config-files">Generate OpenStack Config Files</a>
<ul>
<li><a href="#openstack-service-config-files">OpenStack service config files</a></li>
<li><a href="#globals-file">Globals file</a></li>
<li><a href="#passwords-file">Passwords file</a></li>
<li><a href="#ansible-inventory">Ansible Inventory</a></li>
</ul>
</li>
<li><a href="#install-openstack-using-kolla-ansible">Install OpenStack using Kolla-Ansible</a>
<ul>
<li><a href="#install-the-openstack-command-line">Install the OpenStack Command Line</a></li>
</ul>
</li>
<li><a href="#overcloud-cluster-configuration">Overcloud Cluster Configuration</a>
<ul>
<li><a href="#source-the-admin-openrc-file">Source the Admin OpenRC file</a></li>
<li><a href="#create-a-cinder-volume-type">Create a Cinder Volume Type</a></li>
<li><a href="#create-a-flat-provider-network">Create a Flat Provider Network</a></li>
<li><a href="#up-the-interface">Up the interface</a></li>
<li><a href="#create-a-vxlan-for-workloads">Create a VXLAN for workloads</a></li>
<li><a href="#create-a-flavor">Create a flavor</a></li>
<li><a href="#create-an-image">Create an image</a></li>
<li><a href="#open-up-the-firewall">Open up the firewall</a></li>
</ul>
</li>
<li><a href="#create-a-test-vm">Create a test VM</a></li>
</ul>
<h1 id="environment">Environment</h1>
<p>These specs are what was used while writing this guide.
They are neither the minimum nor recommended requirements for a virtual
OpenStack cluster.</p>
<ul>
<li>VM hosted by <a href="https://breqwatr.com">Breqwatr</a>'s on-prem OpenStack
<ul>
<li>AWS/GCP/Azure can also host suitable VMs</li>
</ul>
</li>
<li>Ubuntu 18.04</li>
<li>2 vSSD volumes: 200GB for the OS and 300GB for Cinder LVM</li>
<li>2 VLAN ports: API interface and neutron interface</li>
<li>Docker images are the public Rocky images from Kolla on Docker Hub</li>
</ul>
<h2 id="openstack-on-openstack-considerations">OpenStack-on-OpenStack Considerations</h2>
<p>Skip this section if the installation is not on OpenStack.</p>
<h3 id="disable-port-security">Disable Port Security</h3>
<p>Turn off neutron port-security on each port.
Otherwise the unknown IPs will be blocked.</p>
<pre><code class="language-bash">for x in \
    3f664e5e-dd67-4fe5-b2cc-86c2918b6a2f \
    c44da3d5-00e7-4e41-af39-7010d70ac6a5; do
  echo $x
  openstack port set --no-security-group $x
  openstack port set --disable-port-security $x
done
</code></pre>
<hr>
<h1 id="update-the-os">Update the OS</h1>
<p>It will never be less risky to run an update than right now!</p>
<pre><code class="language-bash">apt-get update &#x26;&#x26; apt-get upgrade -y
</code></pre>
<hr>
<h1 id="create-volume-group-for-cinder">Create Volume Group for Cinder</h1>
<p>OpenStack Cinder supports a huge <a href="https://docs.openstack.org/cinder/rocky/reference/support-matrix.html">list of back-end storage providers</a>,
but the simplest to set up is just using an LVM volume group on the host.</p>
<p>Cinder will use LVM to carve out logical volumes for each cloud block storage
device. This isn't highly available, but that's OK for a dev cloud.</p>
<p>Create a volume group using a dedicated disk.
In this example, I'll use <code>/dev/vdb</code> for VMs.</p>
<p>The name of this volume group is defined in the globals.yml file as
<code>cinder_volume_group</code>, and can be changed as needed.</p>
<pre><code class="language-bash">pvcreate /dev/vdb
vgcreate cinder-volumes /dev/vdb
</code></pre>
<hr>
<h1 id="install-ansible">Install Ansible</h1>
<p>Kolla-Ansible requires that Ansible be pre-installed. Apt can install it too,
but pip tends to install a newer version. Some old versions aren't compatible
with Kolla-Ansible.</p>
<pre><code class="language-bash">apt-get -y install python python-pip
pip install ansible
</code></pre>
<hr>
<h1 id="install-kolla-ansible">Install Kolla-Ansible</h1>
<h2 id="check-out-the-kolla-ansible-code">Check out the Kolla-Ansible code</h2>
<p>Git comes pre-installed on many Ubuntu 18.04, but install it otherwise.</p>
<pre><code class="language-bash">apt-get install -y git
</code></pre>
<p>Clone a stable branch of Kolla-Ansible.
Check out the code and use pip to install it.</p>
<p>Make sure to use the stable branch associated with Rocky. Kolla-Ansible doesn't
work right when installing versions of OpenStack that don't match git stable
branch used.</p>
<pre><code class="language-bash">cd ~
git clone https://github.com/openstack/kolla-ansible.git
cd ~/kolla-ansible
git checkout stable/rocky
pip install .
</code></pre>
<hr>
<h1 id="generate-openstack-config-files">Generate OpenStack Config Files</h1>
<h2 id="openstack-service-config-files">OpenStack service config files</h2>
<p>Create a directory to store the OpenStack config changes.</p>
<pre><code class="language-bash">mkdir -p /etc/kolla/config
</code></pre>
<h2 id="globals-file">Globals file</h2>
<p>The globals file contains most of the important deployment settings for KA.</p>
<p>Check these two links for more info:</p>
<ul>
<li><a href="https://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration.html">openstack.org KA advanced reference</a></li>
<li><a href="https://github.com/openstack/kolla-ansible/blob/master/etc/kolla/globals.yml">GitHub page for globals file</a></li>
</ul>
<p>Any configs not defined are set by defaults.yml files in KA's Ansible roles.</p>
<p>The globals file can specify your OpenStack version, where it will pull its
containers from, and which containers will be deployed. Be sure to replace
<code>kolla_internal_vip_address</code>'s value with your VIP address.</p>
<p>The VIP can't be used by your existing network card, else HAproxy will throw a
fit and Kolla-Ansible will fail at the MariaDB step.</p>
<p>The networking in this example is as simple as it gets. You can really split
things up in this file if you want to. Note the network interface name too, it
may be different from yours.</p>
<p><code>vi /etc/kolla/globals.yml</code></p>
<pre><code class="language-yaml">---
# APIs listen on this VIP
kolla_internal_vip_address: "10.100.202.254"

# Interface used for VXLANS and APIs. See docs to split this out into many nics
network_interface: "ens3"

# Used for the VLANS, give it the whole device
neutron_external_interface: "ens6"

# If you have more than one install in the broadcast domain, make this unique
keepalived_virtual_router_id: "77"

# Define the docker image names and tag (kolla/ubuntu-source-&#x3C;name>:rocky)
kolla_base_distro: "ubuntu"
kolla_install_type: "source"
openstack_release: "rocky"

# Which containers will be deployed
enable_keepalived: "yes"
enable_mariadb: "yes"
enable_memcached: "yes"
enable_rabbitmq: "yes"
enable_chrony: "yes"
enable_fluentd: "yes"
enable_nova_ssh: "yes"
enable_ceph: "no"
enable_horizon: "yes"

# Use HAProxy load balancer but disable TLS certs
enable_haproxy: "yes"
kolla_enable_tls_external: "yes"

# Use qemu for nested hypervisors, kvm for metal
enable_nova: "yes"
nova_compute_virt_type: "qemu"

# The default username and project can be changed here
enable_keystone: "yes"
keystone_admin_user: "admin"
keystone_admin_project: "admin"

# Glance stores VM templates. Use a file backend with cinder's LVM backend
enable_glance: "yes"
glance_backend_file: "yes"

# Configure Cinder to use the 'cinder-volumes' vg for virtual block devices
enable_cinder: "yes"
enable_cinder_backup: "no"
enable_cinder_backend_lvm: "yes"
cinder_volume_group: "cinder-volumes"
cinder_backend_ceph: "no"

# Enable the DNS ml2 extension driver in neutron &#x26; lbaas
enable_neutron: "yes"
enable_neutron_lbaas: "yes"
neutron_extension_drivers:
  - name: "port_security"
    enabled: true
  - name: "dns"
    enabled: true

# Configure magnum to look for the 'local-lvm' cinder volume type
enable_magnum: "yes"
default_docker_volume_type: "local-lvm"
</code></pre>
<h2 id="passwords-file">Passwords file</h2>
<p>The passwords file is empty initially, its just a template to be filled in with
random password from KA. The <code>kolla-genpwd</code> command populates it.</p>
<pre><code class="language-bash">cp ~/kolla-ansible/etc/kolla/passwords.yml /etc/kolla
kolla-genpwd
</code></pre>
<h2 id="ansible-inventory">Ansible Inventory</h2>
<p>The KA Ansible inventory defines which roles will be assigned to which hosts.
The all-in-one inventory file assigns all roles to localhost.</p>
<pre><code class="language-bash">cp ~/kolla-ansible/ansible/inventory/all-in-one /etc/kolla/inventory
</code></pre>
<hr>
<h1 id="install-openstack-using-kolla-ansible">Install OpenStack using Kolla-Ansible</h1>
<p>First run the bootstrap command to install docker and tune the server. In older
branches it will add the wrong key for the apt repo, so do that yourself.</p>
<pre><code class="language-bash">kolla-ansible certificates
kolla-ansible -i /etc/kolla/inventory bootstrap-servers
kolla-ansible -i /etc/kolla/inventory deploy
kolla-ansible -i /etc/kolla/inventory post-deploy
</code></pre>
<h2 id="install-the-openstack-command-line">Install the OpenStack Command Line</h2>
<p>This will enable the <code>openstack</code> command.</p>
<pre><code class="language-bash">pip install python-openstackclient python-neutronclient python-magnumclient
</code></pre>
<p>To configure the OpenStack CLI, source the openrc file that was created by KA's
post-deploy step.</p>
<pre><code class="language-bash">source /etc/kolla/admin-openrc.sh
</code></pre>
<hr>
<h1 id="overcloud-cluster-configuration">Overcloud Cluster Configuration</h1>
<p>Now that the undercloud is functional and the OpenStack services are running,
its time to configure the overcloud to be useful.</p>
<p>These commands are ran directly on the new virtual OpenStack host.</p>
<h2 id="source-the-admin-openrc-file">Source the Admin OpenRC file</h2>
<p>Kolla-Ansible's post-deploy task created an openrc file which can be used to
authenticate as the admin user it created. It's a good idea to make your own
user and openrc file, but for now this will do.</p>
<pre><code class="language-bash">source /etc/kolla/admin-openrc.sh
</code></pre>
<h2 id="create-a-cinder-volume-type">Create a Cinder Volume Type</h2>
<p>If no cinder volume type has been created, one must be made. These is required
for Magnum and for other storage backends, and also good practice.</p>
<pre><code class="language-bash">openstack volume type create --public local-lvm
</code></pre>
<h2 id="create-a-flat-provider-network">Create a Flat Provider Network</h2>
<p>Since this is OpenStack on OpenStack, using VLANs for the provider network is
more trouble than its worth. Make a flat network that shares the hosts
interface to provide workloads external network connectivity.</p>
<pre><code class="language-bash">openstack network create --share --external --disable-port-security \
  --provider-physical-network physnet1 --provider-network-type flat \
  infra-net

openstack subnet create \
  --no-dhcp --network infra-net \
  --allocation-pool start=10.100.202.180,end=10.100.202.199 \
  --subnet-range 10.100.202.0/24 \
  infra-subnet
</code></pre>
<h2 id="up-the-interface">Up the interface</h2>
<p><code>physnet1</code> is bound to <code>ens6</code> in this example, but that interface is down. Start
the interface.</p>
<pre><code class="language-bash">ip link set dev ens6 up
</code></pre>
<p>This is sort of a pain since it doesn't happen on reboot. There's a
<a href="https://bugs.launchpad.net/netplan/+bug/1763608">bug</a> in netplan about it, so
you can't just make a netplan entry.</p>
<p>One solution would be to make a crontab entry for <code>@reboot</code>, but I won't be
covering that here.</p>
<h2 id="create-a-vxlan-for-workloads">Create a VXLAN for workloads</h2>
<p>Create a VXLAN for VMs to use and plug it into the host's network using the
flat provider network.</p>
<pre><code class="language-bash">openstack network create vx1-net

openstack subnet create --network vx1-net --dhcp \
  --subnet-range 192.168.0.0/24 vx1-subnet

openstack router create --no-ha vx1-router
openstack router set --external-gateway infra-net vx1-router
openstack router add subnet vx1-router vx1-subnet
</code></pre>
<p>This flat provider network will now provide outbound internet access and the
ability to connect into workloads using floating IP addresses.</p>
<h2 id="create-a-flavor">Create a flavor</h2>
<pre><code class="language-bash">openstack flavor create tiny --disk 1 --vcpus 1 --ram 256
</code></pre>
<h2 id="create-an-image">Create an image</h2>
<p>This is the template used to launch our test VM</p>
<pre><code class="language-bash">wget http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
apt-get install -y qemu-utils
qemu-img convert -f qcow2 -O raw cirros-0.4.0-x86_64-disk.img cirros.raw
openstack image create --container-format bare --disk-format raw \
  --file cirros.raw --public "cirros"
</code></pre>
<h2 id="open-up-the-firewall">Open up the firewall</h2>
<p>Get the ID of the security group for the scoped project, in this case <code>admin</code>,
then open up ICMP, TCP, and UDP for that project.</p>
<pre><code>proj_id=$(openstack project list | grep admin | awk '{print $2}')
group_id=$(openstack security group list | grep $proj_id | awk '{print $2}')

openstack security group rule create --proto icmp $group_id
openstack security group rule create --proto tcp --dst-port 1:65535 $group_id
openstack security group rule create --proto udp --dst-port 1:65535 $group_id
</code></pre>
<hr>
<h1 id="create-a-test-vm">Create a test VM</h1>
<pre><code class="language-bash">openstack server create --image cirros --flavor tiny --network vx1-net test
# This server was assigned 192.168.0.13

openstack floating ip create infra-net
# The floating IP created was 10.100.202.226

openstack server add floating ip \
  --fixed-ip-address 192.168.0.13 test 10.100.202.226
</code></pre>
<p>You can now ping or SSH to your VM. The cloud is ready for use.</p>


Kyle Pericak

Adding Virtio drivers for KVM to RHEL's initrd

OpenStack: Fixing the CLI (python2) error: No module named queue

OpenStack Cloudbase-init: PowerShell user_data Scripts

Injecting KVM Drivers to Windows 10 for OpenStack

Operating OpenStack from Ansible

Project: OpenStack Deep Dive

Modifying OpenStack Kolla Docker Images

Install Non-Prod OpenStack on a Physical Server

Install OpenStack with Kolla-Ansible in a VM