Network Architecture

Overview

Mistify will use Open vSwitch (OVS) as the primary component of its network virtualization features. OVS is mature software which implements a flexible combination of the popular network virtualization methods, from basic VLANs to VXLANs and GRE tunnels. Initially, for Mistify, we will implement basic VLAN support into the agent, and then progress agent support to more complicated arrangements such as VXLAN.

An OVS instance can operate and be managed singularly on a given host, or join into a common configuration realm with other hosts running OVS, enabling configurations on all hosts to be centrally managed from one. The protocol in which OVS clients talk with the OVS configuration database is documented as a IETF standard in RFC 7047. With that, several vendors have baked OVSDB support directly into their devices, such as Juniper Networks' Junos OS.

Basic OVS Architecture

Administratively, OVS is centered around a OVS-created "bridge" interface. This bridge (or, optionally, a OVS-created LACP bond of two or more ethernet interfaces) is what connects the hypervisor's physical ethernet interface(s) to the virtual TAP-type interfaces which in turn are consumed by the guests.

In a basic VLAN-style setup, OVS is told which VLAN IDs contained in packets coming in on the physical interfaces to route to a specific TAP interface. In this way, a guest can be isolated into its own broadcast domain in the same way one would use VLANs on a physical host. See Figure A.

Figure A

A drawback of the basic VLAN method is that additional administrative overhead is required to ensure that the physical switching infrastructure is also be aware of the VLAN IDs being allocated to specific physical ports. This is not so much an issue if the VLAN port allocation policy is "everything gets everything", however this policy tends to be the exception and not the rule out in the wild.

Eventually, Mistify (via OVS) should support additional types of network virtualization, such as VXLAN. VXLANs aim to solve the above-mentioned administrative overhead problem of involving the physical switching infrastructure in each VLAN assignment and allow VLANs to be purely software-defined within a group of participating hypervisors. The requirement of the switching infrastructure to have carnal knowledge of VLANs is avoided as VxLAN ethernet frames are encapsulated within UDP.

MistifyOS+OVS Implementation

At a basic level, all physical network interfaces (eth/eno) and virtual devices (tap) are managed by systemd-networkd, the network management portion of systemd. The bridge or bond device which OVS uses to glue these together is only managed by OVS itself. OVS cannot use a pre-existing bridge or bond.

All OVS manipulation is done via the ovs-vsctl command. This client interacts with the ovs-vswitchd daemon, which in turn programs the openvswitch.ko kernel module and general Linux network components to set up packet routing within the kernel.

Creating a OVS bridge

1. This will initialize the OVS management bridge with the name of mistify0. The name can be arbitrary.

   ovs-vsctl add-br mistify0

2. Now we add a physical device to the bridge, thereby availing it to the physical network.

   ovs-vsctl add-port mistify0 eno2

   Important Note: Any interfaces joined to the OVS bridge must be otherwise unused or unmanaged by software external to OVS.

3. Removing ports and the bridge itself can be done by using the del-port and del-br subcommands.

systemd Involvement

systemd still has a role in MistifyOS+OVS. It is required in two ways:

1. Setting up all physical ethernet devices on boot
2. Setting up all virtual network devices, such as TAP, TUN, IP tunneling, and bond devices.

Physical Ethernet

Setup is accomplished by a basic ethernet.network file in /etc/systemd/network with contents similar to:

[Match]
Name=en*

TAP Interfaces

Each guest (KVM, Docker) gets its own TAP type interface to listen use. Each TAP interface is given its own configuration NetDev file under /etc/systemd/network. The name of the TAP interface is arbitrary, for example:

-rw-r--r-- 1 root root 44 May 21 21:39 custAvlan420.netdev
-rw-r--r-- 1 root root 44 May 22 14:22 custBvlan666.netdev
-rw-rw-r-- 1 root root 17 May 21 18:01 ethernet.network

Contents of the netdev files are very simple:

[root@44454c4c network]# cat custAvlan420.netdev 
[NetDev]
Name=custAvlan420
Kind=tap

[Tap]

The format of a .netdev file is documented is documented in the systemd-netdev man page. Specific options for TAP interfaces are available. The only ones which may be of interest to us are the User and Group parameters, to set the UID/GID of /dev device for that interface.

Adding a .netdev file to a already-running system involves emplacing the file, and then executing the following command:

systemctl restart systemd-networkd

Wiring it all together

Bringing a TAP interface into the OVS bridge is done in a similar way as joining a physical interface to it, only we specify the VLAN ID we wish a specific TAP interface to receive and send packets. Using our above examples, we would do the following:

ovs-vsctl add-port mistify0 custAvlan420 tag=420

This assigns TAP interface custAvlan420 to the bridge, and the bridge will pass packets which have the VLAN ID of 420 to it. The KVM or Docker guest is then initialized and given custAvlan420 to bind to.

In the end, we have a basic bridge setup as such:

[root@44454c4c ~]# ovs-vsctl show
e91ef24b-be8b-43a0-afe1-29f88ca6ae4e
    Bridge "mistify0"
        Port "eno3"
            Interface "eno3"
        Port "mistify0"
            Interface "mistify0"
                type: internal
        Port "custAvlan420"
            tag: 420
            Interface "custAvlan420"
    ovs_version: "2.3.90"

Programming Resources

There does not appear to be an official Golang library for OVS, however it does appear that someone has taken a stab at this:

https://github.com/socketplane/libovsdb