Skip to content

Latest commit

 

History

History
286 lines (212 loc) · 8.66 KB

Networking.asciidoc

File metadata and controls

286 lines (212 loc) · 8.66 KB

Networking in OpenQA

This overview is valid only when using QEMU backend!

Which networking type to use is controlled by the NICTYPE variable. If unset or empty NICTYPE defaults to user, ie qemu user networking which requires no further configuration.

For more advanced setups or tests that require multiple jobs to be in the same networking the TAP or VDE based modes can be used.

Qemu user networking

With Qemu [user networking](http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29) each jobs gets it’s own isolated network with TCP and UDP routed to the outside. DHCP is provided by qemu. The MAC address of the machine can be controlled with the NICMAC variable. If not set, it’s 52:54:00:12:34:56.

TAP based network

os-autoinst can connect qemu to TAP devices of the host system to leverage advanced network setups provided by the host by setting NICTYPE=tap.

The TAP device to use can be configured with the TAPDEV variable. If not set defined, ist’s automatically set to "tap" + ($worker_instance - 1), i.e. worker1 uses tap0, worker 2 uses tap1 and so on.

For multiple networks per job (see NETWORKS variable), the following numbering scheme is used:

worker1: tap0 tap64 tap128 ...
worker2: tap1 tap65 tap129 ...
worker3: tap2 tap66 tap130 ...
...

MAC address of virtual NIC is controlled by NICMAC variable or automatically computed from $worker_id if not set.

In TAP mode the system administrator is expected to configure the network, required internet access etc on the host manually.

TAP devices need be owned by the _openqa-worker user for openQA to be able to access them.

tunctl -u _openqa-worker -p -t tap0

or Wicked way:

#/etc/sysconfig/network/ifcfg-tap0
BOOTPROTO='none'
IPADDR=''
NETMASK=''
PREFIXLEN=''
STARTMODE='auto'
TUNNEL='tap'
TUNNEL_SET_GROUP='nogroup'
TUNNEL_SET_OWNER='_openqa-worker'

If you want to use TAP device which doesn’t exist on the system, you need to set CAP_NET_ADMIN capability on qemu binary file:

zypper in libcap-progs
setcap CAP_NET_ADMIN=ep /usr/bin/qemu-system-x86_64

Network setup can be changed after qemu is started using network configure script specified in TAPSCRIPT variable.

Sample script to add TAP device to existing bridge br0:

#!/bin/sh

sudo brctl addif br0 $1
sudo ip link set $1 up

TAP with Open vSwitch

The recomended way to configure the network for TAP devices is using Open vSwitch. There is a support service os-autoinst-openvswitch.service which sets vlan number of Open vSwitch ports based on NICVLAN variable - this separates the groups of tests from each other.

NICVLAN variable is dynamically assigned by OpenQA scheduler.

Compared to VDE setup discussed later, Open vSwitch is more complicated to configure, but provides more robust and scalable network.

Start Open vSwitch and add TAP devices:

# start openvswitch.service
systemctl start openvswitch.service
systemctl enable openvswitch.service

#create bridge
ovs-vsctl add-br br0

# add tap devices, use vlan 999 by default, the vlan number is supposed to be changed when the vm starts
ovs-vsctl add-port br0 tap0 tag=999
ovs-vsctl add-port br0 tap1 tag=999
ovs-vsctl add-port br0 tap2 tag=999
ovs-vsctl add-port br0 tap3 tag=999
ovs-vsctl add-port br0 tap4 tag=999

If the workers with TAP capability are spread across multiple hosts, the network must be connected. See Open vSwitch documentation for details.

Example configuration of GRE tunnel:

ovs-vsctl add-port br0 gre0 -- set interface gre0 type=gre options:remote_ip=<IP address of other host>

The virtual machines need access to os-autoinst webserver acessible via IP 10.0.2.2. The IP addresses of VMs are controlled by tests and are likely to conflict if more independent tests runs in parallel.

The VMs have unique MAC that differs in the last 16 bits (see /usr/lib/os-autoinst/backend/qemu.pm). os-autoinst-openvswitch.service sets up filtering rules for the following translation scheme which provide non-conflicting addresses visible from host:

MAC 52:54:00:12:XX:YY -> IP 10.1.XX.YY

That means that the local port of the bridge must be configured to IP 10.0.2.2 and netmask /15 that covers 10.0.0.0 and 10.1.0.0 ranges.

ip addr add 10.0.2.2/15 dev br0
ip route add 10.0.0.0/15 dev br0
ip link set br0 up

and permanently in /etc/sysconfig/network

#/etc/sysconfig/network/ifcfg-br0
BOOTPROTO='static'
IPADDR='10.0.2.2/15'
STARTMODE='auto'

wicked 0.6.23 and later has enhanced support for the creation and configuration of OpenvSwitch bridges.

Note
 In some cases (e.g. on Leap) can be needed to start the OpenvSwitch service before the Network service by modifying the OpenvSwitch service. For reference see this.

The permanent configuration for wicked 0.6.23 and later should look like this:

#/etc/sysconfig/network/ifcfg-br0
BOOTPROTO='static'
IPADDR='10.0.2.2/15'
STARTMODE='auto'
OVS_BRIDGE='yes'
OVS_BRIDGE_PORT_DEVICE_1='tap0'
OVS_BRIDGE_PORT_DEVICE_2='tap1'
OVS_BRIDGE_PORT_DEVICE_3='tap2'

The IP 10.0.2.2 can also serve as a gateway to access outside network. For this, a NAT between br0 and eth0 must be configured with SuSEfirewall or iptables.

# configuration options for NAT with SuSEfirewall
# /etc/sysconfig/SuSEfirewall

FW_ROUTE="yes"
FW_MASQUERADE="yes"
FW_DEV_INT="br0"

Then it is possible to start the os-autoinst-openvswitch.service The service uses br0 by default. It can be configured for another bridge name by setting /etc/sysconfig/os-autoinst-openvswitch

OS_AUTOINST_USE_BRIDGE=bridge_name

Then, start the service:

systemctl start os-autoinst-openvswitch.service
systemctl enable os-autoinst-openvswitch.service

Debugging Open vSwitch configuration

Boot sequence with wicked < 0.6.23:

  1. wicked - creates tap devices

  2. openvswitch - creates the bridge br0, adds tap devices to it

  3. wicked handles br0 as hotplugged device, assignd the IP 10.0.2.2 to it, updates SuSEFirewall

  4. os-autoinst-openvswitch - installs openflow rules, handles vlan assignment

Boot sequence with wicked 0.6.23 and newer:

  1. openvswitch

  2. wicked - creates the bridge br0 and tap devices, add tap devices to the bridge,

  3. SuSEFirewall

  4. os-autoinst-openvswitch - installs openflow rules, handles vlan assignment

The configuration and operation can be checked by the following commands:

ovs-vsctl show

shows the bridge br0, the tap devices are assigned to it

ovs-ofctl dump-flows br0

shows the rules installed by os-autoinst-openvswitch in table=0

  • packets from tapX to br0 create additional rules in table=1

  • packets from br0 to tapX increase packet counts in table=1

  • empty output indicates a problem with os-autoinst-openvswitch service

  • zero packet count or missing rules in table=1 indicate problem with tap devices

ipables -L -v

As long as the SUT has access to external network, there should be nonzero packet count in the forward chain between br0 and external interface.

VDE based network

Virtual Distributed Ethernet provides a software switch that runs in user space. It allows to connect several qemu instances without affecting the system’s network configuration.

The openQA workers need a vde_switch instance running. The workers reconfigure the switch as needed by the job.

Basic, single machine tests

To start with a basic configuration like qemu user mode networking, create a machine with the following settings:

  • VDE_SOCKETDIR=/run/openqa

  • NICTYPE=vde

  • NICVLAN=0

Start switch and user mode networking:

systemctl start openqa-vde_switch
systemctl start openqa-slirpvde

With this setting all jobs on the same host would be in the same network share the same SLIRP instance though.

Multi machine tests

Create a machine like above but don’t set NICVLAN. openQA will dynamically allocate a VLAN number for all jobs that have dependencies between each other. By default this VLAN is private and has no internet access. To enable user mode networking set VDE_USE_SLIRP=1 on one of the machines. The worker running the job on such a machine will start slirpvde and put it in the correct VLAN then.