Skip to content

Latest commit

 

History

History
67 lines (54 loc) · 4.54 KB

domainmgr.md

File metadata and controls

67 lines (54 loc) · 4.54 KB
Raw

Domain Management Agent in EVE (aka domainmgr in code)

Overview

Domainmgr is the interface to the hypervisor to start DomU instances:

  • It starts, stops, and tracks the domUs
  • It includes doing device assignment, i.e. assigning peripheral devices to DomU based on config.
  • It has retry logic for when a domU fails to boot
  • It reports metrics such as CPU and memory from the domains
  • As part of device assignment it also manages the loading and unloading of USB kernel modules based on the debug.enable.usb configuration knob.

Key Input/Output

Domain Manager interacts with the Cloud controller (e.g. zedcloud) indirectly using four key messages:

  • DomainConfig from controller is fed to domainmgr. This contains the configuration information about a DomU instance. DomainConfig is defined in pillar/types/domainmgrtypes.go
  • DomainStatus is fed from domainmgr to controller. This contains the operational information about a DomU instance. DomainStatus is defined in pillar/types/domainmgrtypes.go
  • DomainMetric with CPU and memory metrics for the host (dom0) and the applications
  • PhysicalIOAdapterList specifes the set of physical I/O adapters which can potentially be used for adapter assignment.

zedmanager publishes DomainConfig and subscribes to DomainStatus, while zedagent publishes PhysicalIOAdapterList and subscribes to DomainMetric.

PCI Device Management

  • XEN uses a PCI backend driver called pciback, to provide PCI passthrough to DomU instances. For more details, see https://wiki.xen.org/wiki/Xen_PCI_Passthrough
  • Domain Manager assigns PCI devices that are not in use in Dom0 to pciback driver for use in DomU instances, i.e. all PCI networking devices are assigned to pciback unless they are a port in DeviceNetworkStatus (from pillar/cmd/nim), and all USB controllers are assigned to pciback unless debug.enable.usb is set to true in Dom0 configuration.
  • Note that the assigning away doesn’t happen until domainmgr starts. domainmgr starts once the device has been successfully onboarded in the Cloud controller.
  • Since each hardware model can have different set of network or USB adapters, for every hardware model, the controller provides a PhysicalIO in the API which zedagent publishes as PhysicalIOAdapterList.

Internal Operation

  • Domain Manager implementation uses separate go routine for each key in DomainConfig
  • Watches for status changes such as halted, or reboot (when the domain ID changes) and reports those in DomainStatus
  • Avoids head-of-line blocking during large copy or slow shutdown operations
  • Copies a read/write virtual disk configured for the guest domain, to a unique one in /persist/img/. This /persist/img/ path is fed in the xl config file to XEN, to create the guest domain.
  • If Activate=false in DomainConfig, or if the DomainStatus deleted then Domain Manager halts the domU
  • When halting Domain manager first attempts a graceful shutdown; if the domU doesn’t shut down, it does a poweroff
  • Creates a xl config file in /run/domainmgr/xen/xen*.cfg. xl is a XEN command to manage XEN guest domains. For more details, see https://xenbits.xen.org/docs/unstable/man/xl.1.html. Sample xl config is given below:
137f6bf6-a581-4193-a8ec-54c44124d367:/run/domainmgr/xen# cat xen1.cfg
#This file is automatically generated by domainmgr
name = "vyos-app.1"
type = "hvm"
uuid = "4a144db0-6b63-405a-b884-7760042023b1"
vnc = 1
vnclisten = "0.0.0.0"
usb=1
usbdevice=["tablet"]
memory = 500
maxmem = 500
vcpus = 1
maxcpus = 1
serial = "pty"
boot = "dc"
disk = ['/persist/img/5E0BEBE7E83EDE141C3B86B3D9E3EE9105F2E93C201F6CD5912BE9EBC965F19B-4a144db0-6b63-405a-b884-7760042023b1.qcow2,qcow2,xvda,rw']
vif = ['bridge=bn2,vifname=nbu1x1,mac=00:16:3e:00:01:01']
pci = [ '07:00.0']

Debugging

  • Look at the respective input/output files:
  • /run/zedmanager/DomainConfig and /run/domainmgr/DomainStatus shows the key input and output
  • We’ve seen cases where PV doesn’t boot but HVM does due to a missing dom0 qemu process (started by /etc/init.d/xencommons). So please check if /usr/lib/xen/bin/qemu-system-i386 -xen-domid 0 is running
  • If domUs fail to boot domainmgr will retry after 10 minutes. You can control it with timer.boot.retry global configuration variable. The option value is in seconds. e.g. if you want to change retry time to 1 minute, then use --config=timer.boot.retry:60
  • If USB keyboard disappears (stops working after boot) could be that domainmgr didn’t initially start; For instance, during onboarding domainmgr does not run hence USB is open