-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathdesign.txt
133 lines (112 loc) · 5.84 KB
/
design.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
1 Overview
The purpose of vDPA framework is to abstract the device and drivers to
be fit for all types of vDPA implementation.
+-------------------------+
|Process |
| +----------------+ |
| | virtio driver | |
| +----+----+------+ |
+----------+----+---------+
userspace | |
=======================+====+==============================================
kernel | |
+------+ +--+ +----------------+
| | | virtio driver |
+---+--+ +----+----+ +--------+-------+
| VFIO | | vhost | |
+---+--+ +----+----+ -------+-+-------- virtio bus
| | |
+----+-----+ | +-----+-------+
|vhost-mdev| | | virtio-vdpa |
+----+-----+ | +---+---------+
| | |
vdpa bus -----+-+---------+----+------+-------+---------
| | |
+---+----+ +---+----+ +---+----+
| MDEV | | VF/PF | | SUBDEV |
| DRV | | DRV | | DRV |
+----+---+ +---+----+ +---+----+
kernel | | |
===============+==========+===========+====================================
hardware | | |
+----+ +-----+---+ +----+
| | | |
+--+--+ +--+---+ +--+---+ +-----+------+
| ADI | | VF | | PF | | SUB FUNC |
+-----+ +------+ +------+ +------------+
2 vDPA bus, device and driver
As illustrated by the above figure. The core of vDPA framework is a
software vpda bus, it abstract the common operations that is used for
communicating between vDPA device and vDPA drivers. Those operations
could be categorized into three different types:
- virtio specific operations: for controlling the virtio device
- interrupt support: the operations that do interrupt relay or
directly interrupt injection (e.g as post interrupt support by VTD).
- doorbell support: the operations that relay doorbell or directly
doorbell mapping to userspace
- migration support: operations to support dirty pages tracking
Any physical device drivers that can support those operations can be
registered to the bus as vDPA device to be driven by vDPA bus
driver. vDPA physical devices are usually implemented with the
following types:
1) PF (Physical Function): A single Physical Function.
2) VF (Virtual Function): Device that supports single root I/O
virtualization (SRIOV), its Virtual Function (VF) represents a
virtualized instance of the device that can be assigned to
different partitions.
3) ADI (Assignable Device Interface): With technology like Intel
Scalable IOV, a device Physical Function may be configured to
support multiple light-weight Assignable Device Interfaces (ADIs)
that can be assigned to different partitions as virtual
devices.
4) Sub Device: Vendor specific interface to slice the Physical
Function to multiple sub functions that can be assigned to
different partitions as virtual devices.
Any driver that use vDPA operations to communicate with the vDPA
device is a vDPA bus driver. 3 different vDPA bus driver was planned
to be implemented:
1) virtio-vdpa driver: this vDPA bus driver implements a new virtio
transport through vdpa bus for kernel virtio driver. This allows
kernel virtio driver to control the vDPA hardware.
2) VFIO driver: this vDPA bus driver exposes a VFIO compatible uABI
for userspace driver and allows the vDPA hardware to be controlled
by userspace driver through a dedicated mdev driver vhost-mdev
driver.
3) vhost driver: this vDPA bus driver exposes a vhost compatible uABI
for userspace driver.
3 DMA model
The bus does have any mandatory DMA model, it's the responsibility of
bus driver and physical device driver to determine the DMA model.
For kernel virtio driver which depends on DMA API to work, DMA
operations was initiated from virtio driver of upper layer by using
its parent so it's the responsibility of virtio-vpda that correctly
set the parent device of virtio device.
For userspace driver, userspace may choose one of the two API to use
depends on whether device have on chip IOMMU:
- For the device that depends on the system IOMMU to work. Userspace
may choose to use VFIO driver and its API to do DMA. It's the
charge of vhost-mdev to attach the device to correct IOMMU
group. That means the DMA setup was done through VFIO container fd
as a typical assigned device. And the device configuration needs to
be done by using vhost device ioctls on VFIO device fd.
- For the device that has its own IOMMU, userspace need to choose to
use vhost API. It's then the responsibility of vhost to establish
the communication to the onchip IOMMU. It means DMA mapping and
unmapping will be done through vhost ioctl, and vhost will pass
those commands to physical vDPA driver to program the on chip
IOMMU. Device configuration needs to be done directly through vhost
as well.
Physical device driver also need to specify the DMA model it supports
when registering to vdpa bus:
1) "virtio": means it supports kernel virtio driver, DMA API could be
used from kernel virtio driver with the help of virito vdpa
transport.
2) "vfio": means it supports VFIO/vhost-mdev driver, VFIO DMA API
could be used from userspace.
3) "vhost": means it support generic vhost driver, VHOST DMA API could
be used from userspace.
4 Management interface:
The framework does not have any mandatory interface:
- sysfs will be used when device is created through mdev or VF
- when device is not created through mdev, any other API (e.g devlink)
could be used