| Message ID | 20180925231641.4954-27-akrowiak@linux.vnet.ibm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | guest dedicated crypto adapters | expand |
On Tue, 25 Sep 2018 19:16:41 -0400 Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote: > From: Tony Krowiak <akrowiak@linux.ibm.com> > > This patch provides documentation describing the AP architecture and > design concepts behind the virtualization of AP devices. It also > includes an example of how to configure AP devices for exclusive > use of KVM guests. > > Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com> > Reviewed-by: Halil Pasic <pasic@linux.ibm.com> > --- > Documentation/s390/vfio-ap.txt | 782 +++++++++++++++++++++++++++++++++ > MAINTAINERS | 1 + > 2 files changed, 783 insertions(+) > create mode 100644 Documentation/s390/vfio-ap.txt ... > +Example: > +======= > +Let's now provide an example to illustrate how KVM guests may be given > +access to AP facilities. For this example, we will show how to configure > +three guests such that executing the lszcrypt command on the guests would > +look like this: > + > +Guest1 > +------ > +CARD.DOMAIN TYPE MODE > +------------------------------ > +05 CEX5C CCA-Coproc > +05.0004 CEX5C CCA-Coproc > +05.00ab CEX5C CCA-Coproc > +06 CEX5A Accelerator > +06.0004 CEX5A Accelerator > +06.00ab CEX5C CCA-Coproc > + > +Guest2 > +------ > +CARD.DOMAIN TYPE MODE > +------------------------------ > +05 CEX5A Accelerator > +05.0047 CEX5A Accelerator > +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Seems like an unfinished thought here. > + > +Guest2 > +------ > +CARD.DOMAIN TYPE MODE > +------------------------------ > +06 CEX5A Accelerator > +06.0047 CEX5A Accelerator > +06.00ff CEX5A Accelerator > + > +These are the steps: > + > +1. Install the vfio_ap module on the linux host. The dependency chain for the > + vfio_ap module is: > + * iommu > + * s390 > + * zcrypt > + * vfio > + * vfio_mdev > + * vfio_mdev_device > + * KVM > + > + To build the vfio_ap module, the kernel build must be configured with the > + following Kconfig elements selected: > + * IOMMU_SUPPORT > + * S390 > + * ZCRYPT > + * S390_AP_IOMMU > + * VFIO > + * VFIO_MDEV > + * VFIO_MDEV_DEVICE > + * KVM > + > + If using make menuconfig select the following to build the vfio_ap module: > + -> Device Drivers > + -> IOMMU Hardware Support > + select S390 AP IOMMU Support > + -> VFIO Non-Privileged userspace driver framework > + -> Mediated device driver frramework > + -> VFIO driver for Mediated devices > + -> I/O subsystem > + -> VFIO support for AP devices > + > +2. Secure the AP queues to be used by the three guests so that the host can not > + access them. To secure them, there are two sysfs files that specify > + bitmasks marking a subset of the APQN range as 'usable by the default AP > + queue device drivers' or 'not usable by the default device drivers' and thus > + available for use by the vfio_ap device driver'. The sysfs files containing > + the sysfs locations of the masks are: > + > + /sys/bus/ap/apmask > + /sys/bus/ap/aqmask > + > + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs > + (APID). Each bit in the mask, from most significant to least significant bit, > + corresponds to an APID from 0-255. If a bit is set, the APID is marked as > + usable only by the default AP queue device drivers; otherwise, the APID is > + usable by the vfio_ap device driver. > + > + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes > + (APQI). Each bit in the mask, from most significant to least significant bit, > + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as > + usable only by the default AP queue device drivers; otherwise, the APQI is > + usable by the vfio_ap device driver. > + > + The APQN of each AP queue device assigned to the linux host is checked by the > + AP bus against the set of APQNs derived from the cross product of APIDs > + and APQIs marked as usable only by the default AP queue device drivers. If a > + match is detected, only the default AP queue device drivers will be probed; > + otherwise, the vfio_ap device driver will be probed. > + > + By default, the two masks are set to reserve all APQNs for use by the default > + AP queue device drivers. There are two ways the default masks can be changed: > + > + 1. The masks can be changed at boot time with the kernel command line > + like this: > + > + ap.apmask=0xffff ap.aqmask=0x40 > + > + This would give these two pools: > + > + default drivers pool: adapter 0-15, domain 1 > + alternate drivers pool: adapter 16-255, domains 2-255 What happened to domain 0? I'm also a little confused by the bit ordering. If 0x40 is bit 1 and 0xffff is bits 0-15, then the least significant bit is furthest left? Did I miss documentation of that? > + > + 2. The sysfs mask files can also be edited by echoing a string into the > + respective file in one of two formats: > + > + * An absolute hex string starting with 0x - like "0x12345678" - sets > + the mask. If the given string is shorter than the mask, it is padded > + with 0s on the right. If the string is longer than the mask, the > + operation is terminated with an error (EINVAL). And this does say zero padding on the right, but then in the next bullet our hex digits use normal least significant bit right notation, ie. 0x41 is 65, not 82, correct? > + > + * A plus ('+') or minus ('-') followed by a numerical value. Valid > + examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only > + the corresponding bit in the mask is switched on ('+') or off ('-'). The > + values may also be specified in a comma-separated list to switch more > + than one bit on or off. > + > + To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, > + 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding > + APQNs must be removed from the masks as follows: > + > + echo -5,-6 > /sys/bus/ap/apmask > + > + echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask Other than the bit ordering confusion, I like this +/- scheme. > + > + This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, > + 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The > + sysfs directory for the vfio_ap device driver will now contain symbolic links > + to the AP queue devices bound to it: > + > + /sys/bus/ap > + ... [drivers] > + ...... [vfio_ap] > + ......... [05.0004] > + ......... [05.0047] > + ......... [05.00ab] > + ......... [05.00ff] > + ......... [06.0004] > + ......... [06.0047] > + ......... [06.00ab] > + ......... [06.00ff] > + > + Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later) > + can be bound to the vfio_ap device driver. The reason for this is to > + simplify the implementation by not needlessly complicating the design by > + supporting older devices that will go out of service in the relatively near > + future and for which there are few older systems on which to test. > + > + The administrator, therefore, must take care to secure only AP queues that > + can be bound to the vfio_ap device driver. The device type for a given AP > + queue device can be read from the parent card's sysfs directory. For example, > + to see the hardware type of the queue 05.0004: > + > + cat /sys/bus/ap/devices/card05/hwtype > + > + The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the > + vfio_ap device driver. > + > +3. Create the mediated devices needed to configure the AP matrixes for the > + three guests and to provide an interface to the vfio_ap driver for > + use by the guests: > + > + /sys/devices/vfio_ap/matrix/ > + --- [mdev_supported_types] > + ------ [vfio_ap-passthrough] (passthrough mediated matrix device type) > + --------- create > + --------- [devices] > + > + To create the mediated devices for the three guests: > + > + uuidgen > create > + uuidgen > create > + uuidgen > create > + > + or > + > + echo $uuid1 > create > + echo $uuid2 > create > + echo $uuid3 > create > + > + This will create three mediated devices in the [devices] subdirectory named > + after the UUID written to the create attribute file. We call them $uuid1, > + $uuid2 and $uuid3 and this is the sysfs directory structure after creation: > + > + /sys/devices/vfio_ap/matrix/ > + --- [mdev_supported_types] > + ------ [vfio_ap-passthrough] > + --------- [devices] > + ------------ [$uuid1] > + --------------- assign_adapter > + --------------- assign_control_domain > + --------------- assign_domain > + --------------- matrix > + --------------- unassign_adapter > + --------------- unassign_control_domain > + --------------- unassign_domain > + > + ------------ [$uuid2] > + --------------- assign_adapter > + --------------- assign_control_domain > + --------------- assign_domain > + --------------- matrix > + --------------- unassign_adapter > + ----------------unassign_control_domain > + ----------------unassign_domain > + > + ------------ [$uuid3] > + --------------- assign_adapter > + --------------- assign_control_domain > + --------------- assign_domain > + --------------- matrix > + --------------- unassign_adapter > + ----------------unassign_control_domain > + ----------------unassign_domain > + > +4. The administrator now needs to configure the matrixes for the mediated > + devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3). > + > + This is how the matrix is configured for Guest1: > + > + echo 5 > assign_adapter > + echo 6 > assign_adapter > + echo 4 > assign_domain > + echo 0xab > assign_domain > + > + Control domains can similarly be assigned using the assign_control_domain > + sysfs file. > + > + If a mistake is made configuring an adapter, domain or control domain, > + you can use the unassign_xxx files to unassign the adapter, domain or > + control domain. > + > + To display the matrix configuration for Guest1: > + > + cat matrix > + > + This is how the matrix is configured for Guest2: > + > + echo 5 > assign_adapter > + echo 0x47 > assign_domain > + echo 0xff > assign_domain > + > + This is how the matrix is configured for Guest3: > + > + echo 6 > assign_adapter > + echo 0x47 > assign_domain > + echo 0xff > assign_domain > + I'm curious why this interface didn't adopt the +/- notation invented above for consistency. Too difficult to do rollbacks with a string on entries? Looks pretty reasonable other than the points of confusion noted. Thanks, Alex
On 27.09.2018 00:42, Alex Williamson wrote: > On Tue, 25 Sep 2018 19:16:41 -0400 > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote: > >> From: Tony Krowiak <akrowiak@linux.ibm.com> >> >> This patch provides documentation describing the AP architecture and >> design concepts behind the virtualization of AP devices. It also >> includes an example of how to configure AP devices for exclusive >> use of KVM guests. >> >> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com> >> Reviewed-by: Halil Pasic <pasic@linux.ibm.com> >> --- >> Documentation/s390/vfio-ap.txt | 782 +++++++++++++++++++++++++++++++++ >> MAINTAINERS | 1 + >> 2 files changed, 783 insertions(+) >> create mode 100644 Documentation/s390/vfio-ap.txt > ... >> +Example: >> +======= >> +Let's now provide an example to illustrate how KVM guests may be given >> +access to AP facilities. For this example, we will show how to configure >> +three guests such that executing the lszcrypt command on the guests would >> +look like this: >> + >> +Guest1 >> +------ >> +CARD.DOMAIN TYPE MODE >> +------------------------------ >> +05 CEX5C CCA-Coproc >> +05.0004 CEX5C CCA-Coproc >> +05.00ab CEX5C CCA-Coproc >> +06 CEX5A Accelerator >> +06.0004 CEX5A Accelerator >> +06.00ab CEX5C CCA-Coproc >> + >> +Guest2 >> +------ >> +CARD.DOMAIN TYPE MODE >> +------------------------------ >> +05 CEX5A Accelerator >> +05.0047 CEX5A Accelerator >> +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Seems like an unfinished thought here. > >> + >> +Guest2 >> +------ >> +CARD.DOMAIN TYPE MODE >> +------------------------------ >> +06 CEX5A Accelerator >> +06.0047 CEX5A Accelerator >> +06.00ff CEX5A Accelerator >> + >> +These are the steps: >> + >> +1. Install the vfio_ap module on the linux host. The dependency chain for the >> + vfio_ap module is: >> + * iommu >> + * s390 >> + * zcrypt >> + * vfio >> + * vfio_mdev >> + * vfio_mdev_device >> + * KVM >> + >> + To build the vfio_ap module, the kernel build must be configured with the >> + following Kconfig elements selected: >> + * IOMMU_SUPPORT >> + * S390 >> + * ZCRYPT >> + * S390_AP_IOMMU >> + * VFIO >> + * VFIO_MDEV >> + * VFIO_MDEV_DEVICE >> + * KVM >> + >> + If using make menuconfig select the following to build the vfio_ap module: >> + -> Device Drivers >> + -> IOMMU Hardware Support >> + select S390 AP IOMMU Support >> + -> VFIO Non-Privileged userspace driver framework >> + -> Mediated device driver frramework >> + -> VFIO driver for Mediated devices >> + -> I/O subsystem >> + -> VFIO support for AP devices >> + >> +2. Secure the AP queues to be used by the three guests so that the host can not >> + access them. To secure them, there are two sysfs files that specify >> + bitmasks marking a subset of the APQN range as 'usable by the default AP >> + queue device drivers' or 'not usable by the default device drivers' and thus >> + available for use by the vfio_ap device driver'. The sysfs files containing >> + the sysfs locations of the masks are: >> + >> + /sys/bus/ap/apmask >> + /sys/bus/ap/aqmask >> + >> + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs >> + (APID). Each bit in the mask, from most significant to least significant bit, >> + corresponds to an APID from 0-255. If a bit is set, the APID is marked as >> + usable only by the default AP queue device drivers; otherwise, the APID is >> + usable by the vfio_ap device driver. >> + >> + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes >> + (APQI). Each bit in the mask, from most significant to least significant bit, >> + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as >> + usable only by the default AP queue device drivers; otherwise, the APQI is >> + usable by the vfio_ap device driver. >> + >> + The APQN of each AP queue device assigned to the linux host is checked by the >> + AP bus against the set of APQNs derived from the cross product of APIDs >> + and APQIs marked as usable only by the default AP queue device drivers. If a >> + match is detected, only the default AP queue device drivers will be probed; >> + otherwise, the vfio_ap device driver will be probed. >> + >> + By default, the two masks are set to reserve all APQNs for use by the default >> + AP queue device drivers. There are two ways the default masks can be changed: >> + >> + 1. The masks can be changed at boot time with the kernel command line >> + like this: >> + >> + ap.apmask=0xffff ap.aqmask=0x40 >> + >> + This would give these two pools: >> + >> + default drivers pool: adapter 0-15, domain 1 >> + alternate drivers pool: adapter 16-255, domains 2-255 > What happened to domain 0? I'm also a little confused by the bit > ordering. If 0x40 is bit 1 and 0xffff is bits 0-15, then the least > significant bit is furthest left? Did I miss documentation of that? > >> + >> + 2. The sysfs mask files can also be edited by echoing a string into the >> + respective file in one of two formats: >> + >> + * An absolute hex string starting with 0x - like "0x12345678" - sets >> + the mask. If the given string is shorter than the mask, it is padded >> + with 0s on the right. If the string is longer than the mask, the >> + operation is terminated with an error (EINVAL). > And this does say zero padding on the right, but then in the next > bullet our hex digits use normal least significant bit right notation, > ie. 0x41 is 65, not 82, correct? > >> + >> + * A plus ('+') or minus ('-') followed by a numerical value. Valid >> + examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only >> + the corresponding bit in the mask is switched on ('+') or off ('-'). The >> + values may also be specified in a comma-separated list to switch more >> + than one bit on or off. >> + >> + To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, >> + 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding >> + APQNs must be removed from the masks as follows: >> + >> + echo -5,-6 > /sys/bus/ap/apmask >> + >> + echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask > Other than the bit ordering confusion, I like this +/- scheme. > >> + >> + This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, >> + 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The >> + sysfs directory for the vfio_ap device driver will now contain symbolic links >> + to the AP queue devices bound to it: >> + >> + /sys/bus/ap >> + ... [drivers] >> + ...... [vfio_ap] >> + ......... [05.0004] >> + ......... [05.0047] >> + ......... [05.00ab] >> + ......... [05.00ff] >> + ......... [06.0004] >> + ......... [06.0047] >> + ......... [06.00ab] >> + ......... [06.00ff] >> + >> + Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later) >> + can be bound to the vfio_ap device driver. The reason for this is to >> + simplify the implementation by not needlessly complicating the design by >> + supporting older devices that will go out of service in the relatively near >> + future and for which there are few older systems on which to test. >> + >> + The administrator, therefore, must take care to secure only AP queues that >> + can be bound to the vfio_ap device driver. The device type for a given AP >> + queue device can be read from the parent card's sysfs directory. For example, >> + to see the hardware type of the queue 05.0004: >> + >> + cat /sys/bus/ap/devices/card05/hwtype >> + >> + The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the >> + vfio_ap device driver. >> + >> +3. Create the mediated devices needed to configure the AP matrixes for the >> + three guests and to provide an interface to the vfio_ap driver for >> + use by the guests: >> + >> + /sys/devices/vfio_ap/matrix/ >> + --- [mdev_supported_types] >> + ------ [vfio_ap-passthrough] (passthrough mediated matrix device type) >> + --------- create >> + --------- [devices] >> + >> + To create the mediated devices for the three guests: >> + >> + uuidgen > create >> + uuidgen > create >> + uuidgen > create >> + >> + or >> + >> + echo $uuid1 > create >> + echo $uuid2 > create >> + echo $uuid3 > create >> + >> + This will create three mediated devices in the [devices] subdirectory named >> + after the UUID written to the create attribute file. We call them $uuid1, >> + $uuid2 and $uuid3 and this is the sysfs directory structure after creation: >> + >> + /sys/devices/vfio_ap/matrix/ >> + --- [mdev_supported_types] >> + ------ [vfio_ap-passthrough] >> + --------- [devices] >> + ------------ [$uuid1] >> + --------------- assign_adapter >> + --------------- assign_control_domain >> + --------------- assign_domain >> + --------------- matrix >> + --------------- unassign_adapter >> + --------------- unassign_control_domain >> + --------------- unassign_domain >> + >> + ------------ [$uuid2] >> + --------------- assign_adapter >> + --------------- assign_control_domain >> + --------------- assign_domain >> + --------------- matrix >> + --------------- unassign_adapter >> + ----------------unassign_control_domain >> + ----------------unassign_domain >> + >> + ------------ [$uuid3] >> + --------------- assign_adapter >> + --------------- assign_control_domain >> + --------------- assign_domain >> + --------------- matrix >> + --------------- unassign_adapter >> + ----------------unassign_control_domain >> + ----------------unassign_domain >> + >> +4. The administrator now needs to configure the matrixes for the mediated >> + devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3). >> + >> + This is how the matrix is configured for Guest1: >> + >> + echo 5 > assign_adapter >> + echo 6 > assign_adapter >> + echo 4 > assign_domain >> + echo 0xab > assign_domain >> + >> + Control domains can similarly be assigned using the assign_control_domain >> + sysfs file. >> + >> + If a mistake is made configuring an adapter, domain or control domain, >> + you can use the unassign_xxx files to unassign the adapter, domain or >> + control domain. >> + >> + To display the matrix configuration for Guest1: >> + >> + cat matrix >> + >> + This is how the matrix is configured for Guest2: >> + >> + echo 5 > assign_adapter >> + echo 0x47 > assign_domain >> + echo 0xff > assign_domain >> + >> + This is how the matrix is configured for Guest3: >> + >> + echo 6 > assign_adapter >> + echo 0x47 > assign_domain >> + echo 0xff > assign_domain >> + > I'm curious why this interface didn't adopt the +/- notation invented > above for consistency. Too difficult to do rollbacks with a string on > entries? > > Looks pretty reasonable other than the points of confusion noted. > Thanks, > > Alex > Hello Alex the AP bus apmask and aqmask interface is not part of this patch series. It's a general concept not only related to KVM and vfio but may be used (and will) by other kernel and userspace features. The bit ordering follows other implementations in the s390 (big endian) realm where bit counting starts with 0 on the very left side and increases to the right. All the AP bus stuff obeys to this (maybe historical) scheme. Please have a look on the patches which introduced this mask API: - s390/zcrypt: AP bus support for alternate driver(s) : https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7e0bdbe5c21cb8316a694e46ad5aad339f6894a6 - s390/zcrypt: hex string mask improvements for apmask and aqmask: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3d8f60d38e249f989a7fca9c2370c31c3d5487e1 The patch headers should describe the behavior and the syntax of the recognized string patterns in more detail. regards Harald Freudenberger (maintainer of the s390 AP bus and zcrypt device driver code)
On 09/27/2018 12:42 AM, Alex Williamson wrote: > On Tue, 25 Sep 2018 19:16:41 -0400 > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote: > >> From: Tony Krowiak <akrowiak@linux.ibm.com> [..] >> + >> +2. Secure the AP queues to be used by the three guests so that the host can not >> + access them. To secure them, there are two sysfs files that specify >> + bitmasks marking a subset of the APQN range as 'usable by the default AP >> + queue device drivers' or 'not usable by the default device drivers' and thus >> + available for use by the vfio_ap device driver'. The sysfs files containing >> + the sysfs locations of the masks are: >> + >> + /sys/bus/ap/apmask >> + /sys/bus/ap/aqmask >> + >> + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs >> + (APID). Each bit in the mask, from most significant to least significant bit, >> + corresponds to an APID from 0-255. If a bit is set, the APID is marked as >> + usable only by the default AP queue device drivers; otherwise, the APID is >> + usable by the vfio_ap device driver. >> + >> + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes >> + (APQI). Each bit in the mask, from most significant to least significant bit, >> + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as >> + usable only by the default AP queue device drivers; otherwise, the APQI is >> + usable by the vfio_ap device driver. >> + >> + The APQN of each AP queue device assigned to the linux host is checked by the >> + AP bus against the set of APQNs derived from the cross product of APIDs >> + and APQIs marked as usable only by the default AP queue device drivers. If a >> + match is detected, only the default AP queue device drivers will be probed; >> + otherwise, the vfio_ap device driver will be probed. >> + >> + By default, the two masks are set to reserve all APQNs for use by the default >> + AP queue device drivers. There are two ways the default masks can be changed: >> + >> + 1. The masks can be changed at boot time with the kernel command line >> + like this: >> + >> + ap.apmask=0xffff ap.aqmask=0x40 >> + >> + This would give these two pools: >> + >> + default drivers pool: adapter 0-15, domain 1 >> + alternate drivers pool: adapter 16-255, domains 2-255 > > What happened to domain 0? Right, domain 0 is also 'alternate'. So it should have been alternate drivers pool: adapter 16-255, domains 0,2-255 > I'm also a little confused by the bit > ordering. If 0x40 is bit 1 and 0xffff is bits 0-15, then the least > significant bit is furthest left? Did I miss documentation of that? > Harald already tried to explain this, let me give it a try too. Yes it is a bit confusing. I would try to describe it like this: the big endian mask, which is of fixed length of 256 bytes is specified byte-wise using hexadecimal notation. If only a prefix of the whole mask is specified, the not explicitly specified bytes are specified are as if they were specified as zero. I didn't quite get this thing with 'the least significant bit is furthest left'. I think it is to the right if we assume we are reading left-to-right. It is big endian, so we consider the most significant bit of a byte to be the first bit, and the byte with the lowest address to be the first byte of the mask (that holds the first 8 bits of the mask). >> + >> + 2. The sysfs mask files can also be edited by echoing a string into the >> + respective file in one of two formats: >> + >> + * An absolute hex string starting with 0x - like "0x12345678" - sets >> + the mask. If the given string is shorter than the mask, it is padded >> + with 0s on the right. If the string is longer than the mask, the >> + operation is terminated with an error (EINVAL). > > And this does say zero padding on the right, but then in the next > bullet our hex digits use normal least significant bit right notation, > ie. 0x41 is 65, not 82, correct? The zero padding on the right is about the non specified bytes of the mask. While this bullet is about specifying a whole mask, the next butlet is about changing a mask by setting the value of bits at a certain position. So in the context of the next bullet point, the hex string here specifies an integer value -- plainly a number written in hexadecimal notation (pure math with no significant bits whatsoever) - in the range 0-256: the index of the bit to be set ('+') or cleared ('-'). I hope that makes some sense. As I said it's indeed a bit confusing. >> + >> + * A plus ('+') or minus ('-') followed by a numerical value. Valid >> + examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only >> + the corresponding bit in the mask is switched on ('+') or off ('-'). The >> + values may also be specified in a comma-separated list to switch more >> + than one bit on or off. >> + >> + To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, >> + 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding >> + APQNs must be removed from the masks as follows: >> + >> + echo -5,-6 > /sys/bus/ap/apmask >> + >> + echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask > > Other than the bit ordering confusion, I like this +/- scheme. > >> + >> + This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, >> + 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The >> + sysfs directory for the vfio_ap device driver will now contain symbolic links >> + to the AP queue devices bound to it: >> + >> + /sys/bus/ap >> + ... [drivers] >> + ...... [vfio_ap] >> + ......... [05.0004] >> + ......... [05.0047] >> + ......... [05.00ab] >> + ......... [05.00ff] >> + ......... [06.0004] >> + ......... [06.0047] >> + ......... [06.00ab] >> + ......... [06.00ff] >> + >> + Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later) >> + can be bound to the vfio_ap device driver. The reason for this is to >> + simplify the implementation by not needlessly complicating the design by >> + supporting older devices that will go out of service in the relatively near >> + future and for which there are few older systems on which to test. >> + >> + The administrator, therefore, must take care to secure only AP queues that >> + can be bound to the vfio_ap device driver. The device type for a given AP >> + queue device can be read from the parent card's sysfs directory. For example, >> + to see the hardware type of the queue 05.0004: >> + >> + cat /sys/bus/ap/devices/card05/hwtype >> + >> + The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the >> + vfio_ap device driver. >> + >> +3. Create the mediated devices needed to configure the AP matrixes for the >> + three guests and to provide an interface to the vfio_ap driver for >> + use by the guests: >> + >> + /sys/devices/vfio_ap/matrix/ >> + --- [mdev_supported_types] >> + ------ [vfio_ap-passthrough] (passthrough mediated matrix device type) >> + --------- create >> + --------- [devices] >> + >> + To create the mediated devices for the three guests: >> + >> + uuidgen > create >> + uuidgen > create >> + uuidgen > create >> + >> + or >> + >> + echo $uuid1 > create >> + echo $uuid2 > create >> + echo $uuid3 > create >> + >> + This will create three mediated devices in the [devices] subdirectory named >> + after the UUID written to the create attribute file. We call them $uuid1, >> + $uuid2 and $uuid3 and this is the sysfs directory structure after creation: >> + >> + /sys/devices/vfio_ap/matrix/ >> + --- [mdev_supported_types] >> + ------ [vfio_ap-passthrough] >> + --------- [devices] >> + ------------ [$uuid1] >> + --------------- assign_adapter >> + --------------- assign_control_domain >> + --------------- assign_domain >> + --------------- matrix >> + --------------- unassign_adapter >> + --------------- unassign_control_domain >> + --------------- unassign_domain >> + >> + ------------ [$uuid2] >> + --------------- assign_adapter >> + --------------- assign_control_domain >> + --------------- assign_domain >> + --------------- matrix >> + --------------- unassign_adapter >> + ----------------unassign_control_domain >> + ----------------unassign_domain >> + >> + ------------ [$uuid3] >> + --------------- assign_adapter >> + --------------- assign_control_domain >> + --------------- assign_domain >> + --------------- matrix >> + --------------- unassign_adapter >> + ----------------unassign_control_domain >> + ----------------unassign_domain >> + >> +4. The administrator now needs to configure the matrixes for the mediated >> + devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3). >> + >> + This is how the matrix is configured for Guest1: >> + >> + echo 5 > assign_adapter >> + echo 6 > assign_adapter >> + echo 4 > assign_domain >> + echo 0xab > assign_domain >> + >> + Control domains can similarly be assigned using the assign_control_domain >> + sysfs file. >> + >> + If a mistake is made configuring an adapter, domain or control domain, >> + you can use the unassign_xxx files to unassign the adapter, domain or >> + control domain. >> + >> + To display the matrix configuration for Guest1: >> + >> + cat matrix >> + >> + This is how the matrix is configured for Guest2: >> + >> + echo 5 > assign_adapter >> + echo 0x47 > assign_domain >> + echo 0xff > assign_domain >> + >> + This is how the matrix is configured for Guest3: >> + >> + echo 6 > assign_adapter >> + echo 0x47 > assign_domain >> + echo 0xff > assign_domain >> + > > I'm curious why this interface didn't adopt the +/- notation invented > above for consistency. Too difficult to do rollbacks with a string on > entries? > I remember that we did discuss that possibility around v9, but I can't tell why did we decide to not implement it. Maybe Tony has an answer. Anyway, if we were to do that, we would use different attribute names (e.g. just domain_mask, or something similar instead of (assign|unassign)_xxx). So I think such an interface can still be added on top of the existing one. Having that said having multiple interfaces for the very same thing is usually not so nice IMHO. Regards, Halil
On Thu, 27 Sep 2018 13:29:43 +0200 Halil Pasic <pasic@linux.ibm.com> wrote: > On 09/27/2018 12:42 AM, Alex Williamson wrote: > > On Tue, 25 Sep 2018 19:16:41 -0400 > > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote: > >> + This is how the matrix is configured for Guest2: > >> + > >> + echo 5 > assign_adapter > >> + echo 0x47 > assign_domain > >> + echo 0xff > assign_domain > >> + > >> + This is how the matrix is configured for Guest3: > >> + > >> + echo 6 > assign_adapter > >> + echo 0x47 > assign_domain > >> + echo 0xff > assign_domain > >> + > > > > I'm curious why this interface didn't adopt the +/- notation invented > > above for consistency. Too difficult to do rollbacks with a string on > > entries? > > > > I remember that we did discuss that possibility around v9, but I can't > tell why did we decide to not implement it. Maybe Tony has an answer. IIRC, that was a discussion on the base ap driver interfaces rather than vfio-ap. > > Anyway, if we were to do that, we would use different attribute names > (e.g. just domain_mask, or something similar instead of > (assign|unassign)_xxx). So I think such an interface can still be added > on top of the existing one. Having that said having multiple interfaces > for the very same thing is usually not so nice IMHO. Nod to all of your points. As we do the configuration while the guest is not running anyway, the different interfaces probably do not make that much difference in practice. It should be fine to stick to the current interface for now and only add a new one if we really think it is significantly better.
On 09/27/2018 01:51 PM, Cornelia Huck wrote: > On Thu, 27 Sep 2018 13:29:43 +0200 > Halil Pasic <pasic@linux.ibm.com> wrote: > >> On 09/27/2018 12:42 AM, Alex Williamson wrote: >>> On Tue, 25 Sep 2018 19:16:41 -0400 >>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote: >>>> + This is how the matrix is configured for Guest2: >>>> + >>>> + echo 5 > assign_adapter >>>> + echo 0x47 > assign_domain >>>> + echo 0xff > assign_domain >>>> + >>>> + This is how the matrix is configured for Guest3: >>>> + >>>> + echo 6 > assign_adapter >>>> + echo 0x47 > assign_domain >>>> + echo 0xff > assign_domain >>>> + >>> >>> I'm curious why this interface didn't adopt the +/- notation invented >>> above for consistency. Too difficult to do rollbacks with a string on >>> entries? >>> >> >> I remember that we did discuss that possibility around v9, but I can't >> tell why did we decide to not implement it. Maybe Tony has an answer. > > IIRC, that was a discussion on the base ap driver interfaces rather > than vfio-ap. > >> >> Anyway, if we were to do that, we would use different attribute names >> (e.g. just domain_mask, or something similar instead of >> (assign|unassign)_xxx). So I think such an interface can still be added >> on top of the existing one. Having that said having multiple interfaces >> for the very same thing is usually not so nice IMHO. > > Nod to all of your points. > > As we do the configuration while the guest is not running anyway, the > different interfaces probably do not make that much difference in > practice. It should be fine to stick to the current interface for now > and only add a new one if we really think it is significantly better. Tony, can you maybe provide a quick on-top patch that clarifies Alex comments regarding the documentation? (State that is is big endian, fixup the small things etc). I can then either fold it in or provide it as an on top patch depending on how much has changed.
On 09/27/2018 07:59 AM, Christian Borntraeger wrote: > > > On 09/27/2018 01:51 PM, Cornelia Huck wrote: >> On Thu, 27 Sep 2018 13:29:43 +0200 >> Halil Pasic <pasic@linux.ibm.com> wrote: >> >>> On 09/27/2018 12:42 AM, Alex Williamson wrote: >>>> On Tue, 25 Sep 2018 19:16:41 -0400 >>>> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote: >>>>> + This is how the matrix is configured for Guest2: >>>>> + >>>>> + echo 5 > assign_adapter >>>>> + echo 0x47 > assign_domain >>>>> + echo 0xff > assign_domain >>>>> + >>>>> + This is how the matrix is configured for Guest3: >>>>> + >>>>> + echo 6 > assign_adapter >>>>> + echo 0x47 > assign_domain >>>>> + echo 0xff > assign_domain >>>>> + >>>> >>>> I'm curious why this interface didn't adopt the +/- notation invented >>>> above for consistency. Too difficult to do rollbacks with a string on >>>> entries? >>>> >>> >>> I remember that we did discuss that possibility around v9, but I can't >>> tell why did we decide to not implement it. Maybe Tony has an answer. >> >> IIRC, that was a discussion on the base ap driver interfaces rather >> than vfio-ap. >> >>> >>> Anyway, if we were to do that, we would use different attribute names >>> (e.g. just domain_mask, or something similar instead of >>> (assign|unassign)_xxx). So I think such an interface can still be added >>> on top of the existing one. Having that said having multiple interfaces >>> for the very same thing is usually not so nice IMHO. >> >> Nod to all of your points. >> >> As we do the configuration while the guest is not running anyway, the >> different interfaces probably do not make that much difference in >> practice. It should be fine to stick to the current interface for now >> and only add a new one if we really think it is significantly better. > > Tony, can you maybe provide a quick on-top patch that clarifies Alex > comments regarding the documentation? (State that is is big endian, > fixup the small things etc). > I can then either fold it in or provide it as an on top patch depending > on how much has changed. Will do. >
On 09/27/2018 07:29 AM, Halil Pasic wrote: > > > On 09/27/2018 12:42 AM, Alex Williamson wrote: >> On Tue, 25 Sep 2018 19:16:41 -0400 >> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote: >> >>> From: Tony Krowiak <akrowiak@linux.ibm.com> > [..] >>> + >>> +2. Secure the AP queues to be used by the three guests so that the host can not >>> + access them. To secure them, there are two sysfs files that specify >>> + bitmasks marking a subset of the APQN range as 'usable by the default AP >>> + queue device drivers' or 'not usable by the default device drivers' and thus >>> + available for use by the vfio_ap device driver'. The sysfs files containing >>> + the sysfs locations of the masks are: >>> + >>> + /sys/bus/ap/apmask >>> + /sys/bus/ap/aqmask >>> + >>> + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs >>> + (APID). Each bit in the mask, from most significant to least significant bit, >>> + corresponds to an APID from 0-255. If a bit is set, the APID is marked as >>> + usable only by the default AP queue device drivers; otherwise, the APID is >>> + usable by the vfio_ap device driver. >>> + >>> + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes >>> + (APQI). Each bit in the mask, from most significant to least significant bit, >>> + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as >>> + usable only by the default AP queue device drivers; otherwise, the APQI is >>> + usable by the vfio_ap device driver. >>> + >>> + The APQN of each AP queue device assigned to the linux host is checked by the >>> + AP bus against the set of APQNs derived from the cross product of APIDs >>> + and APQIs marked as usable only by the default AP queue device drivers. If a >>> + match is detected, only the default AP queue device drivers will be probed; >>> + otherwise, the vfio_ap device driver will be probed. >>> + >>> + By default, the two masks are set to reserve all APQNs for use by the default >>> + AP queue device drivers. There are two ways the default masks can be changed: >>> + >>> + 1. The masks can be changed at boot time with the kernel command line >>> + like this: >>> + >>> + ap.apmask=0xffff ap.aqmask=0x40 >>> + >>> + This would give these two pools: >>> + >>> + default drivers pool: adapter 0-15, domain 1 >>> + alternate drivers pool: adapter 16-255, domains 2-255 >> >> What happened to domain 0? > > Right, domain 0 is also 'alternate'. So it should have been > alternate drivers pool: adapter 16-255, domains 0,2-255 My mistake. > >> I'm also a little confused by the bit >> ordering. If 0x40 is bit 1 and 0xffff is bits 0-15, then the least >> significant bit is furthest left? Did I miss documentation of that? >> > > Harald already tried to explain this, let me give it a try too. > > Yes it is a bit confusing. I would try to describe it like this: the big endian mask, > which is of fixed length of 256 bytes is specified byte-wise using hexadecimal > notation. If only a prefix of the whole mask is specified, the not explicitly > specified bytes are specified are as if they were specified as zero. > > I didn't quite get this thing with 'the least significant bit is furthest left'. > I think it is to the right if we assume we are reading left-to-right. It is big > endian, so we consider the most significant bit of a byte to be the first bit, > and the byte with the lowest address to be the first byte of the mask (that holds the > first 8 bits of the mask). I'm not quite sure to what you are referring, but the description of the apqmask and aqmask above states: "Each bit in the mask, from most significant to least significant bit, corresponds to an APID from 0-255." It should probably mention that the ordering is big endian, or say something like "each bit in the mask, from left to right ...". > >>> + >>> + 2. The sysfs mask files can also be edited by echoing a string into the >>> + respective file in one of two formats: >>> + >>> + * An absolute hex string starting with 0x - like "0x12345678" - sets >>> + the mask. If the given string is shorter than the mask, it is padded >>> + with 0s on the right. If the string is longer than the mask, the >>> + operation is terminated with an error (EINVAL). >> >> And this does say zero padding on the right, but then in the next >> bullet our hex digits use normal least significant bit right notation, >> ie. 0x41 is 65, not 82, correct? > > The zero padding on the right is about the non specified bytes of the mask. > > While this bullet is about specifying a whole mask, the next butlet is about > changing a mask by setting the value of bits at a certain position. So in the > context of the next bullet point, the hex string here specifies an integer > value -- plainly a number written in hexadecimal notation (pure math with no > significant bits whatsoever) - in the range 0-256: the index of the bit to be > set ('+') or cleared ('-'). > > > I hope that makes some sense. As I said it's indeed a bit confusing. > >>> + >>> + * A plus ('+') or minus ('-') followed by a numerical value. Valid >>> + examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only >>> + the corresponding bit in the mask is switched on ('+') or off ('-'). The >>> + values may also be specified in a comma-separated list to switch more >>> + than one bit on or off. >>> + >>> + To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, >>> + 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding >>> + APQNs must be removed from the masks as follows: >>> + >>> + echo -5,-6 > /sys/bus/ap/apmask >>> + >>> + echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask >> >> Other than the bit ordering confusion, I like this +/- scheme. >> >>> + >>> + This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, >>> + 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The >>> + sysfs directory for the vfio_ap device driver will now contain symbolic links >>> + to the AP queue devices bound to it: >>> + >>> + /sys/bus/ap >>> + ... [drivers] >>> + ...... [vfio_ap] >>> + ......... [05.0004] >>> + ......... [05.0047] >>> + ......... [05.00ab] >>> + ......... [05.00ff] >>> + ......... [06.0004] >>> + ......... [06.0047] >>> + ......... [06.00ab] >>> + ......... [06.00ff] >>> + >>> + Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later) >>> + can be bound to the vfio_ap device driver. The reason for this is to >>> + simplify the implementation by not needlessly complicating the design by >>> + supporting older devices that will go out of service in the relatively near >>> + future and for which there are few older systems on which to test. >>> + >>> + The administrator, therefore, must take care to secure only AP queues that >>> + can be bound to the vfio_ap device driver. The device type for a given AP >>> + queue device can be read from the parent card's sysfs directory. For example, >>> + to see the hardware type of the queue 05.0004: >>> + >>> + cat /sys/bus/ap/devices/card05/hwtype >>> + >>> + The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the >>> + vfio_ap device driver. >>> + >>> +3. Create the mediated devices needed to configure the AP matrixes for the >>> + three guests and to provide an interface to the vfio_ap driver for >>> + use by the guests: >>> + >>> + /sys/devices/vfio_ap/matrix/ >>> + --- [mdev_supported_types] >>> + ------ [vfio_ap-passthrough] (passthrough mediated matrix device type) >>> + --------- create >>> + --------- [devices] >>> + >>> + To create the mediated devices for the three guests: >>> + >>> + uuidgen > create >>> + uuidgen > create >>> + uuidgen > create >>> + >>> + or >>> + >>> + echo $uuid1 > create >>> + echo $uuid2 > create >>> + echo $uuid3 > create >>> + >>> + This will create three mediated devices in the [devices] subdirectory named >>> + after the UUID written to the create attribute file. We call them $uuid1, >>> + $uuid2 and $uuid3 and this is the sysfs directory structure after creation: >>> + >>> + /sys/devices/vfio_ap/matrix/ >>> + --- [mdev_supported_types] >>> + ------ [vfio_ap-passthrough] >>> + --------- [devices] >>> + ------------ [$uuid1] >>> + --------------- assign_adapter >>> + --------------- assign_control_domain >>> + --------------- assign_domain >>> + --------------- matrix >>> + --------------- unassign_adapter >>> + --------------- unassign_control_domain >>> + --------------- unassign_domain >>> + >>> + ------------ [$uuid2] >>> + --------------- assign_adapter >>> + --------------- assign_control_domain >>> + --------------- assign_domain >>> + --------------- matrix >>> + --------------- unassign_adapter >>> + ----------------unassign_control_domain >>> + ----------------unassign_domain >>> + >>> + ------------ [$uuid3] >>> + --------------- assign_adapter >>> + --------------- assign_control_domain >>> + --------------- assign_domain >>> + --------------- matrix >>> + --------------- unassign_adapter >>> + ----------------unassign_control_domain >>> + ----------------unassign_domain >>> + >>> +4. The administrator now needs to configure the matrixes for the mediated >>> + devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3). >>> + >>> + This is how the matrix is configured for Guest1: >>> + >>> + echo 5 > assign_adapter >>> + echo 6 > assign_adapter >>> + echo 4 > assign_domain >>> + echo 0xab > assign_domain >>> + >>> + Control domains can similarly be assigned using the assign_control_domain >>> + sysfs file. >>> + >>> + If a mistake is made configuring an adapter, domain or control domain, >>> + you can use the unassign_xxx files to unassign the adapter, domain or >>> + control domain. >>> + >>> + To display the matrix configuration for Guest1: >>> + >>> + cat matrix >>> + >>> + This is how the matrix is configured for Guest2: >>> + >>> + echo 5 > assign_adapter >>> + echo 0x47 > assign_domain >>> + echo 0xff > assign_domain >>> + >>> + This is how the matrix is configured for Guest3: >>> + >>> + echo 6 > assign_adapter >>> + echo 0x47 > assign_domain >>> + echo 0xff > assign_domain >>> + >> >> I'm curious why this interface didn't adopt the +/- notation invented >> above for consistency. Too difficult to do rollbacks with a string on >> entries? >> > > I remember that we did discuss that possibility around v9, but I can't > tell why did we decide to not implement it. Maybe Tony has an answer. The syntax for assigning adapters, domains and control domains predates Harald's patches implementing the apmask and aqmask by well over six months (since v1). As Harald stated, his patches do not belong to this series, and are not directly related to mediated device configuration. We may have discussed implementing similar interfaces for mdev configuration, but since the mdev assignment concept had already undergone multiple reviews since v1 without objection, it was decided that introducing this at such a late stage would be a potential impediment to acceptance. > > Anyway, if we were to do that, we would use different attribute names > (e.g. just domain_mask, or something similar instead of > (assign|unassign)_xxx). So I think such an interface can still be added > on top of the existing one. Having that said having multiple interfaces > for the very same thing is usually not so nice IMHO. In my opinion, it ought to be one or the other. > > Regards, > Halil >
On 09/26/2018 06:42 PM, Alex Williamson wrote: > On Tue, 25 Sep 2018 19:16:41 -0400 > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote: > >> From: Tony Krowiak <akrowiak@linux.ibm.com> >> >> This patch provides documentation describing the AP architecture and >> design concepts behind the virtualization of AP devices. It also >> includes an example of how to configure AP devices for exclusive >> use of KVM guests. >> >> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com> >> Reviewed-by: Halil Pasic <pasic@linux.ibm.com> >> --- >> Documentation/s390/vfio-ap.txt | 782 +++++++++++++++++++++++++++++++++ >> MAINTAINERS | 1 + >> 2 files changed, 783 insertions(+) >> create mode 100644 Documentation/s390/vfio-ap.txt > ... >> +Example: >> +======= >> +Let's now provide an example to illustrate how KVM guests may be given >> +access to AP facilities. For this example, we will show how to configure >> +three guests such that executing the lszcrypt command on the guests would >> +look like this: >> + >> +Guest1 >> +------ >> +CARD.DOMAIN TYPE MODE >> +------------------------------ >> +05 CEX5C CCA-Coproc >> +05.0004 CEX5C CCA-Coproc >> +05.00ab CEX5C CCA-Coproc >> +06 CEX5A Accelerator >> +06.0004 CEX5A Accelerator >> +06.00ab CEX5C CCA-Coproc >> + >> +Guest2 >> +------ >> +CARD.DOMAIN TYPE MODE >> +------------------------------ >> +05 CEX5A Accelerator >> +05.0047 CEX5A Accelerator >> +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Seems like an unfinished thought here. I have no idea how that got in there, there must be a gremlin in my laptop. It does not belong. > >> + >> +Guest2 >> +------ >> +CARD.DOMAIN TYPE MODE >> +------------------------------ >> +06 CEX5A Accelerator >> +06.0047 CEX5A Accelerator >> +06.00ff CEX5A Accelerator >> + >> +These are the steps: >> + >> +1. Install the vfio_ap module on the linux host. The dependency chain for the >> + vfio_ap module is: >> + * iommu >> + * s390 >> + * zcrypt >> + * vfio >> + * vfio_mdev >> + * vfio_mdev_device >> + * KVM >> + >> + To build the vfio_ap module, the kernel build must be configured with the >> + following Kconfig elements selected: >> + * IOMMU_SUPPORT >> + * S390 >> + * ZCRYPT >> + * S390_AP_IOMMU >> + * VFIO >> + * VFIO_MDEV >> + * VFIO_MDEV_DEVICE >> + * KVM >> + >> + If using make menuconfig select the following to build the vfio_ap module: >> + -> Device Drivers >> + -> IOMMU Hardware Support >> + select S390 AP IOMMU Support >> + -> VFIO Non-Privileged userspace driver framework >> + -> Mediated device driver frramework >> + -> VFIO driver for Mediated devices >> + -> I/O subsystem >> + -> VFIO support for AP devices >> + >> +2. Secure the AP queues to be used by the three guests so that the host can not >> + access them. To secure them, there are two sysfs files that specify >> + bitmasks marking a subset of the APQN range as 'usable by the default AP >> + queue device drivers' or 'not usable by the default device drivers' and thus >> + available for use by the vfio_ap device driver'. The sysfs files containing >> + the sysfs locations of the masks are: >> + >> + /sys/bus/ap/apmask >> + /sys/bus/ap/aqmask >> + >> + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs >> + (APID). Each bit in the mask, from most significant to least significant bit, >> + corresponds to an APID from 0-255. If a bit is set, the APID is marked as >> + usable only by the default AP queue device drivers; otherwise, the APID is >> + usable by the vfio_ap device driver. >> + >> + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes >> + (APQI). Each bit in the mask, from most significant to least significant bit, >> + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as >> + usable only by the default AP queue device drivers; otherwise, the APQI is >> + usable by the vfio_ap device driver. >> + >> + The APQN of each AP queue device assigned to the linux host is checked by the >> + AP bus against the set of APQNs derived from the cross product of APIDs >> + and APQIs marked as usable only by the default AP queue device drivers. If a >> + match is detected, only the default AP queue device drivers will be probed; >> + otherwise, the vfio_ap device driver will be probed. >> + >> + By default, the two masks are set to reserve all APQNs for use by the default >> + AP queue device drivers. There are two ways the default masks can be changed: >> + >> + 1. The masks can be changed at boot time with the kernel command line >> + like this: >> + >> + ap.apmask=0xffff ap.aqmask=0x40 >> + >> + This would give these two pools: >> + >> + default drivers pool: adapter 0-15, domain 1 >> + alternate drivers pool: adapter 16-255, domains 2-255 > > What happened to domain 0? As Halil stated, it should be: alternate drivers pool: adapter 16-255, domains 0, 2-255 I'm also a little confused by the bit > ordering. If 0x40 is bit 1 and 0xffff is bits 0-15, then the least > significant bit is furthest left? Did I miss documentation of that? Up above, the apmask and aqmask are described. Both descriptions state, "each bit in the mask, from most significant to least significant bit, corresponds to an ... from 0-255". It could probably be clarified by stating: Each bit in the mask, from most significant to least significant bit in big endian order, corresponds to an ... from 0-255 or Each bit in the mask, from left to right, corresponds to an ... from 0-255 > >> + >> + 2. The sysfs mask files can also be edited by echoing a string into the >> + respective file in one of two formats: >> + >> + * An absolute hex string starting with 0x - like "0x12345678" - sets >> + the mask. If the given string is shorter than the mask, it is padded >> + with 0s on the right. If the string is longer than the mask, the >> + operation is terminated with an error (EINVAL). > > And this does say zero padding on the right, but then in the next > bullet our hex digits use normal least significant bit right notation, > ie. 0x41 is 65, not 82, correct? Again, I refer you back to the descriptions above for aqmask and apmask. When configuring a mask, the value specified encompasses the entire 256-bit mask, so specifying 0x41 (65) is exactly like specifying the following: 0x4100000000000000000000000000000000000000000000000000000000000000 The most significant byte is 0x41 (big endian), so if the mask bits read from left to right (i.e., most significant to least significant bit), the mask above identifies devices 1 and 7. > >> + >> + * A plus ('+') or minus ('-') followed by a numerical value. Valid >> + examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only >> + the corresponding bit in the mask is switched on ('+') or off ('-'). The >> + values may also be specified in a comma-separated list to switch more >> + than one bit on or off. >> + >> + To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, >> + 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding >> + APQNs must be removed from the masks as follows: >> + >> + echo -5,-6 > /sys/bus/ap/apmask >> + >> + echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask > > Other than the bit ordering confusion, I like this +/- scheme. > >> + >> + This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, >> + 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The >> + sysfs directory for the vfio_ap device driver will now contain symbolic links >> + to the AP queue devices bound to it: >> + >> + /sys/bus/ap >> + ... [drivers] >> + ...... [vfio_ap] >> + ......... [05.0004] >> + ......... [05.0047] >> + ......... [05.00ab] >> + ......... [05.00ff] >> + ......... [06.0004] >> + ......... [06.0047] >> + ......... [06.00ab] >> + ......... [06.00ff] >> + >> + Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later) >> + can be bound to the vfio_ap device driver. The reason for this is to >> + simplify the implementation by not needlessly complicating the design by >> + supporting older devices that will go out of service in the relatively near >> + future and for which there are few older systems on which to test. >> + >> + The administrator, therefore, must take care to secure only AP queues that >> + can be bound to the vfio_ap device driver. The device type for a given AP >> + queue device can be read from the parent card's sysfs directory. For example, >> + to see the hardware type of the queue 05.0004: >> + >> + cat /sys/bus/ap/devices/card05/hwtype >> + >> + The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the >> + vfio_ap device driver. >> + >> +3. Create the mediated devices needed to configure the AP matrixes for the >> + three guests and to provide an interface to the vfio_ap driver for >> + use by the guests: >> + >> + /sys/devices/vfio_ap/matrix/ >> + --- [mdev_supported_types] >> + ------ [vfio_ap-passthrough] (passthrough mediated matrix device type) >> + --------- create >> + --------- [devices] >> + >> + To create the mediated devices for the three guests: >> + >> + uuidgen > create >> + uuidgen > create >> + uuidgen > create >> + >> + or >> + >> + echo $uuid1 > create >> + echo $uuid2 > create >> + echo $uuid3 > create >> + >> + This will create three mediated devices in the [devices] subdirectory named >> + after the UUID written to the create attribute file. We call them $uuid1, >> + $uuid2 and $uuid3 and this is the sysfs directory structure after creation: >> + >> + /sys/devices/vfio_ap/matrix/ >> + --- [mdev_supported_types] >> + ------ [vfio_ap-passthrough] >> + --------- [devices] >> + ------------ [$uuid1] >> + --------------- assign_adapter >> + --------------- assign_control_domain >> + --------------- assign_domain >> + --------------- matrix >> + --------------- unassign_adapter >> + --------------- unassign_control_domain >> + --------------- unassign_domain >> + >> + ------------ [$uuid2] >> + --------------- assign_adapter >> + --------------- assign_control_domain >> + --------------- assign_domain >> + --------------- matrix >> + --------------- unassign_adapter >> + ----------------unassign_control_domain >> + ----------------unassign_domain >> + >> + ------------ [$uuid3] >> + --------------- assign_adapter >> + --------------- assign_control_domain >> + --------------- assign_domain >> + --------------- matrix >> + --------------- unassign_adapter >> + ----------------unassign_control_domain >> + ----------------unassign_domain >> + >> +4. The administrator now needs to configure the matrixes for the mediated >> + devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3). >> + >> + This is how the matrix is configured for Guest1: >> + >> + echo 5 > assign_adapter >> + echo 6 > assign_adapter >> + echo 4 > assign_domain >> + echo 0xab > assign_domain >> + >> + Control domains can similarly be assigned using the assign_control_domain >> + sysfs file. >> + >> + If a mistake is made configuring an adapter, domain or control domain, >> + you can use the unassign_xxx files to unassign the adapter, domain or >> + control domain. >> + >> + To display the matrix configuration for Guest1: >> + >> + cat matrix >> + >> + This is how the matrix is configured for Guest2: >> + >> + echo 5 > assign_adapter >> + echo 0x47 > assign_domain >> + echo 0xff > assign_domain >> + >> + This is how the matrix is configured for Guest3: >> + >> + echo 6 > assign_adapter >> + echo 0x47 > assign_domain >> + echo 0xff > assign_domain >> + > > I'm curious why this interface didn't adopt the +/- notation invented > above for consistency. Too difficult to do rollbacks with a string on > entries? These interfaces predate the AP bus interfaces by over six months, and the AP bus interfaces are not part of this patch series. > > Looks pretty reasonable other than the points of confusion noted. > Thanks, > > Alex >
On 09/26/2018 06:42 PM, Alex Williamson wrote: > On Tue, 25 Sep 2018 19:16:41 -0400 > Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote: > >> From: Tony Krowiak <akrowiak@linux.ibm.com> >> >> This patch provides documentation describing the AP architecture and >> design concepts behind the virtualization of AP devices. It also >> includes an example of how to configure AP devices for exclusive >> use of KVM guests. >> >> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com> >> Reviewed-by: Halil Pasic <pasic@linux.ibm.com> >> --- >> Documentation/s390/vfio-ap.txt | 782 +++++++++++++++++++++++++++++++++ >> MAINTAINERS | 1 + >> 2 files changed, 783 insertions(+) >> create mode 100644 Documentation/s390/vfio-ap.txt > ... >> +Example: >> +======= >> +Let's now provide an example to illustrate how KVM guests may be given >> +access to AP facilities. For this example, we will show how to configure >> +three guests such that executing the lszcrypt command on the guests would >> +look like this: >> + >> +Guest1 >> +------ >> +CARD.DOMAIN TYPE MODE >> +------------------------------ >> +05 CEX5C CCA-Coproc >> +05.0004 CEX5C CCA-Coproc >> +05.00ab CEX5C CCA-Coproc >> +06 CEX5A Accelerator >> +06.0004 CEX5A Accelerator >> +06.00ab CEX5C CCA-Coproc >> + >> +Guest2 >> +------ >> +CARD.DOMAIN TYPE MODE >> +------------------------------ >> +05 CEX5A Accelerator >> +05.0047 CEX5A Accelerator >> +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > Seems like an unfinished thought here. > >> + >> +Guest2 >> +------ >> +CARD.DOMAIN TYPE MODE >> +------------------------------ >> +06 CEX5A Accelerator >> +06.0047 CEX5A Accelerator >> +06.00ff CEX5A Accelerator >> + >> +These are the steps: >> + >> +1. Install the vfio_ap module on the linux host. The dependency chain for the >> + vfio_ap module is: >> + * iommu >> + * s390 >> + * zcrypt >> + * vfio >> + * vfio_mdev >> + * vfio_mdev_device >> + * KVM >> + >> + To build the vfio_ap module, the kernel build must be configured with the >> + following Kconfig elements selected: >> + * IOMMU_SUPPORT >> + * S390 >> + * ZCRYPT >> + * S390_AP_IOMMU >> + * VFIO >> + * VFIO_MDEV >> + * VFIO_MDEV_DEVICE >> + * KVM >> + >> + If using make menuconfig select the following to build the vfio_ap module: >> + -> Device Drivers >> + -> IOMMU Hardware Support >> + select S390 AP IOMMU Support >> + -> VFIO Non-Privileged userspace driver framework >> + -> Mediated device driver frramework >> + -> VFIO driver for Mediated devices >> + -> I/O subsystem >> + -> VFIO support for AP devices >> + >> +2. Secure the AP queues to be used by the three guests so that the host can not >> + access them. To secure them, there are two sysfs files that specify >> + bitmasks marking a subset of the APQN range as 'usable by the default AP >> + queue device drivers' or 'not usable by the default device drivers' and thus >> + available for use by the vfio_ap device driver'. The sysfs files containing >> + the sysfs locations of the masks are: >> + >> + /sys/bus/ap/apmask >> + /sys/bus/ap/aqmask >> + >> + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs >> + (APID). Each bit in the mask, from most significant to least significant bit, >> + corresponds to an APID from 0-255. If a bit is set, the APID is marked as >> + usable only by the default AP queue device drivers; otherwise, the APID is >> + usable by the vfio_ap device driver. >> + >> + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes >> + (APQI). Each bit in the mask, from most significant to least significant bit, >> + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as >> + usable only by the default AP queue device drivers; otherwise, the APQI is >> + usable by the vfio_ap device driver. >> + >> + The APQN of each AP queue device assigned to the linux host is checked by the >> + AP bus against the set of APQNs derived from the cross product of APIDs >> + and APQIs marked as usable only by the default AP queue device drivers. If a >> + match is detected, only the default AP queue device drivers will be probed; >> + otherwise, the vfio_ap device driver will be probed. >> + >> + By default, the two masks are set to reserve all APQNs for use by the default >> + AP queue device drivers. There are two ways the default masks can be changed: >> + >> + 1. The masks can be changed at boot time with the kernel command line >> + like this: >> + >> + ap.apmask=0xffff ap.aqmask=0x40 >> + >> + This would give these two pools: >> + >> + default drivers pool: adapter 0-15, domain 1 >> + alternate drivers pool: adapter 16-255, domains 2-255 > > What happened to domain 0? I'm also a little confused by the bit > ordering. If 0x40 is bit 1 and 0xffff is bits 0-15, then the least > significant bit is furthest left? Did I miss documentation of that? > >> + >> + 2. The sysfs mask files can also be edited by echoing a string into the >> + respective file in one of two formats: >> + >> + * An absolute hex string starting with 0x - like "0x12345678" - sets >> + the mask. If the given string is shorter than the mask, it is padded >> + with 0s on the right. If the string is longer than the mask, the >> + operation is terminated with an error (EINVAL). > > And this does say zero padding on the right, but then in the next > bullet our hex digits use normal least significant bit right notation, > ie. 0x41 is 65, not 82, correct? > >> + >> + * A plus ('+') or minus ('-') followed by a numerical value. Valid >> + examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only >> + the corresponding bit in the mask is switched on ('+') or off ('-'). The >> + values may also be specified in a comma-separated list to switch more >> + than one bit on or off. >> + >> + To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, >> + 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding >> + APQNs must be removed from the masks as follows: >> + >> + echo -5,-6 > /sys/bus/ap/apmask >> + >> + echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask > > Other than the bit ordering confusion, I like this +/- scheme. The following fixup attempts to clarify the bit ordering confusion, hopefully this is acceptable. -----------------------------------8<----------------------------------- From: Tony Krowiak <akrowiak@linux.ibm.com> Date: Thu, 27 Sep 2018 14:51:12 -0400 Subject: [FIXUP v10] fixup! s390: doc: detailed specifications for AP virtualization Better explains mask bit ordering. Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com> --- Documentation/s390/vfio-ap.txt | 127 +++++++++++++++++++++++---------- 1 file changed, 91 insertions(+), 36 deletions(-) diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt index bec67eb7141c..599eb0f75c07 100644 --- a/Documentation/s390/vfio-ap.txt +++ b/Documentation/s390/vfio-ap.txt @@ -123,21 +123,24 @@ to identify the adapters, usage domains and control domains assigned to the KVM guest: * The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned - to the KVM guest. Each bit in the mask, from most significant to least - significant bit, corresponds to an APID from 0-255. If a bit is set, the - corresponding adapter is valid for use by the KVM guest. + to the KVM guest. Each bit in the mask, from left to right (i.e. from most + significant to least significant bit in big endian order), corresponds to + an APID from 0-255. If a bit is set, the corresponding adapter is valid for + use by the KVM guest. * The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains - assigned to the KVM guest. Each bit in the mask, from most significant to - least significant bit, corresponds to an AP queue index (APQI) from 0-255. If - a bit is set, the corresponding queue is valid for use by the KVM guest. + assigned to the KVM guest. Each bit in the mask, from left to right (i.e. from + most significant to least significant bit in big endian order), corresponds to + an AP queue index (APQI) from 0-255. If a bit is set, the corresponding queue + is valid for use by the KVM guest. * The AP Domain Mask field is a bit mask that identifies the AP control domains assigned to the KVM guest. The ADM bit mask controls which domains can be changed by an AP command-request message sent to a usage domain from the - guest. Each bit in the mask, from least significant to most significant bit, - corresponds to a domain from 0-255. If a bit is set, the corresponding domain - can be modified by an AP command-request message sent to a usage domain. + guest. Each bit in the mask, from left to right (i.e. from most significant to + least significant bit in big endian order), corresponds to a domain from + 0-255. If a bit is set, the corresponding domain can be modified by an AP + command-request message sent to a usage domain. If you recall from the description of an AP Queue, AP instructions include an APQN to identify the AP queue to which an AP command-request message is to be @@ -503,23 +506,34 @@ These are the steps: access them. To secure them, there are two sysfs files that specify bitmasks marking a subset of the APQN range as 'usable by the default AP queue device drivers' or 'not usable by the default device drivers' and thus - available for use by the vfio_ap device driver'. The sysfs files containing - the sysfs locations of the masks are: + available for use by the vfio_ap device driver'. The location of the sysfs + files containing the masks are: /sys/bus/ap/apmask /sys/bus/ap/aqmask The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs - (APID). Each bit in the mask, from most significant to least significant bit, - corresponds to an APID from 0-255. If a bit is set, the APID is marked as - usable only by the default AP queue device drivers; otherwise, the APID is - usable by the vfio_ap device driver. + (APID). Each bit in the mask, from left to right (i.e., from most significant + to least significant bit in big endian order), corresponds to an APID from + 0-255. If a bit is set, the APID is marked as usable only by the default AP + queue device drivers; otherwise, the APID is usable by the vfio_ap + device driver. The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes - (APQI). Each bit in the mask, from most significant to least significant bit, - corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as - usable only by the default AP queue device drivers; otherwise, the APQI is - usable by the vfio_ap device driver. + (APQI). Each bit in the mask, from left to right (i.e., from most significant + to least significant bit in big endian order), corresponds to an APQI from + 0-255. If a bit is set, the APQI is marked as usable only by the default AP + queue device drivers; otherwise, the APQI is usable by the vfio_ap device + driver. + + Take, for example, the following mask: + + 0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff + + It indicates: + + 1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6 + belong to the vfio_ap device driver's pool. The APQN of each AP queue device assigned to the linux host is checked by the AP bus against the set of APQNs derived from the cross product of APIDs @@ -530,38 +544,79 @@ These are the steps: By default, the two masks are set to reserve all APQNs for use by the default AP queue device drivers. There are two ways the default masks can be changed: - 1. The masks can be changed at boot time with the kernel command line - like this: + 1. The sysfs mask files can be edited by echoing a string into the + respective sysfs mask file in one of two formats: + + * An absolute hex string starting with 0x - like "0x12345678" - sets + the mask. If the given string is shorter than the mask, it is padded + with 0s on the right; for example, specifying a mask value of 0x41 is + the same as specifying: + + 0x4100000000000000000000000000000000000000000000000000000000000000 + + Keep in mind that the mask reads from left to right (i.e., most + significant to least significant bit in big endian order), so the mask + above identifies device numbers 1 and 7 (01000001). + + If the string is longer than the mask, the operation is terminated with + an error (EINVAL). + + * Individual bits in the mask can be switched on and off by specifying + each bit number to be switched in a comma separated list. Each bit + number string must be prepended with a ('+') or minus ('-') to indicate + the corresponding bit is to be switched on ('+') or off ('-'). Some + valid values are: + + "+0" switches bit 0 on + "-13" switches bit 13 off + "+0x41" switches bit 65 on + "-0xff" switches bit 255 off + + The following example: + +0,-6,+0x47,-0xf0 + + Switches bits 0 and 71 (0x47) on + Switches bits 6 and 240 (0xf0) off + + Note that the bits not specified in the list remain as they were before + the operation. + + 2. The masks can also be changed at boot time via parameters on the kernel + command line like this: ap.apmask=0xffff ap.aqmask=0x40 - This would give these two pools: + This would create the following masks: - default drivers pool: adapter 0-15, domain 1 - alternate drivers pool: adapter 16-255, domains 2-255 + apmask: + 0xffff000000000000000000000000000000000000000000000000000000000000 - 2. The sysfs mask files can also be edited by echoing a string into the - respective file in one of two formats: + aqmask: + 0x4000000000000000000000000000000000000000000000000000000000000000 - * An absolute hex string starting with 0x - like "0x12345678" - sets - the mask. If the given string is shorter than the mask, it is padded - with 0s on the right. If the string is longer than the mask, the - operation is terminated with an error (EINVAL). + Resulting in these two pools: - * A plus ('+') or minus ('-') followed by a numerical value. Valid - examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only - the corresponding bit in the mask is switched on ('+') or off ('-'). The - values may also be specified in a comma-separated list to switch more - than one bit on or off. + default drivers pool: adapter 0-15, domain 1 + alternate drivers pool: adapter 16-255, domains 0, 2-255 + Securing the APQNs for our example: + ---------------------------------- To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding - APQNs must be removed from the masks as follows: + APQNs can either be removed from the default masks: echo -5,-6 > /sys/bus/ap/apmask echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask + Or the masks can be set as follows: + + echo 0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \ + > apmask + + echo 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \ + > aqmask + This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The sysfs directory for the vfio_ap device driver will now contain symbolic links
On 09/27/2018 09:19 PM, Tony Krowiak wrote: > On 09/26/2018 06:42 PM, Alex Williamson wrote: >> On Tue, 25 Sep 2018 19:16:41 -0400 >> Tony Krowiak <akrowiak@linux.vnet.ibm.com> wrote: >> >>> From: Tony Krowiak <akrowiak@linux.ibm.com> >>> >>> This patch provides documentation describing the AP architecture and >>> design concepts behind the virtualization of AP devices. It also >>> includes an example of how to configure AP devices for exclusive >>> use of KVM guests. >>> >>> Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com> >>> Reviewed-by: Halil Pasic <pasic@linux.ibm.com> >>> --- >>> Documentation/s390/vfio-ap.txt | 782 +++++++++++++++++++++++++++++++++ >>> MAINTAINERS | 1 + >>> 2 files changed, 783 insertions(+) >>> create mode 100644 Documentation/s390/vfio-ap.txt >> ... >>> +Example: >>> +======= >>> +Let's now provide an example to illustrate how KVM guests may be given >>> +access to AP facilities. For this example, we will show how to configure >>> +three guests such that executing the lszcrypt command on the guests would >>> +look like this: >>> + >>> +Guest1 >>> +------ >>> +CARD.DOMAIN TYPE MODE >>> +------------------------------ >>> +05 CEX5C CCA-Coproc >>> +05.0004 CEX5C CCA-Coproc >>> +05.00ab CEX5C CCA-Coproc >>> +06 CEX5A Accelerator >>> +06.0004 CEX5A Accelerator >>> +06.00ab CEX5C CCA-Coproc >>> + >>> +Guest2 >>> +------ >>> +CARD.DOMAIN TYPE MODE >>> +------------------------------ >>> +05 CEX5A Accelerator >>> +05.0047 CEX5A Accelerator >>> +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> Seems like an unfinished thought here. Can you submit another fixup that takes care of this comment?
On 09/27/2018 09:19 PM, Tony Krowiak wrote: > The following fixup attempts to clarify the bit ordering confusion, > hopefully this is acceptable. > looks good to me, I will fold in. > -----------------------------------8<----------------------------------- > > From: Tony Krowiak <akrowiak@linux.ibm.com> > Date: Thu, 27 Sep 2018 14:51:12 -0400 > Subject: [FIXUP v10] fixup! s390: doc: detailed specifications for AP > virtualization > > Better explains mask bit ordering. > > Signed-off-by: Tony Krowiak <akrowiak@linux.ibm.com> > --- > Documentation/s390/vfio-ap.txt | 127 +++++++++++++++++++++++---------- > 1 file changed, 91 insertions(+), 36 deletions(-) > > diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt > index bec67eb7141c..599eb0f75c07 100644 > --- a/Documentation/s390/vfio-ap.txt > +++ b/Documentation/s390/vfio-ap.txt > @@ -123,21 +123,24 @@ to identify the adapters, usage domains and control domains assigned to the KVM > guest: > > * The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned > - to the KVM guest. Each bit in the mask, from most significant to least > - significant bit, corresponds to an APID from 0-255. If a bit is set, the > - corresponding adapter is valid for use by the KVM guest. > + to the KVM guest. Each bit in the mask, from left to right (i.e. from most > + significant to least significant bit in big endian order), corresponds to > + an APID from 0-255. If a bit is set, the corresponding adapter is valid for > + use by the KVM guest. > > * The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains > - assigned to the KVM guest. Each bit in the mask, from most significant to > - least significant bit, corresponds to an AP queue index (APQI) from 0-255. If > - a bit is set, the corresponding queue is valid for use by the KVM guest. > + assigned to the KVM guest. Each bit in the mask, from left to right (i.e. from > + most significant to least significant bit in big endian order), corresponds to > + an AP queue index (APQI) from 0-255. If a bit is set, the corresponding queue > + is valid for use by the KVM guest. > > * The AP Domain Mask field is a bit mask that identifies the AP control domains > assigned to the KVM guest. The ADM bit mask controls which domains can be > changed by an AP command-request message sent to a usage domain from the > - guest. Each bit in the mask, from least significant to most significant bit, > - corresponds to a domain from 0-255. If a bit is set, the corresponding domain > - can be modified by an AP command-request message sent to a usage domain. > + guest. Each bit in the mask, from left to right (i.e. from most significant to > + least significant bit in big endian order), corresponds to a domain from > + 0-255. If a bit is set, the corresponding domain can be modified by an AP > + command-request message sent to a usage domain. > > If you recall from the description of an AP Queue, AP instructions include > an APQN to identify the AP queue to which an AP command-request message is to be > @@ -503,23 +506,34 @@ These are the steps: > access them. To secure them, there are two sysfs files that specify > bitmasks marking a subset of the APQN range as 'usable by the default AP > queue device drivers' or 'not usable by the default device drivers' and thus > - available for use by the vfio_ap device driver'. The sysfs files containing > - the sysfs locations of the masks are: > + available for use by the vfio_ap device driver'. The location of the sysfs > + files containing the masks are: > > /sys/bus/ap/apmask > /sys/bus/ap/aqmask > > The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs > - (APID). Each bit in the mask, from most significant to least significant bit, > - corresponds to an APID from 0-255. If a bit is set, the APID is marked as > - usable only by the default AP queue device drivers; otherwise, the APID is > - usable by the vfio_ap device driver. > + (APID). Each bit in the mask, from left to right (i.e., from most significant > + to least significant bit in big endian order), corresponds to an APID from > + 0-255. If a bit is set, the APID is marked as usable only by the default AP > + queue device drivers; otherwise, the APID is usable by the vfio_ap > + device driver. > > The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes > - (APQI). Each bit in the mask, from most significant to least significant bit, > - corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as > - usable only by the default AP queue device drivers; otherwise, the APQI is > - usable by the vfio_ap device driver. > + (APQI). Each bit in the mask, from left to right (i.e., from most significant > + to least significant bit in big endian order), corresponds to an APQI from > + 0-255. If a bit is set, the APQI is marked as usable only by the default AP > + queue device drivers; otherwise, the APQI is usable by the vfio_ap device > + driver. > + > + Take, for example, the following mask: > + > + 0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff > + > + It indicates: > + > + 1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6 > + belong to the vfio_ap device driver's pool. > > The APQN of each AP queue device assigned to the linux host is checked by the > AP bus against the set of APQNs derived from the cross product of APIDs > @@ -530,38 +544,79 @@ These are the steps: > By default, the two masks are set to reserve all APQNs for use by the default > AP queue device drivers. There are two ways the default masks can be changed: > > - 1. The masks can be changed at boot time with the kernel command line > - like this: > + 1. The sysfs mask files can be edited by echoing a string into the > + respective sysfs mask file in one of two formats: > + > + * An absolute hex string starting with 0x - like "0x12345678" - sets > + the mask. If the given string is shorter than the mask, it is padded > + with 0s on the right; for example, specifying a mask value of 0x41 is > + the same as specifying: > + > + 0x4100000000000000000000000000000000000000000000000000000000000000 > + > + Keep in mind that the mask reads from left to right (i.e., most > + significant to least significant bit in big endian order), so the mask > + above identifies device numbers 1 and 7 (01000001). > + > + If the string is longer than the mask, the operation is terminated with > + an error (EINVAL). > + > + * Individual bits in the mask can be switched on and off by specifying > + each bit number to be switched in a comma separated list. Each bit > + number string must be prepended with a ('+') or minus ('-') to indicate > + the corresponding bit is to be switched on ('+') or off ('-'). Some > + valid values are: > + > + "+0" switches bit 0 on > + "-13" switches bit 13 off > + "+0x41" switches bit 65 on > + "-0xff" switches bit 255 off > + > + The following example: > + +0,-6,+0x47,-0xf0 > + > + Switches bits 0 and 71 (0x47) on > + Switches bits 6 and 240 (0xf0) off > + > + Note that the bits not specified in the list remain as they were before > + the operation. > + > + 2. The masks can also be changed at boot time via parameters on the kernel > + command line like this: > > ap.apmask=0xffff ap.aqmask=0x40 > > - This would give these two pools: > + This would create the following masks: > > - default drivers pool: adapter 0-15, domain 1 > - alternate drivers pool: adapter 16-255, domains 2-255 > + apmask: > + 0xffff000000000000000000000000000000000000000000000000000000000000 > > - 2. The sysfs mask files can also be edited by echoing a string into the > - respective file in one of two formats: > + aqmask: > + 0x4000000000000000000000000000000000000000000000000000000000000000 > > - * An absolute hex string starting with 0x - like "0x12345678" - sets > - the mask. If the given string is shorter than the mask, it is padded > - with 0s on the right. If the string is longer than the mask, the > - operation is terminated with an error (EINVAL). > + Resulting in these two pools: > > - * A plus ('+') or minus ('-') followed by a numerical value. Valid > - examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only > - the corresponding bit in the mask is switched on ('+') or off ('-'). The > - values may also be specified in a comma-separated list to switch more > - than one bit on or off. > + default drivers pool: adapter 0-15, domain 1 > + alternate drivers pool: adapter 16-255, domains 0, 2-255 > > + Securing the APQNs for our example: > + ---------------------------------- > To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, > 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding > - APQNs must be removed from the masks as follows: > + APQNs can either be removed from the default masks: > > echo -5,-6 > /sys/bus/ap/apmask > > echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask > > + Or the masks can be set as follows: > + > + echo 0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \ > + > apmask > + > + echo 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \ > + > aqmask > + > This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, > 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The > sysfs directory for the vfio_ap device driver will now contain symbolic links
diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt new file mode 100644 index 000000000000..bec67eb7141c --- /dev/null +++ b/Documentation/s390/vfio-ap.txt @@ -0,0 +1,782 @@ +Introduction: +============ +The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised +of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards. +The AP devices provide cryptographic functions to all CPUs assigned to a +linux system running in an IBM Z system LPAR. + +The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap +is to make AP cards available to KVM guests using the VFIO mediated device +framework. This implementation relies considerably on the s390 virtualization +facilities which do most of the hard work of providing direct access to AP +devices. + +AP Architectural Overview: +========================= +To facilitate the comprehension of the design, let's start with some +definitions: + +* AP adapter + + An AP adapter is an IBM Z adapter card that can perform cryptographic + functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters + assigned to the LPAR in which a linux host is running will be available to + the linux host. Each adapter is identified by a number from 0 to 255; however, + the maximum adapter number is determined by machine model and/or adapter type. + When installed, an AP adapter is accessed by AP instructions executed by any + CPU. + + The AP adapter cards are assigned to a given LPAR via the system's Activation + Profile which can be edited via the HMC. When the linux host system is IPL'd + in the LPAR, the AP bus detects the AP adapter cards assigned to the LPAR and + creates a sysfs device for each assigned adapter. For example, if AP adapters + 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will create the following + sysfs device entries: + + /sys/devices/ap/card04 + /sys/devices/ap/card0a + + Symbolic links to these devices will also be created in the AP bus devices + sub-directory: + + /sys/bus/ap/devices/[card04] + /sys/bus/ap/devices/[card04] + +* AP domain + + An adapter is partitioned into domains. An adapter can hold up to 256 domains + depending upon the adapter type and hardware configuration. A domain is + identified by a number from 0 to 255; however, the maximum domain number is + determined by machine model and/or adapter type.. A domain can be thought of + as a set of hardware registers and memory used for processing AP commands. A + domain can be configured with a secure private key used for clear key + encryption. A domain is classified in one of two ways depending upon how it + may be accessed: + + * Usage domains are domains that are targeted by an AP instruction to + process an AP command. + + * Control domains are domains that are changed by an AP command sent to a + usage domain; for example, to set the secure private key for the control + domain. + + The AP usage and control domains are assigned to a given LPAR via the system's + Activation Profile which can be edited via the HMC. When a linux host system + is IPL'd in the LPAR, the AP bus module detects the AP usage and control + domains assigned to the LPAR. The domain number of each usage domain and + adapter number of each AP adapter are combined to create AP queue devices + (see AP Queue section below). The domain number of each control domain will be + represented in a bitmask and stored in a sysfs file + /sys/bus/ap/ap_control_domain_mask. The bits in the mask, from most to least + significant bit, correspond to domains 0-255. + +* AP Queue + + An AP queue is the means by which an AP command is sent to a usage domain + inside a specific adapter. An AP queue is identified by a tuple + comprised of an AP adapter ID (APID) and an AP queue index (APQI). The + APQI corresponds to a given usage domain number within the adapter. This tuple + forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP + instructions include a field containing the APQN to identify the AP queue to + which the AP command is to be sent for processing. + + The AP bus will create a sysfs device for each APQN that can be derived from + the cross product of the AP adapter and usage domain numbers detected when the + AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage + domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the + following sysfs entries: + + /sys/devices/ap/card04/04.0006 + /sys/devices/ap/card04/04.0047 + /sys/devices/ap/card0a/0a.0006 + /sys/devices/ap/card0a/0a.0047 + + The following symbolic links to these devices will be created in the AP bus + devices subdirectory: + + /sys/bus/ap/devices/[04.0006] + /sys/bus/ap/devices/[04.0047] + /sys/bus/ap/devices/[0a.0006] + /sys/bus/ap/devices/[0a.0047] + +* AP Instructions: + + There are three AP instructions: + + * NQAP: to enqueue an AP command-request message to a queue + * DQAP: to dequeue an AP command-reply message from a queue + * PQAP: to administer the queues + + AP instructions identify the domain that is targeted to process the AP + command; this must be one of the usage domains. An AP command may modify a + domain that is not one of the usage domains, but the modified domain + must be one of the control domains. + +AP and SIE: +========== +Let's now take a look at how AP instructions executed on a guest are interpreted +by the hardware. + +A satellite control block called the Crypto Control Block (CRYCB) is attached to +our main hardware virtualization control block. The CRYCB contains three fields +to identify the adapters, usage domains and control domains assigned to the KVM +guest: + +* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned + to the KVM guest. Each bit in the mask, from most significant to least + significant bit, corresponds to an APID from 0-255. If a bit is set, the + corresponding adapter is valid for use by the KVM guest. + +* The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains + assigned to the KVM guest. Each bit in the mask, from most significant to + least significant bit, corresponds to an AP queue index (APQI) from 0-255. If + a bit is set, the corresponding queue is valid for use by the KVM guest. + +* The AP Domain Mask field is a bit mask that identifies the AP control domains + assigned to the KVM guest. The ADM bit mask controls which domains can be + changed by an AP command-request message sent to a usage domain from the + guest. Each bit in the mask, from least significant to most significant bit, + corresponds to a domain from 0-255. If a bit is set, the corresponding domain + can be modified by an AP command-request message sent to a usage domain. + +If you recall from the description of an AP Queue, AP instructions include +an APQN to identify the AP queue to which an AP command-request message is to be +sent (NQAP and PQAP instructions), or from which a command-reply message is to +be received (DQAP instruction). The validity of an APQN is defined by the matrix +calculated from the APM and AQM; it is the cross product of all assigned adapter +numbers (APM) with all assigned queue indexes (AQM). For example, if adapters 1 +and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs (1,5), (1,6), +(2,5) and (2,6) will be valid for the guest. + +The APQNs can provide secure key functionality - i.e., a private key is stored +on the adapter card for each of its domains - so each APQN must be assigned to +at most one guest or to the linux host. + + Example 1: Valid configuration: + ------------------------------ + Guest1: adapters 1,2 domains 5,6 + Guest2: adapter 1,2 domain 7 + + This is valid because both guests have a unique set of APQNs: + Guest1 has APQNs (1,5), (1,6), (2,5), (2,6); + Guest2 has APQNs (1,7), (2,7) + + Example 2: Valid configuration: + ------------------------------ + Guest1: adapters 1,2 domains 5,6 + Guest2: adapters 3,4 domains 5,6 + + This is also valid because both guests have a unique set of APQNs: + Guest1 has APQNs (1,5), (1,6), (2,5), (2,6); + Guest2 has APQNs (3,5), (3,6), (4,5), (4,6) + + Example 3: Invalid configuration: + -------------------------------- + Guest1: adapters 1,2 domains 5,6 + Guest2: adapter 1 domains 6,7 + + This is an invalid configuration because both guests have access to + APQN (1,6). + +The Design: +=========== +The design introduces three new objects: + +1. AP matrix device +2. VFIO AP device driver (vfio_ap.ko) +3. VFIO AP mediated matrix pass-through device + +The VFIO AP device driver +------------------------- +The VFIO AP (vfio_ap) device driver serves the following purposes: + +1. Provides the interfaces to secure APQNs for exclusive use of KVM guests. + +2. Sets up the VFIO mediated device interfaces to manage a mediated matrix + device and creates the sysfs interfaces for assigning adapters, usage + domains, and control domains comprising the matrix for a KVM guest. + +3. Configures the APM, AQM and ADM in the CRYCB referenced by a KVM guest's + SIE state description to grant the guest access to a matrix of AP devices + +Reserve APQNs for exclusive use of KVM guests +--------------------------------------------- +The following block diagram illustrates the mechanism by which APQNs are +reserved: + + +------------------+ + 7 remove | | + +--------------------> cex4queue driver | + | | | + | +------------------+ + | + | + | +------------------+ +-----------------+ + | 5 register driver | | 3 create | | + | +----------------> Device core +----------> matrix device | + | | | | | | + | | +--------^---------+ +-----------------+ + | | | + | | +-------------------+ + | | +-----------------------------------+ | + | | | 4 register AP driver | | 2 register device + | | | | | ++--------+---+-v---+ +--------+-------+-+ +| | | | +| ap_bus +--------------------- > vfio_ap driver | +| | 8 probe | | ++--------^---------+ +--^--^------------+ +6 edit | | | + apmask | +-----------------------------+ | 9 mdev create + aqmask | | 1 modprobe | ++--------+-----+---+ +----------------+-+ +------------------+ +| | | |8 create | mediated | +| admin | | VFIO device core |---------> matrix | +| + | | | device | ++------+-+---------+ +--------^---------+ +--------^---------+ + | | | | + | | 9 create vfio_ap-passthrough | | + | +------------------------------+ | + +-------------------------------------------------------------+ + 10 assign adapter/domain/control domain + +The process for reserving an AP queue for use by a KVM guest is: + +1. The administrator loads the vfio_ap device driver +2. The vfio-ap driver during its initialization will register a single 'matrix' + device with the device core. This will serve as the parent device for + all mediated matrix devices used to configure an AP matrix for a guest. +3. The /sys/devices/vfio_ap/matrix device is created by the device core +4 The vfio_ap device driver will register with the AP bus for AP queue devices + of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap + driver's probe and remove callback interfaces. Devices older than CEX4 queues + are not supported to simplify the implementation by not needlessly + complicating the design by supporting older devices that will go out of + service in the relatively near future, and for which there are few older + systems around on which to test. +5. The AP bus registers the vfio_ap device driver with the device core +6. The administrator edits the AP adapter and queue masks to reserve AP queues + for use by the vfio_ap device driver. +7. The AP bus removes the AP queues reserved for the vfio_ap driver from the + default zcrypt cex4queue driver. +8. The AP bus probes the vfio_ap device driver to bind the queues reserved for + it. +9. The administrator creates a passthrough type mediated matrix device to be + used by a guest +10 The administrator assigns the adapters, usage domains and control domains + to be exclusively used by a guest. + +Set up the VFIO mediated device interfaces +------------------------------------------ +The VFIO AP device driver utilizes the common interface of the VFIO mediated +device core driver to: +* Register an AP mediated bus driver to add a mediated matrix device to and + remove it from a VFIO group. +* Create and destroy a mediated matrix device +* Add a mediated matrix device to and remove it from the AP mediated bus driver +* Add a mediated matrix device to and remove it from an IOMMU group + +The following high-level block diagram shows the main components and interfaces +of the VFIO AP mediated matrix device driver: + + +-------------+ + | | + | +---------+ | mdev_register_driver() +--------------+ + | | Mdev | +<-----------------------+ | + | | bus | | | vfio_mdev.ko | + | | driver | +----------------------->+ |<-> VFIO user + | +---------+ | probe()/remove() +--------------+ APIs + | | + | MDEV CORE | + | MODULE | + | mdev.ko | + | +---------+ | mdev_register_device() +--------------+ + | |Physical | +<-----------------------+ | + | | device | | | vfio_ap.ko |<-> matrix + | |interface| +----------------------->+ | device + | +---------+ | callback +--------------+ + +-------------+ + +During initialization of the vfio_ap module, the matrix device is registered +with an 'mdev_parent_ops' structure that provides the sysfs attribute +structures, mdev functions and callback interfaces for managing the mediated +matrix device. + +* sysfs attribute structures: + * supported_type_groups + The VFIO mediated device framework supports creation of user-defined + mediated device types. These mediated device types are specified + via the 'supported_type_groups' structure when a device is registered + with the mediated device framework. The registration process creates the + sysfs structures for each mediated device type specified in the + 'mdev_supported_types' sub-directory of the device being registered. Along + with the device type, the sysfs attributes of the mediated device type are + provided. + + The VFIO AP device driver will register one mediated device type for + passthrough devices: + /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough + Only the read-only attributes required by the VFIO mdev framework will + be provided: + ... name + ... device_api + ... available_instances + ... device_api + Where: + * name: specifies the name of the mediated device type + * device_api: the mediated device type's API + * available_instances: the number of mediated matrix passthrough devices + that can be created + * device_api: specifies the VFIO API + * mdev_attr_groups + This attribute group identifies the user-defined sysfs attributes of the + mediated device. When a device is registered with the VFIO mediated device + framework, the sysfs attribute files identified in the 'mdev_attr_groups' + structure will be created in the mediated matrix device's directory. The + sysfs attributes for a mediated matrix device are: + * assign_adapter: + * unassign_adapter: + Write-only attributes for assigning/unassigning an AP adapter to/from the + mediated matrix device. To assign/unassign an adapter, the APID of the + adapter is echoed to the respective attribute file. + * assign_domain: + * unassign_domain: + Write-only attributes for assigning/unassigning an AP usage domain to/from + the mediated matrix device. To assign/unassign a domain, the domain + number of the the usage domain is echoed to the respective attribute + file. + * matrix: + A read-only file for displaying the APQNs derived from the cross product + of the adapter and domain numbers assigned to the mediated matrix device. + * assign_control_domain: + * unassign_control_domain: + Write-only attributes for assigning/unassigning an AP control domain + to/from the mediated matrix device. To assign/unassign a control domain, + the ID of the domain to be assigned/unassigned is echoed to the respective + attribute file. + * control_domains: + A read-only file for displaying the control domain numbers assigned to the + mediated matrix device. + +* functions: + * create: + allocates the ap_matrix_mdev structure used by the vfio_ap driver to: + * Store the reference to the KVM structure for the guest using the mdev + * Store the AP matrix configuration for the adapters, domains, and control + domains assigned via the corresponding sysfs attributes files + * remove: + deallocates the mediated matrix device's ap_matrix_mdev structure. This will + be allowed only if a running guest is not using the mdev. + +* callback interfaces + * open: + The vfio_ap driver uses this callback to register a + VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the mdev matrix + device. The open is invoked when QEMU connects the VFIO iommu group + for the mdev matrix device to the MDEV bus. Access to the KVM structure used + to configure the KVM guest is provided via this callback. The KVM structure, + is used to configure the guest's access to the AP matrix defined via the + mediated matrix device's sysfs attribute files. + * release: + unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the + mdev matrix device and deconfigures the guest's AP matrix. + +Configure the APM, AQM and ADM in the CRYCB: +------------------------------------------- +Configuring the AP matrix for a KVM guest will be performed when the +VFIO_GROUP_NOTIFY_SET_KVM notifier callback is invoked. The notifier +function is called when QEMU connects to KVM. The guest's AP matrix is +configured via it's CRYCB by: +* Setting the bits in the APM corresponding to the APIDs assigned to the + mediated matrix device via its 'assign_adapter' interface. +* Setting the bits in the AQM corresponding to the domains assigned to the + mediated matrix device via its 'assign_domain' interface. +* Setting the bits in the ADM corresponding to the domain dIDs assigned to the + mediated matrix device via its 'assign_control_domains' interface. + +The CPU model features for AP +----------------------------- +The AP stack relies on the presence of the AP instructions as well as two +facilities: The AP Facilities Test (APFT) facility; and the AP Query +Configuration Information (QCI) facility. These features/facilities are made +available to a KVM guest via the following CPU model features: + +1. ap: Indicates whether the AP instructions are installed on the guest. This + feature will be enabled by KVM only if the AP instructions are installed + on the host. + +2. apft: Indicates the APFT facility is available on the guest. This facility + can be made available to the guest only if it is available on the host (i.e., + facility bit 15 is set). + +3. apqci: Indicates the AP QCI facility is available on the guest. This facility + can be made available to the guest only if it is available on the host (i.e., + facility bit 12 is set). + +Note: If the user chooses to specify a CPU model different than the 'host' +model to QEMU, the CPU model features and facilities need to be turned on +explicitly; for example: + + /usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on + +A guest can be precluded from using AP features/facilities by turning them off +explicitly; for example: + + /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off + +Note: If the APFT facility is turned off (apft=off) for the guest, the guest +will not see any AP devices. The zcrypt device drivers that register for type 10 +and newer AP devices - i.e., the cex4card and cex4queue device drivers - need +the APFT facility to ascertain the facilities installed on a given AP device. If +the APFT facility is not installed on the guest, then the probe of device +drivers will fail since only type 10 and newer devices can be configured for +guest use. + +Example: +======= +Let's now provide an example to illustrate how KVM guests may be given +access to AP facilities. For this example, we will show how to configure +three guests such that executing the lszcrypt command on the guests would +look like this: + +Guest1 +------ +CARD.DOMAIN TYPE MODE +------------------------------ +05 CEX5C CCA-Coproc +05.0004 CEX5C CCA-Coproc +05.00ab CEX5C CCA-Coproc +06 CEX5A Accelerator +06.0004 CEX5A Accelerator +06.00ab CEX5C CCA-Coproc + +Guest2 +------ +CARD.DOMAIN TYPE MODE +------------------------------ +05 CEX5A Accelerator +05.0047 CEX5A Accelerator +05.00ff CEX5A Accelerator (5,4), (5,171), (6,4), (6,171), + +Guest2 +------ +CARD.DOMAIN TYPE MODE +------------------------------ +06 CEX5A Accelerator +06.0047 CEX5A Accelerator +06.00ff CEX5A Accelerator + +These are the steps: + +1. Install the vfio_ap module on the linux host. The dependency chain for the + vfio_ap module is: + * iommu + * s390 + * zcrypt + * vfio + * vfio_mdev + * vfio_mdev_device + * KVM + + To build the vfio_ap module, the kernel build must be configured with the + following Kconfig elements selected: + * IOMMU_SUPPORT + * S390 + * ZCRYPT + * S390_AP_IOMMU + * VFIO + * VFIO_MDEV + * VFIO_MDEV_DEVICE + * KVM + + If using make menuconfig select the following to build the vfio_ap module: + -> Device Drivers + -> IOMMU Hardware Support + select S390 AP IOMMU Support + -> VFIO Non-Privileged userspace driver framework + -> Mediated device driver frramework + -> VFIO driver for Mediated devices + -> I/O subsystem + -> VFIO support for AP devices + +2. Secure the AP queues to be used by the three guests so that the host can not + access them. To secure them, there are two sysfs files that specify + bitmasks marking a subset of the APQN range as 'usable by the default AP + queue device drivers' or 'not usable by the default device drivers' and thus + available for use by the vfio_ap device driver'. The sysfs files containing + the sysfs locations of the masks are: + + /sys/bus/ap/apmask + /sys/bus/ap/aqmask + + The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs + (APID). Each bit in the mask, from most significant to least significant bit, + corresponds to an APID from 0-255. If a bit is set, the APID is marked as + usable only by the default AP queue device drivers; otherwise, the APID is + usable by the vfio_ap device driver. + + The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes + (APQI). Each bit in the mask, from most significant to least significant bit, + corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as + usable only by the default AP queue device drivers; otherwise, the APQI is + usable by the vfio_ap device driver. + + The APQN of each AP queue device assigned to the linux host is checked by the + AP bus against the set of APQNs derived from the cross product of APIDs + and APQIs marked as usable only by the default AP queue device drivers. If a + match is detected, only the default AP queue device drivers will be probed; + otherwise, the vfio_ap device driver will be probed. + + By default, the two masks are set to reserve all APQNs for use by the default + AP queue device drivers. There are two ways the default masks can be changed: + + 1. The masks can be changed at boot time with the kernel command line + like this: + + ap.apmask=0xffff ap.aqmask=0x40 + + This would give these two pools: + + default drivers pool: adapter 0-15, domain 1 + alternate drivers pool: adapter 16-255, domains 2-255 + + 2. The sysfs mask files can also be edited by echoing a string into the + respective file in one of two formats: + + * An absolute hex string starting with 0x - like "0x12345678" - sets + the mask. If the given string is shorter than the mask, it is padded + with 0s on the right. If the string is longer than the mask, the + operation is terminated with an error (EINVAL). + + * A plus ('+') or minus ('-') followed by a numerical value. Valid + examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only + the corresponding bit in the mask is switched on ('+') or off ('-'). The + values may also be specified in a comma-separated list to switch more + than one bit on or off. + + To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, + 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding + APQNs must be removed from the masks as follows: + + echo -5,-6 > /sys/bus/ap/apmask + + echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask + + This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, + 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The + sysfs directory for the vfio_ap device driver will now contain symbolic links + to the AP queue devices bound to it: + + /sys/bus/ap + ... [drivers] + ...... [vfio_ap] + ......... [05.0004] + ......... [05.0047] + ......... [05.00ab] + ......... [05.00ff] + ......... [06.0004] + ......... [06.0047] + ......... [06.00ab] + ......... [06.00ff] + + Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later) + can be bound to the vfio_ap device driver. The reason for this is to + simplify the implementation by not needlessly complicating the design by + supporting older devices that will go out of service in the relatively near + future and for which there are few older systems on which to test. + + The administrator, therefore, must take care to secure only AP queues that + can be bound to the vfio_ap device driver. The device type for a given AP + queue device can be read from the parent card's sysfs directory. For example, + to see the hardware type of the queue 05.0004: + + cat /sys/bus/ap/devices/card05/hwtype + + The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the + vfio_ap device driver. + +3. Create the mediated devices needed to configure the AP matrixes for the + three guests and to provide an interface to the vfio_ap driver for + use by the guests: + + /sys/devices/vfio_ap/matrix/ + --- [mdev_supported_types] + ------ [vfio_ap-passthrough] (passthrough mediated matrix device type) + --------- create + --------- [devices] + + To create the mediated devices for the three guests: + + uuidgen > create + uuidgen > create + uuidgen > create + + or + + echo $uuid1 > create + echo $uuid2 > create + echo $uuid3 > create + + This will create three mediated devices in the [devices] subdirectory named + after the UUID written to the create attribute file. We call them $uuid1, + $uuid2 and $uuid3 and this is the sysfs directory structure after creation: + + /sys/devices/vfio_ap/matrix/ + --- [mdev_supported_types] + ------ [vfio_ap-passthrough] + --------- [devices] + ------------ [$uuid1] + --------------- assign_adapter + --------------- assign_control_domain + --------------- assign_domain + --------------- matrix + --------------- unassign_adapter + --------------- unassign_control_domain + --------------- unassign_domain + + ------------ [$uuid2] + --------------- assign_adapter + --------------- assign_control_domain + --------------- assign_domain + --------------- matrix + --------------- unassign_adapter + ----------------unassign_control_domain + ----------------unassign_domain + + ------------ [$uuid3] + --------------- assign_adapter + --------------- assign_control_domain + --------------- assign_domain + --------------- matrix + --------------- unassign_adapter + ----------------unassign_control_domain + ----------------unassign_domain + +4. The administrator now needs to configure the matrixes for the mediated + devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3). + + This is how the matrix is configured for Guest1: + + echo 5 > assign_adapter + echo 6 > assign_adapter + echo 4 > assign_domain + echo 0xab > assign_domain + + Control domains can similarly be assigned using the assign_control_domain + sysfs file. + + If a mistake is made configuring an adapter, domain or control domain, + you can use the unassign_xxx files to unassign the adapter, domain or + control domain. + + To display the matrix configuration for Guest1: + + cat matrix + + This is how the matrix is configured for Guest2: + + echo 5 > assign_adapter + echo 0x47 > assign_domain + echo 0xff > assign_domain + + This is how the matrix is configured for Guest3: + + echo 6 > assign_adapter + echo 0x47 > assign_domain + echo 0xff > assign_domain + + In order to successfully assign an adapter: + + * The adapter number specified must represent a value from 0 up to the + maximum adapter number configured for the system. If an adapter number + higher than the maximum is specified, the operation will terminate with + an error (ENODEV). + + * All APQNs that can be derived from the adapter ID and the IDs of + the previously assigned domains must be bound to the vfio_ap device + driver. If no domains have yet been assigned, then there must be at least + one APQN with the specified APID bound to the vfio_ap driver. If no such + APQNs are bound to the driver, the operation will terminate with an + error (EADDRNOTAVAIL). + + No APQN that can be derived from the adapter ID and the IDs of the + previously assigned domains can be assigned to another mediated matrix + device. If an APQN is assigned to another mediated matrix device, the + operation will terminate with an error (EADDRINUSE). + + In order to successfully assign a domain: + + * The domain number specified must represent a value from 0 up to the + maximum domain number configured for the system. If a domain number + higher than the maximum is specified, the operation will terminate with + an error (ENODEV). + + * All APQNs that can be derived from the domain ID and the IDs of + the previously assigned adapters must be bound to the vfio_ap device + driver. If no domains have yet been assigned, then there must be at least + one APQN with the specified APQI bound to the vfio_ap driver. If no such + APQNs are bound to the driver, the operation will terminate with an + error (EADDRNOTAVAIL). + + No APQN that can be derived from the domain ID and the IDs of the + previously assigned adapters can be assigned to another mediated matrix + device. If an APQN is assigned to another mediated matrix device, the + operation will terminate with an error (EADDRINUSE). + + In order to successfully assign a control domain, the domain number + specified must represent a value from 0 up to the maximum domain number + configured for the system. If a control domain number higher than the maximum + is specified, the operation will terminate with an error (ENODEV). + +5. Start Guest1: + + /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \ + -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ... + +7. Start Guest2: + + /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \ + -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ... + +7. Start Guest3: + + /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \ + -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ... + +When the guest is shut down, the mediated matrix devices may be removed. + +Using our example again, to remove the mediated matrix device $uuid1: + + /sys/devices/vfio_ap/matrix/ + --- [mdev_supported_types] + ------ [vfio_ap-passthrough] + --------- [devices] + ------------ [$uuid1] + --------------- remove + + + echo 1 > remove + + This will remove all of the mdev matrix device's sysfs structures including + the mdev device itself. To recreate and reconfigure the mdev matrix device, + all of the steps starting with step 3 will have to be performed again. Note + that the remove will fail if a guest using the mdev is still running. + + It is not necessary to remove an mdev matrix device, but one may want to + remove it if no guest will use it during the remaining lifetime of the linux + host. If the mdev matrix device is removed, one may want to also reconfigure + the pool of adapters and queues reserved for use by the default drivers. + +Limitations +=========== +* The KVM/kernel interfaces do not provide a way to prevent restoring an APQN + to the default drivers pool of a queue that is still assigned to a mediated + device in use by a guest. It is incumbent upon the administrator to + ensure there is no mediated device in use by a guest to which the APQN is + assigned lest the host be given access to the private data of the AP queue + device such as a private key configured specifically for the guest. + +* Dynamically modifying the AP matrix for a running guest (which would amount to + hot(un)plug of AP devices for the guest) is currently not supported + +* Live guest migration is not supported for guests using AP devices. diff --git a/MAINTAINERS b/MAINTAINERS index 5db5b7e49d08..1610fb26bdac 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -12678,6 +12678,7 @@ S: Supported F: drivers/s390/crypto/vfio_ap_drv.c F: drivers/s390/crypto/vfio_ap_private.h F: drivers/s390/crypto/vfio_ap_ops.c +F: Documentation/s390/vfio-ap.txt S390 ZFCP DRIVER M: Steffen Maier <maier@linux.ibm.com>