mbox series

[v5,0/2] vfio/pci: add msi interrupt affinity support

Message ID 20240610125713.86750-1-fgriffo@amazon.co.uk (mailing list archive)
Headers show
Series vfio/pci: add msi interrupt affinity support | expand

Message

Fred Griffoul June 10, 2024, 12:57 p.m. UTC
v5:
 - vfio_pci_ioctl_set_irqs(): fix copy_from_user() check when copying
   the cpumask argument. Reported by Dan Carpenter
   <dan.carpenter@linaro.org>
 - vfio_set_irqs_validate_and_prepare(): use size_mul() to compute the
   data size of a VFIO_IRQ_SET_DATA_EVENTFD ioctl() to avoid a possible
   overflow on 32-bit system. Reported by Dan Carpenter
   <dan.carpenter@linaro.org>
 - export system_32bit_el0_cpumask() to fix yet another missing symbol
   for arm64 architecture.

v4:
 - export arm64_mismatched_32bit_el0 to compile the vfio driver as
   a kernel module on arm64 if CONFIG_CPUSETS is not defined.
 - vfio_pci_ioctl_set_irqs(): free the cpumask_var_t only if data_size
   is not zero, otherwise it was not allocated.
 - vfio_pci_set_msi_trigger(): call the new function
   vfio_pci_set_msi_affinity() later, after the DATA_EVENTFD
   processing and the vdev index check.

v3:
 - add a first patch to export cpuset_cpus_allowed() to be able to
   compile the vfio driver as a kernel module.

v2:
 - change the ioctl() interface to use a cpu_set_t in vfio_irq_set
   'data' to keep the 'start' and 'count' semantic, as suggested by
    David Woodhouse <dwmw2@infradead.org>

v1:

The usual way to configure a device interrupt from userland is to write
the /proc/irq/<irq>/smp_affinity or smp_affinity_list files. When using
vfio to implement a device driver or a virtual machine monitor, this may
not be ideal: the process managing the vfio device interrupts may not be
granted root privilege, for security reasons. Thus it cannot directly
control the interrupt affinity and has to rely on an external command.

This patch extends the VFIO_DEVICE_SET_IRQS ioctl() with a new data flag
to specify the affinity of a vfio pci device interrupt.

The affinity argument must be a subset of the process cpuset, otherwise
an error -EPERM is returned.

The vfio_irq_set argument shall be set-up in the following way:

- the 'flags' field have the new flag VFIO_IRQ_SET_DATA_AFFINITY set
as well as VFIO_IRQ_SET_ACTION_TRIGGER.

- the 'start' field is the device interrupt index. Only one interrupt
can be configured per ioctl().

- the variable-length array consists of one or more CPU index
encoded as __u32, the number of entries in the array is specified in the
'count' field.


Fred Griffoul (2):
  cgroup/cpuset: export cpuset_cpus_allowed()
  vfio/pci: add msi interrupt affinity support

 arch/arm64/kernel/cpufeature.c    |  2 ++
 drivers/vfio/pci/vfio_pci_core.c  | 27 +++++++++++++++++----
 drivers/vfio/pci/vfio_pci_intrs.c | 39 +++++++++++++++++++++++++++++++
 drivers/vfio/vfio_main.c          | 13 +++++++----
 include/uapi/linux/vfio.h         | 10 +++++++-
 kernel/cgroup/cpuset.c            |  1 +
 6 files changed, 83 insertions(+), 9 deletions(-)


base-commit: cbb325e77fbe62a06184175aa98c9eb98736c3e8
--
2.40.1

Comments

Michal Koutný June 10, 2024, 4:31 p.m. UTC | #1
Hello Fred.

On Mon, Jun 10, 2024 at 12:57:06PM GMT, Fred Griffoul <fgriffo@amazon.co.uk> wrote:
> The usual way to configure a device interrupt from userland is to write
> the /proc/irq/<irq>/smp_affinity or smp_affinity_list files. When using
> vfio to implement a device driver or a virtual machine monitor, this may
> not be ideal: the process managing the vfio device interrupts may not be
> granted root privilege, for security reasons. Thus it cannot directly
> control the interrupt affinity and has to rely on an external command.

External commands something privileged? (I'm curious of an example how
this is setup.)

> The affinity argument must be a subset of the process cpuset, otherwise
> an error -EPERM is returned.

I'm not sure you want to look at task's cpuset mask for this purposes.

Consider setups without cpuset or a change of (cpuset) mask anytime
during lifetime of the task...

Michal
Frederic Griffoul June 12, 2024, 2:45 p.m. UTC | #2
MIchal

To be honest my initial idea was to store an affinity mask per vfio group, which
can be done in the privileged process setting the vfio group/device owner, and
later apply the mask to each interrupt of each device in the group.

It would still require to fix the affinity of all the interrupts if
the vfio group affinity is
changed (or deliberately ignore this case). And it did not match
exactly my use case
where I need the process handling the interrupts to sometimes be able
to change them
but always within the cpuset. So I would still need the current patch,
in addition to
a new ioctl() to set the affinity mask of a vfio group.

Br,

Fred

On Mon, Jun 10, 2024 at 5:31 PM Michal Koutný <mkoutny@suse.com> wrote:
>
> Hello Fred.
>
> On Mon, Jun 10, 2024 at 12:57:06PM GMT, Fred Griffoul <fgriffo@amazon.co.uk> wrote:
> > The usual way to configure a device interrupt from userland is to write
> > the /proc/irq/<irq>/smp_affinity or smp_affinity_list files. When using
> > vfio to implement a device driver or a virtual machine monitor, this may
> > not be ideal: the process managing the vfio device interrupts may not be
> > granted root privilege, for security reasons. Thus it cannot directly
> > control the interrupt affinity and has to rely on an external command.
>
> External commands something privileged? (I'm curious of an example how
> this is setup.)
>
> > The affinity argument must be a subset of the process cpuset, otherwise
> > an error -EPERM is returned.
>
> I'm not sure you want to look at task's cpuset mask for this purposes.
>
> Consider setups without cpuset or a change of (cpuset) mask anytime
> during lifetime of the task...
>
> Michal
Michal Koutný June 26, 2024, 11:49 a.m. UTC | #3
On Wed, Jun 12, 2024 at 03:45:48PM GMT, Frederic Griffoul <griffoul@gmail.com> wrote:
> To be honest my initial idea was to store an affinity mask per vfio group, which
> can be done in the privileged process setting the vfio group/device owner, and
> later apply the mask to each interrupt of each device in the group.
> 
> It would still require to fix the affinity of all the interrupts if
> the vfio group affinity is
> changed (or deliberately ignore this case). And it did not match
> exactly my use case
> where I need the process handling the interrupts to sometimes be able
> to change them
> but always within the cpuset. So I would still need the current patch,
> in addition to
> a new ioctl() to set the affinity mask of a vfio group.

It's not clear to me what is the relation between the process A calling
that ioctl() (and whose cpuset is used to check affinity for)
and a process B that ends up handling the IRQ.
At which place would process B be scheduled on IRQ's affinity CPUs?
(I'm not familiar with VFIO.)

Thanks,
Michal