[0/1] PCI: Add decode disable/enable to device level and separate BAR info logging into separate function

Message

Mitchell Augustin Dec. 6, 2024, 12:03 a.m. UTC

In a virtualized environment with PCI devices passed-through,
disabling and enabling decoding is not trivial, and in some
configurations can take up to 2 seconds of wall clock time to
complete. Since this operation could be done once around each
device for devices with multiple BARs (reducing redundancy),
add an additional decode disable/enable mask at the device
level to prevent redundant disable/enables from occurring
during each BAR sizing operation, when pci_read_bases() is
the originator.

Since __pci_read_base() can also be called independently,
keep the current disable/enable mask in that function as-is.

Since printk cannot be used while decoding is disabled,
move the debug prints in __pci_read_base() to
a separate function, __pci_print_bar_status().
To enable this, add pointers to the signature for
__pci_read_base() through which the caller can access
necessary data from __pci_read_base() and pass it to
__pci_print_bar_status().

This has been tested on an SR670v2 host and guest VM,
a DGX H100 host and guest VM, and a DGX A100 host and guest
VM. I confirmed that BAR info logged to dmesg was consistent
on each between unmodified 6.12.1 and with this patch, that
BAR mappings in /proc/iomem were consistent between
versions, and that lspci -vv results were consistent
between versions. On the A100/H100, I also confirmed that the
Nvidia driver loads as expected with the patch, and that
VM boot time with cold-plugged, passed-through GPUs is about
2x faster. No regressions were observed.

Originally developed/tested against upstream 6.12.1, but
I also confirmed that this applies cleanly to the pci tree
@ v6.13-rc1 and that it results in the same boot speed
decrease there when tested in an H100 guest VM.

Link: https://lore.kernel.org/all/[e-mail]/
Reported-by: "Mitchell Augustin" <[e-mail]>
Closes: https://lore.kernel.org/all/[e-mail]/

Mitchell Augustin (1):
  PCI: Add decode disable/enable to device level and separate BAR info
    logging into separate function

 drivers/pci/iov.c   |  16 ++++-
 drivers/pci/pci.h   |   7 ++-
 drivers/pci/probe.c | 149 +++++++++++++++++++++++++++++++++++++++++---
 3 files changed, 158 insertions(+), 14 deletions(-)
---
base-commit: d390303b28dabbb91b2d32016a4f72da478733b9 

--
2.43.0