Message ID | 20240514143400.152280-2-stewart.hildebrand@amd.com (mailing list archive) |
---|---|
State | Superseded |
Headers | show |
Series | PCI devices passthrough on Arm, part 3 | expand |
On 14.05.2024 16:33, Stewart Hildebrand wrote: > From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> > > Xen and/or Dom0 may have put values in PCI_COMMAND which they expect > to remain unaltered. PCI_COMMAND_SERR bit is a good example: while the > guest's (domU) view of this will want to be zero (for now), the host > having set it to 1 should be preserved, or else we'd effectively be > giving the domU control of the bit. Thus, PCI_COMMAND register needs > proper emulation in order to honor host's settings. > > According to "PCI LOCAL BUS SPECIFICATION, REV. 3.0", section "6.2.2 > Device Control" the reset state of the command register is typically 0, > so when assigning a PCI device use 0 as the initial state for the > guest's (domU) view of the command register. > > Here is the full list of command register bits with notes about > PCI/PCIe specification, and how Xen handles the bit. QEMU's behavior is > also documented here since that is our current reference implementation > for PCI passthrough. > > PCI_COMMAND_IO (bit 0) > PCIe 6.1: RW > PCI LB 3.0: RW > QEMU: (emu_mask) QEMU provides an emulated view of this bit. Guest > writes do not propagate to hardware. QEMU sets this bit to 1 in > hardware if an I/O BAR is exposed to the guest. > Xen domU: (rsvdp_mask) We treat this bit as RsvdP for now since we > don't yet support I/O BARs for domUs. > Xen dom0: We allow dom0 to control this bit freely. > > PCI_COMMAND_MEMORY (bit 1) > PCIe 6.1: RW > PCI LB 3.0: RW > QEMU: (emu_mask) QEMU provides an emulated view of this bit. Guest > writes do not propagate to hardware. QEMU sets this bit to 1 in > hardware if a Memory BAR is exposed to the guest. > Xen domU/dom0: We handle writes to this bit by mapping/unmapping BAR > regions. > Xen domU: For devices assigned to DomUs, memory decoding will be > disabled at the time of initialization. > > PCI_COMMAND_MASTER (bit 2) > PCIe 6.1: RW > PCI LB 3.0: RW > QEMU: Pass through writes to hardware. > Xen domU/dom0: Pass through writes to hardware. > > PCI_COMMAND_SPECIAL (bit 3) > PCIe 6.1: RO, hardwire to 0 > PCI LB 3.0: RW > QEMU: Pass through writes to hardware. > Xen domU/dom0: Pass through writes to hardware. > > PCI_COMMAND_INVALIDATE (bit 4) > PCIe 6.1: RO, hardwire to 0 > PCI LB 3.0: RW > QEMU: Pass through writes to hardware. > Xen domU/dom0: Pass through writes to hardware. > > PCI_COMMAND_VGA_PALETTE (bit 5) > PCIe 6.1: RO, hardwire to 0 > PCI LB 3.0: RW > QEMU: Pass through writes to hardware. > Xen domU/dom0: Pass through writes to hardware. > > PCI_COMMAND_PARITY (bit 6) > PCIe 6.1: RW > PCI LB 3.0: RW > QEMU: (emu_mask) QEMU provides an emulated view of this bit. Guest > writes do not propagate to hardware. > Xen domU: (rsvdp_mask) We treat this bit as RsvdP. > Xen dom0: We allow dom0 to control this bit freely. > > PCI_COMMAND_WAIT (bit 7) > PCIe 6.1: RO, hardwire to 0 > PCI LB 3.0: hardwire to 0 > QEMU: res_mask > Xen domU: (rsvdp_mask) We treat this bit as RsvdP. > Xen dom0: We allow dom0 to control this bit freely. > > PCI_COMMAND_SERR (bit 8) > PCIe 6.1: RW > PCI LB 3.0: RW > QEMU: (emu_mask) QEMU provides an emulated view of this bit. Guest > writes do not propagate to hardware. > Xen domU: (rsvdp_mask) We treat this bit as RsvdP. > Xen dom0: We allow dom0 to control this bit freely. > > PCI_COMMAND_FAST_BACK (bit 9) > PCIe 6.1: RO, hardwire to 0 > PCI LB 3.0: RW > QEMU: (emu_mask) QEMU provides an emulated view of this bit. Guest > writes do not propagate to hardware. > Xen domU: (rsvdp_mask) We treat this bit as RsvdP. > Xen dom0: We allow dom0 to control this bit freely. > > PCI_COMMAND_INTX_DISABLE (bit 10) > PCIe 6.1: RW > PCI LB 3.0: RW > QEMU: (emu_mask) QEMU provides an emulated view of this bit. Guest > writes do not propagate to hardware. QEMU checks if INTx was mapped > for a device. If it is not, then guest can't control > PCI_COMMAND_INTX_DISABLE bit. > Xen domU: We prohibit a guest from enabling INTx if MSI(X) is enabled. > Xen dom0: We allow dom0 to control this bit freely. > > Bits 11-15 > PCIe 6.1: RsvdP > PCI LB 3.0: Reserved > QEMU: res_mask > Xen domU/dom0: rsvdp_mask > > Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> > Signed-off-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com> > Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com> Reviewed-by: Jan Beulich <jbeulich@suse.com> with one further nit: > --- a/xen/drivers/vpci/msi.c > +++ b/xen/drivers/vpci/msi.c > @@ -57,6 +57,7 @@ static void cf_check control_write( > > if ( new_enabled ) > { > + bool old_enabled = msi->enabled; > /* > * If the device is already enabled it means the number of > * enabled messages has changed. Disable and re-enable the Blank line please to separate declaration(s) from statement(s) (or, in this case, a comment ahead of a statement). Happy to insert while committing. Jan
diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 47648c395132..2491dbae8901 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -524,9 +524,21 @@ static void cf_check cmd_write( { struct vpci_header *header = data; + if ( !is_hardware_domain(pdev->domain) ) + { + const struct vpci *vpci = pdev->vpci; + + if ( (vpci->msi && vpci->msi->enabled) || + (vpci->msix && vpci->msix->enabled) ) + cmd |= PCI_COMMAND_INTX_DISABLE; + + header->guest_cmd = cmd; + } + /* * Let Dom0 play with all the bits directly except for the memory - * decoding one. + * decoding one. Bits that are not allowed for DomU are already + * handled above and by the rsvdp_mask. */ if ( header->bars_mapped != !!(cmd & PCI_COMMAND_MEMORY) ) /* @@ -540,6 +552,14 @@ static void cf_check cmd_write( pci_conf_write16(pdev->sbdf, reg, cmd); } +static uint32_t cf_check guest_cmd_read( + const struct pci_dev *pdev, unsigned int reg, void *data) +{ + const struct vpci_header *header = data; + + return header->guest_cmd; +} + static void cf_check bar_write( const struct pci_dev *pdev, unsigned int reg, uint32_t val, void *data) { @@ -754,9 +774,23 @@ static int cf_check init_header(struct pci_dev *pdev) return -EOPNOTSUPP; } - /* Setup a handler for the command register. */ - rc = vpci_add_register(pdev->vpci, vpci_hw_read16, cmd_write, PCI_COMMAND, - 2, header); + /* + * Setup a handler for the command register. + * + * TODO: If support for emulated bits is added, re-visit how to handle + * PCI_COMMAND_PARITY, PCI_COMMAND_SERR, and PCI_COMMAND_FAST_BACK. + */ + rc = vpci_add_register_mask(pdev->vpci, + is_hwdom ? vpci_hw_read16 : guest_cmd_read, + cmd_write, PCI_COMMAND, 2, header, 0, 0, + PCI_COMMAND_RSVDP_MASK | + (is_hwdom ? 0 + : PCI_COMMAND_IO | + PCI_COMMAND_PARITY | + PCI_COMMAND_WAIT | + PCI_COMMAND_SERR | + PCI_COMMAND_FAST_BACK), + 0); if ( rc ) return rc; @@ -836,9 +870,23 @@ static int cf_check init_header(struct pci_dev *pdev) if ( pdev->ignore_bars ) return 0; - /* Disable memory decoding before sizing. */ cmd = pci_conf_read16(pdev->sbdf, PCI_COMMAND); - if ( cmd & PCI_COMMAND_MEMORY ) + + /* + * For DomUs, clear PCI_COMMAND_{MASTER,MEMORY,IO} and other + * DomU-controllable bits in PCI_COMMAND. Devices assigned to DomUs will + * start with memory decoding disabled, and modify_bars() will not be called + * at the end of this function. + */ + if ( !is_hwdom ) + cmd &= ~(PCI_COMMAND_VGA_PALETTE | PCI_COMMAND_INVALIDATE | + PCI_COMMAND_SPECIAL | PCI_COMMAND_MASTER | PCI_COMMAND_MEMORY | + PCI_COMMAND_IO); + + header->guest_cmd = cmd; + + /* Disable memory decoding before sizing. */ + if ( !is_hwdom || (cmd & PCI_COMMAND_MEMORY) ) pci_conf_write16(pdev->sbdf, PCI_COMMAND, cmd & ~PCI_COMMAND_MEMORY); for ( i = 0; i < num_bars; i++ ) diff --git a/xen/drivers/vpci/msi.c b/xen/drivers/vpci/msi.c index 30adcf7df05d..3e414e69a432 100644 --- a/xen/drivers/vpci/msi.c +++ b/xen/drivers/vpci/msi.c @@ -57,6 +57,7 @@ static void cf_check control_write( if ( new_enabled ) { + bool old_enabled = msi->enabled; /* * If the device is already enabled it means the number of * enabled messages has changed. Disable and re-enable the @@ -70,6 +71,13 @@ static void cf_check control_write( if ( vpci_msi_arch_enable(msi, pdev, vectors) ) return; + + /* Make sure domU doesn't enable INTx while enabling MSI. */ + if ( !old_enabled && !is_hardware_domain(pdev->domain) ) + { + pci_intx(pdev, false); + pdev->vpci->header.guest_cmd |= PCI_COMMAND_INTX_DISABLE; + } } else vpci_msi_arch_disable(msi, pdev); diff --git a/xen/drivers/vpci/msix.c b/xen/drivers/vpci/msix.c index 58c16ebdf283..fbe710ab92ef 100644 --- a/xen/drivers/vpci/msix.c +++ b/xen/drivers/vpci/msix.c @@ -135,6 +135,13 @@ static void cf_check control_write( } } + /* Make sure domU doesn't enable INTx while enabling MSI-X. */ + if ( new_enabled && !msix->enabled && !is_hardware_domain(pdev->domain) ) + { + pci_intx(pdev, false); + pdev->vpci->header.guest_cmd |= PCI_COMMAND_INTX_DISABLE; + } + msix->masked = new_masked; msix->enabled = new_enabled; diff --git a/xen/include/xen/pci_regs.h b/xen/include/xen/pci_regs.h index 0bc18efabb74..250ba106dbd3 100644 --- a/xen/include/xen/pci_regs.h +++ b/xen/include/xen/pci_regs.h @@ -48,6 +48,7 @@ #define PCI_COMMAND_SERR 0x100 /* Enable SERR */ #define PCI_COMMAND_FAST_BACK 0x200 /* Enable back-to-back writes */ #define PCI_COMMAND_INTX_DISABLE 0x400 /* INTx Emulation Disable */ +#define PCI_COMMAND_RSVDP_MASK 0xf800 #define PCI_STATUS 0x06 /* 16 bits */ #define PCI_STATUS_IMM_READY 0x01 /* Immediate Readiness */ diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 6e4c972f35ed..2064d9354f5b 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -107,6 +107,9 @@ struct vpci { } bars[PCI_HEADER_NORMAL_NR_BARS + 1]; /* At most 6 BARS + 1 expansion ROM BAR. */ + /* Guest (domU only) view of the PCI_COMMAND register. */ + uint16_t guest_cmd; + /* * Store whether the ROM enable bit is set (doesn't imply ROM BAR * is mapped into guest p2m) if there's a ROM BAR on the device.