| Message ID | 20250211022257.1690366-1-Jiqian.Chen@amd.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | [v8] vpci: Add resizable bar support | expand |
On Tue, Feb 11, 2025 at 10:22:57AM +0800, Jiqian Chen wrote: > Some devices, like AMDGPU, support resizable bar capability, > but vpci of Xen doesn't support this feature, so they fail > to resize bars and then cause probing failure. > > According to PCIe spec, each bar that supports resizing has > two registers, PCI_REBAR_CAP and PCI_REBAR_CTRL. So, add > handlers to support resizing the size of BARs. > > Note that Xen will only trap PCI_REBAR_CTRL, as PCI_REBAR_CAP > is read-only register and the hardware domain already gets > access to it without needing any setup. > > Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> > --- > Hi all, > v7->v8 changes: > * Modified commit message and some comments. > * Deleted unused function vpci_hw_write32. > > Best regards, > Jiqian Chen. > > v6->v7 changes: > * Deleted codes that add register for PCI_REBAR_CAP, and added comments to explain why. > * Added comments to explain why use "continue" when fail to add register for PCI_REBAR_CTRL. > > v5->v6 changes: > * Changed "1UL" to "1ULL" in PCI_REBAR_CTRL_SIZE idefinition for 32 bit architecture. > * In rebar_ctrl_write used "bar - pdev->vpci->header.bars" to get index instead of reading > from register. > * Added the index of BAR to error messages. > * Changed to "continue" instead of "return an error" when vpci_add_register failed. > > v4->v5 changes: > * Called pci_size_mem_bar in rebar_ctrl_write to get addr and size of BAR instead of setting > their values directly after writing new size to hardware. > * Changed from "return" to "continue" when index/type of BAR are not correct during initializing > BAR. > * Corrected the value of PCI_REBAR_CTRL_BAR_SIZE from "0x00001F00" to "0x00003F00". > * Renamed PCI_REBAR_SIZE_BIAS to PCI_REBAR_CTRL_SIZE_BIAS. > * Re-defined "PCI_REBAR_CAP_SHIFT 4" to "PCI_REBAR_CAP_SIZES_MASK 0xFFFFFFF0U". > > v3->v4 changes: > * Removed PCI_REBAR_CAP_SIZES since it was not needed, and added > PCI_REBAR_CAP_SHIFT and PCI_REBAR_CTRL_SIZES. > * Added parameter resizable_sizes to struct vpci_bar to cache the support resizable sizes and > added the logic in init_rebar(). > * Changed PCI_REBAR_CAP to PCI_REBAR_CAP(n) (4+8*(n)), changed PCI_REBAR_CTRL to > PCI_REBAR_CTRL(n) (8+8*(n)). > * Added domain info of pci_dev to printings of init_rebar(). > > v2->v3 changes: > * Used "bar->enabled" to replace "pci_conf_read16(pdev->sbdf, PCI_COMMAND) & PCI_COMMAND_MEMORY", > and added comments why it needs this check. > * Added "!is_hardware_domain(pdev->domain)" check in init_rebar() to return EOPNOTSUPP for domUs. > * Moved BAR type and index check into init_rebar(), then only need to check once. > * Added 'U' suffix for macro PCI_REBAR_CAP_SIZES. > * Added macro PCI_REBAR_SIZE_BIAS to represent 20. > TODO: need to hide ReBar capability from hardware domain when init_rebar() fails. > > v1->v2 changes: > * In rebar_ctrl_write, to check if memory decoding is enabled, and added > some checks for the type of Bar. > * Added vpci_hw_write32 to handle PCI_REBAR_CAP's write, since there is > no write limitation of dom0. > * And has many other minor modifications as well. > --- > xen/drivers/vpci/Makefile | 2 +- > xen/drivers/vpci/rebar.c | 131 +++++++++++++++++++++++++++++++++++++ > xen/include/xen/pci_regs.h | 15 +++++ > xen/include/xen/vpci.h | 1 + > 4 files changed, 148 insertions(+), 1 deletion(-) > create mode 100644 xen/drivers/vpci/rebar.c > > diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile > index 1a1413b93e76..a7c8a30a8956 100644 > --- a/xen/drivers/vpci/Makefile > +++ b/xen/drivers/vpci/Makefile > @@ -1,2 +1,2 @@ > -obj-y += vpci.o header.o > +obj-y += vpci.o header.o rebar.o > obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o > diff --git a/xen/drivers/vpci/rebar.c b/xen/drivers/vpci/rebar.c > new file mode 100644 > index 000000000000..794f1168adf8 > --- /dev/null > +++ b/xen/drivers/vpci/rebar.c > @@ -0,0 +1,131 @@ > +/* SPDX-License-Identifier: GPL-2.0-only */ > +/* > + * Copyright (C) 2025 Advanced Micro Devices, Inc. All Rights Reserved. > + * > + * Author: Jiqian Chen <Jiqian.Chen@amd.com> > + */ > + > +#include <xen/sched.h> > +#include <xen/vpci.h> > + > +static void cf_check rebar_ctrl_write(const struct pci_dev *pdev, > + unsigned int reg, > + uint32_t val, > + void *data) > +{ > + struct vpci_bar *bar = data; > + const unsigned int index = bar - pdev->vpci->header.bars; > + uint64_t size = PCI_REBAR_CTRL_SIZE(val); Since you define index as const you could also do the same with size. Can adjust at commit, but I also don't have a strong opinion about it. Thanks, Roger.
On 2025/2/11 17:21, Roger Pau Monné wrote: > On Tue, Feb 11, 2025 at 10:22:57AM +0800, Jiqian Chen wrote: >> Some devices, like AMDGPU, support resizable bar capability, >> but vpci of Xen doesn't support this feature, so they fail >> to resize bars and then cause probing failure. >> >> According to PCIe spec, each bar that supports resizing has >> two registers, PCI_REBAR_CAP and PCI_REBAR_CTRL. So, add >> handlers to support resizing the size of BARs. >> >> Note that Xen will only trap PCI_REBAR_CTRL, as PCI_REBAR_CAP >> is read-only register and the hardware domain already gets >> access to it without needing any setup. >> >> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> > > Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> Thank you! May I know whether this can be merged in Xen version 4.20? > >> --- >> Hi all, >> v7->v8 changes: >> * Modified commit message and some comments. >> * Deleted unused function vpci_hw_write32. >> >> Best regards, >> Jiqian Chen. >> --- >> xen/drivers/vpci/Makefile | 2 +- >> xen/drivers/vpci/rebar.c | 131 +++++++++++++++++++++++++++++++++++++ >> xen/include/xen/pci_regs.h | 15 +++++ >> xen/include/xen/vpci.h | 1 + >> 4 files changed, 148 insertions(+), 1 deletion(-) >> create mode 100644 xen/drivers/vpci/rebar.c >> >> diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile >> index 1a1413b93e76..a7c8a30a8956 100644 >> --- a/xen/drivers/vpci/Makefile >> +++ b/xen/drivers/vpci/Makefile >> @@ -1,2 +1,2 @@ >> -obj-y += vpci.o header.o >> +obj-y += vpci.o header.o rebar.o >> obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o >> diff --git a/xen/drivers/vpci/rebar.c b/xen/drivers/vpci/rebar.c >> new file mode 100644 >> index 000000000000..794f1168adf8 >> --- /dev/null >> +++ b/xen/drivers/vpci/rebar.c >> @@ -0,0 +1,131 @@ >> +/* SPDX-License-Identifier: GPL-2.0-only */ >> +/* >> + * Copyright (C) 2025 Advanced Micro Devices, Inc. All Rights Reserved. >> + * >> + * Author: Jiqian Chen <Jiqian.Chen@amd.com> >> + */ >> + >> +#include <xen/sched.h> >> +#include <xen/vpci.h> >> + >> +static void cf_check rebar_ctrl_write(const struct pci_dev *pdev, >> + unsigned int reg, >> + uint32_t val, >> + void *data) >> +{ >> + struct vpci_bar *bar = data; >> + const unsigned int index = bar - pdev->vpci->header.bars; >> + uint64_t size = PCI_REBAR_CTRL_SIZE(val); > > Since you define index as const you could also do the same with size. > Can adjust at commit, but I also don't have a strong opinion about > it. Got it. If there aren't other comments, then I don't need to send a new version, right? > > Thanks, Roger.
On 14.02.2025 04:32, Chen, Jiqian wrote: > On 2025/2/11 17:21, Roger Pau Monné wrote: >> On Tue, Feb 11, 2025 at 10:22:57AM +0800, Jiqian Chen wrote: >>> Some devices, like AMDGPU, support resizable bar capability, >>> but vpci of Xen doesn't support this feature, so they fail >>> to resize bars and then cause probing failure. >>> >>> According to PCIe spec, each bar that supports resizing has >>> two registers, PCI_REBAR_CAP and PCI_REBAR_CTRL. So, add >>> handlers to support resizing the size of BARs. >>> >>> Note that Xen will only trap PCI_REBAR_CTRL, as PCI_REBAR_CAP >>> is read-only register and the hardware domain already gets >>> access to it without needing any setup. >>> >>> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> >> >> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> > Thank you! > May I know whether this can be merged in Xen version 4.20? That's a question Oleksii would have to answer. My take is that it's (far) too late in the cycle for a feature addition. Jan
On 2/14/25 8:35 AM, Jan Beulich wrote: > On 14.02.2025 04:32, Chen, Jiqian wrote: >> On 2025/2/11 17:21, Roger Pau Monné wrote: >>> On Tue, Feb 11, 2025 at 10:22:57AM +0800, Jiqian Chen wrote: >>>> Some devices, like AMDGPU, support resizable bar capability, >>>> but vpci of Xen doesn't support this feature, so they fail >>>> to resize bars and then cause probing failure. >>>> >>>> According to PCIe spec, each bar that supports resizing has >>>> two registers, PCI_REBAR_CAP and PCI_REBAR_CTRL. So, add >>>> handlers to support resizing the size of BARs. >>>> >>>> Note that Xen will only trap PCI_REBAR_CTRL, as PCI_REBAR_CAP >>>> is read-only register and the hardware domain already gets >>>> access to it without needing any setup. >>>> >>>> Signed-off-by: Jiqian Chen<Jiqian.Chen@amd.com> >>> Reviewed-by: Roger Pau Monné<roger.pau@citrix.com> >> Thank you! >> May I know whether this can be merged in Xen version 4.20? It would be better to merge it after the Xen 4.20 release. (It will happen in the next 2 weeks). Thanks. ~ Oleksii > That's a question Oleksii would have to answer. My take is that it's (far) > too late in the cycle for a feature addition.
On 2025/2/14 16:18, Oleksii Kurochko wrote: > > On 2/14/25 8:35 AM, Jan Beulich wrote: >> On 14.02.2025 04:32, Chen, Jiqian wrote: >>> On 2025/2/11 17:21, Roger Pau Monné wrote: >>>> On Tue, Feb 11, 2025 at 10:22:57AM +0800, Jiqian Chen wrote: >>>>> Some devices, like AMDGPU, support resizable bar capability, >>>>> but vpci of Xen doesn't support this feature, so they fail >>>>> to resize bars and then cause probing failure. >>>>> >>>>> According to PCIe spec, each bar that supports resizing has >>>>> two registers, PCI_REBAR_CAP and PCI_REBAR_CTRL. So, add >>>>> handlers to support resizing the size of BARs. >>>>> >>>>> Note that Xen will only trap PCI_REBAR_CTRL, as PCI_REBAR_CAP >>>>> is read-only register and the hardware domain already gets >>>>> access to it without needing any setup. >>>>> >>>>> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> >>>> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> >>> Thank you! >>> May I know whether this can be merged in Xen version 4.20? > > It would be better to merge it after the Xen 4.20 release. > (It will happen in the next 2 weeks). Got it. Thank you for reply. > > Thanks. > > ~ Oleksii > >> That's a question Oleksii would have to answer. My take is that it's (far) >> too late in the cycle for a feature addition.
On Fri, Feb 14, 2025 at 08:22:39AM +0000, Chen, Jiqian wrote: > On 2025/2/14 16:18, Oleksii Kurochko wrote: > > > > On 2/14/25 8:35 AM, Jan Beulich wrote: > >> On 14.02.2025 04:32, Chen, Jiqian wrote: > >>> On 2025/2/11 17:21, Roger Pau Monné wrote: > >>>> On Tue, Feb 11, 2025 at 10:22:57AM +0800, Jiqian Chen wrote: > >>>>> Some devices, like AMDGPU, support resizable bar capability, > >>>>> but vpci of Xen doesn't support this feature, so they fail > >>>>> to resize bars and then cause probing failure. > >>>>> > >>>>> According to PCIe spec, each bar that supports resizing has > >>>>> two registers, PCI_REBAR_CAP and PCI_REBAR_CTRL. So, add > >>>>> handlers to support resizing the size of BARs. > >>>>> > >>>>> Note that Xen will only trap PCI_REBAR_CTRL, as PCI_REBAR_CAP > >>>>> is read-only register and the hardware domain already gets > >>>>> access to it without needing any setup. > >>>>> > >>>>> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> > >>>> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> > >>> Thank you! > >>> May I know whether this can be merged in Xen version 4.20? > > > > It would be better to merge it after the Xen 4.20 release. > > (It will happen in the next 2 weeks). > Got it. Thank you for reply. Could you also add an entry to the CHANGELOG.md file to note that ReBAR is now supported on PVH dom0? Thanks, Roger.
On 2025/2/14 16:39, Roger Pau Monné wrote: > On Fri, Feb 14, 2025 at 08:22:39AM +0000, Chen, Jiqian wrote: >> On 2025/2/14 16:18, Oleksii Kurochko wrote: >>> >>> On 2/14/25 8:35 AM, Jan Beulich wrote: >>>> On 14.02.2025 04:32, Chen, Jiqian wrote: >>>>> On 2025/2/11 17:21, Roger Pau Monné wrote: >>>>>> On Tue, Feb 11, 2025 at 10:22:57AM +0800, Jiqian Chen wrote: >>>>>>> Some devices, like AMDGPU, support resizable bar capability, >>>>>>> but vpci of Xen doesn't support this feature, so they fail >>>>>>> to resize bars and then cause probing failure. >>>>>>> >>>>>>> According to PCIe spec, each bar that supports resizing has >>>>>>> two registers, PCI_REBAR_CAP and PCI_REBAR_CTRL. So, add >>>>>>> handlers to support resizing the size of BARs. >>>>>>> >>>>>>> Note that Xen will only trap PCI_REBAR_CTRL, as PCI_REBAR_CAP >>>>>>> is read-only register and the hardware domain already gets >>>>>>> access to it without needing any setup. >>>>>>> >>>>>>> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> >>>>>> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> >>>>> Thank you! >>>>> May I know whether this can be merged in Xen version 4.20? >>> >>> It would be better to merge it after the Xen 4.20 release. >>> (It will happen in the next 2 weeks). >> Got it. Thank you for reply. > > Could you also add an entry to the CHANGELOG.md file to note that > ReBAR is now supported on PVH dom0? Sure, I will add it in next version. And I think I need to wait until there is a 4.21.0 entry. In SUPPORT.md, do I need to change the sentence " * PCI SR-IOV and Resizable BARs." to " * PCI SR-IOV. "? > > Thanks, Roger.
On Fri, Feb 14, 2025 at 09:01:12AM +0000, Chen, Jiqian wrote: > On 2025/2/14 16:39, Roger Pau Monné wrote: > > On Fri, Feb 14, 2025 at 08:22:39AM +0000, Chen, Jiqian wrote: > >> On 2025/2/14 16:18, Oleksii Kurochko wrote: > >>> > >>> On 2/14/25 8:35 AM, Jan Beulich wrote: > >>>> On 14.02.2025 04:32, Chen, Jiqian wrote: > >>>>> On 2025/2/11 17:21, Roger Pau Monné wrote: > >>>>>> On Tue, Feb 11, 2025 at 10:22:57AM +0800, Jiqian Chen wrote: > >>>>>>> Some devices, like AMDGPU, support resizable bar capability, > >>>>>>> but vpci of Xen doesn't support this feature, so they fail > >>>>>>> to resize bars and then cause probing failure. > >>>>>>> > >>>>>>> According to PCIe spec, each bar that supports resizing has > >>>>>>> two registers, PCI_REBAR_CAP and PCI_REBAR_CTRL. So, add > >>>>>>> handlers to support resizing the size of BARs. > >>>>>>> > >>>>>>> Note that Xen will only trap PCI_REBAR_CTRL, as PCI_REBAR_CAP > >>>>>>> is read-only register and the hardware domain already gets > >>>>>>> access to it without needing any setup. > >>>>>>> > >>>>>>> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> > >>>>>> Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> > >>>>> Thank you! > >>>>> May I know whether this can be merged in Xen version 4.20? > >>> > >>> It would be better to merge it after the Xen 4.20 release. > >>> (It will happen in the next 2 weeks). > >> Got it. Thank you for reply. > > > > Could you also add an entry to the CHANGELOG.md file to note that > > ReBAR is now supported on PVH dom0? > Sure, I will add it in next version. And I think I need to wait until there is a 4.21.0 entry. Yes, I don't recall if the entry for 4.21 is created as part of the 4.20 release process, or the first commit to introduce a new feature also adds the boilerplate 4.21 section. > In SUPPORT.md, do I need to change the sentence " * PCI SR-IOV and Resizable BARs." to " * PCI SR-IOV. "? Oh, indeed, I forgot about that one. Yes please, also update SUPPORT.md. Thanks, Roger.
diff --git a/xen/drivers/vpci/Makefile b/xen/drivers/vpci/Makefile index 1a1413b93e76..a7c8a30a8956 100644 --- a/xen/drivers/vpci/Makefile +++ b/xen/drivers/vpci/Makefile @@ -1,2 +1,2 @@ -obj-y += vpci.o header.o +obj-y += vpci.o header.o rebar.o obj-$(CONFIG_HAS_PCI_MSI) += msi.o msix.o diff --git a/xen/drivers/vpci/rebar.c b/xen/drivers/vpci/rebar.c new file mode 100644 index 000000000000..794f1168adf8 --- /dev/null +++ b/xen/drivers/vpci/rebar.c @@ -0,0 +1,131 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2025 Advanced Micro Devices, Inc. All Rights Reserved. + * + * Author: Jiqian Chen <Jiqian.Chen@amd.com> + */ + +#include <xen/sched.h> +#include <xen/vpci.h> + +static void cf_check rebar_ctrl_write(const struct pci_dev *pdev, + unsigned int reg, + uint32_t val, + void *data) +{ + struct vpci_bar *bar = data; + const unsigned int index = bar - pdev->vpci->header.bars; + uint64_t size = PCI_REBAR_CTRL_SIZE(val); + + if ( bar->enabled ) + { + /* + * Refuse to resize a BAR while memory decoding is enabled, as + * otherwise the size of the mapped region in the p2m would become + * stale with the newly set BAR size, and the position of the BAR + * would be reset to undefined. Note the PCIe specification also + * forbids resizing a BAR with memory decoding enabled. + */ + if ( size != bar->size ) + gprintk(XENLOG_ERR, + "%pp: refuse to resize BAR#%u with memory decoding enabled\n", + &pdev->sbdf, index); + return; + } + + if ( !((size >> PCI_REBAR_CTRL_SIZE_BIAS) & bar->resizable_sizes) ) + gprintk(XENLOG_WARNING, + "%pp: new BAR#%u size %#lx is not supported by hardware\n", + &pdev->sbdf, index, size); + + pci_conf_write32(pdev->sbdf, reg, val); + + pci_size_mem_bar(pdev->sbdf, + PCI_BASE_ADDRESS_0 + index * 4, + &bar->addr, + &bar->size, + (index == PCI_HEADER_NORMAL_NR_BARS - 1) ? + PCI_BAR_LAST : 0); + bar->guest_addr = bar->addr; +} + +static int cf_check init_rebar(struct pci_dev *pdev) +{ + uint32_t ctrl; + unsigned int nbars; + unsigned int rebar_offset = pci_find_ext_capability(pdev->sbdf, + PCI_EXT_CAP_ID_REBAR); + + if ( !rebar_offset ) + return 0; + + if ( !is_hardware_domain(pdev->domain) ) + { + printk(XENLOG_ERR "%pp: resizable BARs unsupported for unpriv %pd\n", + &pdev->sbdf, pdev->domain); + return -EOPNOTSUPP; + } + + ctrl = pci_conf_read32(pdev->sbdf, rebar_offset + PCI_REBAR_CTRL(0)); + nbars = MASK_EXTR(ctrl, PCI_REBAR_CTRL_NBAR_MASK); + for ( unsigned int i = 0; i < nbars; i++ ) + { + int rc; + struct vpci_bar *bar; + unsigned int index; + + ctrl = pci_conf_read32(pdev->sbdf, rebar_offset + PCI_REBAR_CTRL(i)); + index = ctrl & PCI_REBAR_CTRL_BAR_IDX; + if ( index >= PCI_HEADER_NORMAL_NR_BARS ) + { + printk(XENLOG_ERR "%pd %pp: too big BAR number %u in REBAR_CTRL\n", + pdev->domain, &pdev->sbdf, index); + continue; + } + + bar = &pdev->vpci->header.bars[index]; + if ( bar->type != VPCI_BAR_MEM64_LO && bar->type != VPCI_BAR_MEM32 ) + { + printk(XENLOG_ERR "%pd %pp: BAR%u is not in memory space\n", + pdev->domain, &pdev->sbdf, index); + continue; + } + + rc = vpci_add_register(pdev->vpci, vpci_hw_read32, rebar_ctrl_write, + rebar_offset + PCI_REBAR_CTRL(i), 4, bar); + if ( rc ) + { + printk(XENLOG_ERR "%pd %pp: BAR%u fail to add reg of REBAR_CTRL rc=%d\n", + pdev->domain, &pdev->sbdf, index, rc); + /* + * Ideally we would hide the ReBar capability on error, but code + * for doing so still needs to be written. Use continue instead + * to keep any already setup register hooks, as returning an + * error will cause the hardware domain to get unmediated access + * to all device registers. + */ + continue; + } + + bar->resizable_sizes = + MASK_EXTR(pci_conf_read32(pdev->sbdf, + rebar_offset + PCI_REBAR_CAP(i)), + PCI_REBAR_CAP_SIZES_MASK); + bar->resizable_sizes |= + (((uint64_t)MASK_EXTR(ctrl, PCI_REBAR_CTRL_SIZES_MASK) << 32) / + ISOLATE_LSB(PCI_REBAR_CAP_SIZES_MASK)); + } + + return 0; +} +REGISTER_VPCI_INIT(init_rebar, VPCI_PRIORITY_LOW); + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/include/xen/pci_regs.h b/xen/include/xen/pci_regs.h index 250ba106dbd3..2f1d0d63e962 100644 --- a/xen/include/xen/pci_regs.h +++ b/xen/include/xen/pci_regs.h @@ -459,6 +459,7 @@ #define PCI_EXT_CAP_ID_ARI 14 #define PCI_EXT_CAP_ID_ATS 15 #define PCI_EXT_CAP_ID_SRIOV 16 +#define PCI_EXT_CAP_ID_REBAR 21 /* Resizable BAR */ /* Advanced Error Reporting */ #define PCI_ERR_UNCOR_STATUS 4 /* Uncorrectable Error Status */ @@ -541,6 +542,20 @@ #define PCI_VNDR_HEADER_REV(x) (((x) >> 16) & 0xf) #define PCI_VNDR_HEADER_LEN(x) (((x) >> 20) & 0xfff) +/* Resizable BARs */ +#define PCI_REBAR_CAP(n) (4 + 8 * (n)) /* capability register */ +#define PCI_REBAR_CAP_SIZES_MASK 0xFFFFFFF0U /* supported BAR sizes in CAP */ +#define PCI_REBAR_CTRL(n) (8 + 8 * (n)) /* control register */ +#define PCI_REBAR_CTRL_BAR_IDX 0x00000007 /* BAR index */ +#define PCI_REBAR_CTRL_NBAR_MASK 0x000000E0 /* # of resizable BARs */ +#define PCI_REBAR_CTRL_BAR_SIZE 0x00003F00 /* BAR size */ +#define PCI_REBAR_CTRL_SIZES_MASK 0xFFFF0000U /* supported BAR sizes in CTRL */ + +#define PCI_REBAR_CTRL_SIZE_BIAS 20 +#define PCI_REBAR_CTRL_SIZE(v) \ + (1ULL << (MASK_EXTR(v, PCI_REBAR_CTRL_BAR_SIZE) \ + + PCI_REBAR_CTRL_SIZE_BIAS)) + /* * Hypertransport sub capability types * diff --git a/xen/include/xen/vpci.h b/xen/include/xen/vpci.h index 41e7c3bc2791..807401b2eaa2 100644 --- a/xen/include/xen/vpci.h +++ b/xen/include/xen/vpci.h @@ -100,6 +100,7 @@ struct vpci { /* Guest address. */ uint64_t guest_addr; uint64_t size; + uint64_t resizable_sizes; struct rangeset *mem; enum { VPCI_BAR_EMPTY,
Some devices, like AMDGPU, support resizable bar capability, but vpci of Xen doesn't support this feature, so they fail to resize bars and then cause probing failure. According to PCIe spec, each bar that supports resizing has two registers, PCI_REBAR_CAP and PCI_REBAR_CTRL. So, add handlers to support resizing the size of BARs. Note that Xen will only trap PCI_REBAR_CTRL, as PCI_REBAR_CAP is read-only register and the hardware domain already gets access to it without needing any setup. Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> --- Hi all, v7->v8 changes: * Modified commit message and some comments. * Deleted unused function vpci_hw_write32. Best regards, Jiqian Chen. v6->v7 changes: * Deleted codes that add register for PCI_REBAR_CAP, and added comments to explain why. * Added comments to explain why use "continue" when fail to add register for PCI_REBAR_CTRL. v5->v6 changes: * Changed "1UL" to "1ULL" in PCI_REBAR_CTRL_SIZE idefinition for 32 bit architecture. * In rebar_ctrl_write used "bar - pdev->vpci->header.bars" to get index instead of reading from register. * Added the index of BAR to error messages. * Changed to "continue" instead of "return an error" when vpci_add_register failed. v4->v5 changes: * Called pci_size_mem_bar in rebar_ctrl_write to get addr and size of BAR instead of setting their values directly after writing new size to hardware. * Changed from "return" to "continue" when index/type of BAR are not correct during initializing BAR. * Corrected the value of PCI_REBAR_CTRL_BAR_SIZE from "0x00001F00" to "0x00003F00". * Renamed PCI_REBAR_SIZE_BIAS to PCI_REBAR_CTRL_SIZE_BIAS. * Re-defined "PCI_REBAR_CAP_SHIFT 4" to "PCI_REBAR_CAP_SIZES_MASK 0xFFFFFFF0U". v3->v4 changes: * Removed PCI_REBAR_CAP_SIZES since it was not needed, and added PCI_REBAR_CAP_SHIFT and PCI_REBAR_CTRL_SIZES. * Added parameter resizable_sizes to struct vpci_bar to cache the support resizable sizes and added the logic in init_rebar(). * Changed PCI_REBAR_CAP to PCI_REBAR_CAP(n) (4+8*(n)), changed PCI_REBAR_CTRL to PCI_REBAR_CTRL(n) (8+8*(n)). * Added domain info of pci_dev to printings of init_rebar(). v2->v3 changes: * Used "bar->enabled" to replace "pci_conf_read16(pdev->sbdf, PCI_COMMAND) & PCI_COMMAND_MEMORY", and added comments why it needs this check. * Added "!is_hardware_domain(pdev->domain)" check in init_rebar() to return EOPNOTSUPP for domUs. * Moved BAR type and index check into init_rebar(), then only need to check once. * Added 'U' suffix for macro PCI_REBAR_CAP_SIZES. * Added macro PCI_REBAR_SIZE_BIAS to represent 20. TODO: need to hide ReBar capability from hardware domain when init_rebar() fails. v1->v2 changes: * In rebar_ctrl_write, to check if memory decoding is enabled, and added some checks for the type of Bar. * Added vpci_hw_write32 to handle PCI_REBAR_CAP's write, since there is no write limitation of dom0. * And has many other minor modifications as well. --- xen/drivers/vpci/Makefile | 2 +- xen/drivers/vpci/rebar.c | 131 +++++++++++++++++++++++++++++++++++++ xen/include/xen/pci_regs.h | 15 +++++ xen/include/xen/vpci.h | 1 + 4 files changed, 148 insertions(+), 1 deletion(-) create mode 100644 xen/drivers/vpci/rebar.c