| Message ID | 20190522201252.2997-1-logang@deltatee.com (mailing list archive) |
|---|---|
| State | Superseded, archived |
| Headers | show |
| Series | PCI/P2PDMA: Root complex whitelist should not apply when an IOMMU is present | expand |
On Wed, May 22, 2019 at 02:12:52PM -0600, Logan Gunthorpe wrote: > Presently, there is no path to DMA map P2PDMA memory, so if a TLP > targeting this memory hits the root complex and an IOMMU is present, > the IOMMU will reject the transaction, even if the RC would support > P2PDMA. > > So until the kernel knows to map these DMA addresses in the IOMMU, > we should not enable the whitelist when an IOMMU is present. > > While we are at it, remove the comment mentioning future work > to add a white list. There was a lot of discussion about this. Did everybody come to a consensus about what should be done? Can you post a patch with reviewed-by if appropriate? > Fixes: 0f97da831026 ("PCI/P2PDMA: Allow P2P DMA between any devices under AMD ZEN Root Complex") > Signed-off-by: Logan Gunthorpe <logang@deltatee.com> > Cc: Christian König <christian.koenig@amd.com> > Cc: Bjorn Helgaas <bhelgaas@google.com> > --- > drivers/pci/p2pdma.c | 7 +++++-- > 1 file changed, 5 insertions(+), 2 deletions(-) > > Hey, > > I realized recently that I missed this issue between the IOMMU and > the whitelist when reviewing Christian's patch. > > Unless there are any objections, I think this should be squashed > with the commit marked in the Fixes tag (from pci-v5.2-changes). > > Thanks, > > Logan > > diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c > index 742928d0053e..4d2f6a44cba3 100644 > --- a/drivers/pci/p2pdma.c > +++ b/drivers/pci/p2pdma.c > @@ -18,6 +18,7 @@ > #include <linux/percpu-refcount.h> > #include <linux/random.h> > #include <linux/seq_buf.h> > +#include <linux/iommu.h> > > struct pci_p2pdma { > struct percpu_ref devmap_ref; > @@ -284,6 +285,9 @@ static bool root_complex_whitelist(struct pci_dev *dev) > struct pci_dev *root = pci_get_slot(host->bus, PCI_DEVFN(0, 0)); > unsigned short vendor, device; > > + if (iommu_present(dev->dev.bus)) > + return false; > + > if (!root) > return false; > > @@ -453,8 +457,7 @@ static int upstream_bridge_distance_warn(struct pci_dev *provider, > * > * For now, "compatible" means the provider and the clients are all behind > * the same PCI root port. This cuts out cases that may work but is safest > - * for the user. Future work can expand this to white-list root complexes that > - * can safely forward between each ports. > + * for the user. > */ > int pci_p2pdma_distance_many(struct pci_dev *provider, struct device **clients, > int num_clients, bool verbose) > -- > 2.20.1
On 2019-06-18 2:40 p.m., Bjorn Helgaas wrote: > On Wed, May 22, 2019 at 02:12:52PM -0600, Logan Gunthorpe wrote: >> Presently, there is no path to DMA map P2PDMA memory, so if a TLP >> targeting this memory hits the root complex and an IOMMU is present, >> the IOMMU will reject the transaction, even if the RC would support >> P2PDMA. >> >> So until the kernel knows to map these DMA addresses in the IOMMU, >> we should not enable the whitelist when an IOMMU is present. >> >> While we are at it, remove the comment mentioning future work >> to add a white list. > > There was a lot of discussion about this. Did everybody come to a > consensus about what should be done? Can you post a patch with > reviewed-by if appropriate? I think we have consensus that it's broken and needs to be fixed for the short term. Preferably before 5.3. I'm not sure we have consensus on the proper fix. The two easy things I can see to do is to either revert it or add the iommu_is_present() check that I did in the above patch. @Christian, which do you prefer? I think I'd prefer the iommu_is_present() route as it maintains the information about white-listed devices and is easier to change once we have the correct solution. I can send a patch tomorrow one way or another. Thanks, Logan
On 2019-06-18 2:51 p.m., Logan Gunthorpe wrote: > On 2019-06-18 2:40 p.m., Bjorn Helgaas wrote: >> On Wed, May 22, 2019 at 02:12:52PM -0600, Logan Gunthorpe wrote: >>> Presently, there is no path to DMA map P2PDMA memory, so if a TLP >>> targeting this memory hits the root complex and an IOMMU is present, >>> the IOMMU will reject the transaction, even if the RC would support >>> P2PDMA. >>> >>> So until the kernel knows to map these DMA addresses in the IOMMU, >>> we should not enable the whitelist when an IOMMU is present. >>> >>> While we are at it, remove the comment mentioning future work >>> to add a white list. >> >> There was a lot of discussion about this. Did everybody come to a >> consensus about what should be done? Can you post a patch with >> reviewed-by if appropriate? > > I think we have consensus that it's broken and needs to be fixed for the > short term. Preferably before 5.3. I'm not sure we have consensus on the > proper fix. > > The two easy things I can see to do is to either revert it or add the > iommu_is_present() check that I did in the above patch. > > @Christian, which do you prefer? I think I'd prefer the > iommu_is_present() route as it maintains the information about > white-listed devices and is easier to change once we have the correct > solution. > > I can send a patch tomorrow one way or another. Also, looks like one of my clients has an interest in seeing work like this happen. So I'll be writing some patches in the next couple weeks to do this properly. I'll try to send them to the lists early next cycle. Logan
Am 19.06.19 um 01:50 schrieb Logan Gunthorpe: > > On 2019-06-18 2:51 p.m., Logan Gunthorpe wrote: >> On 2019-06-18 2:40 p.m., Bjorn Helgaas wrote: >>> On Wed, May 22, 2019 at 02:12:52PM -0600, Logan Gunthorpe wrote: >>>> Presently, there is no path to DMA map P2PDMA memory, so if a TLP >>>> targeting this memory hits the root complex and an IOMMU is present, >>>> the IOMMU will reject the transaction, even if the RC would support >>>> P2PDMA. >>>> >>>> So until the kernel knows to map these DMA addresses in the IOMMU, >>>> we should not enable the whitelist when an IOMMU is present. >>>> >>>> While we are at it, remove the comment mentioning future work >>>> to add a white list. >>> There was a lot of discussion about this. Did everybody come to a >>> consensus about what should be done? Can you post a patch with >>> reviewed-by if appropriate? >> I think we have consensus that it's broken and needs to be fixed for the >> short term. Preferably before 5.3. Yeah, completely agree. >> I'm not sure we have consensus on the >> proper fix. >> >> The two easy things I can see to do is to either revert it or add the >> iommu_is_present() check that I did in the above patch. >> >> @Christian, which do you prefer? I think I'd prefer the >> iommu_is_present() route as it maintains the information about >> white-listed devices and is easier to change once we have the correct >> solution. Your original iommu_is_prevent() patch sound like the best option so far. If that hasn't changed feel free to add a Reviewed-by: Christian König <christian.koenig@amd.com> to that one. >> I can send a patch tomorrow one way or another. > Also, looks like one of my clients has an interest in seeing work like > this happen. So I'll be writing some patches in the next couple weeks to > do this properly. I'll try to send them to the lists early next cycle. Oh, nice. I was hoping to get my use case into 5.4 or 5.5, but we are still stuck with some of the DMA-buf related pieces. Regards, Christian. > > Logan
On Wed, Jun 19, 2019 at 09:26:42AM +0000, Koenig, Christian wrote: > I was hoping to get my use case into 5.4 or 5.5, but we are still stuck > with some of the DMA-buf related pieces. Can you make sure you have Logand and me, and maybe the iommu list Cced when you posted it? I'd like to make sure we all have a common understanding where we are moving with the APIs and use cases.
Am 19.06.19 um 11:29 schrieb Christoph Hellwig: > On Wed, Jun 19, 2019 at 09:26:42AM +0000, Koenig, Christian wrote: >> I was hoping to get my use case into 5.4 or 5.5, but we are still stuck >> with some of the DMA-buf related pieces. > Can you make sure you have Logand and me, and maybe the iommu list > Cced when you posted it? I'd like to make sure we all have a common > understanding where we are moving with the APIs and use cases. Yeah, sure. I'm still not settled on APIs anyway, so feedback on this is highly welcome. Regards, Christian.
diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 742928d0053e..4d2f6a44cba3 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -18,6 +18,7 @@ #include <linux/percpu-refcount.h> #include <linux/random.h> #include <linux/seq_buf.h> +#include <linux/iommu.h> struct pci_p2pdma { struct percpu_ref devmap_ref; @@ -284,6 +285,9 @@ static bool root_complex_whitelist(struct pci_dev *dev) struct pci_dev *root = pci_get_slot(host->bus, PCI_DEVFN(0, 0)); unsigned short vendor, device; + if (iommu_present(dev->dev.bus)) + return false; + if (!root) return false; @@ -453,8 +457,7 @@ static int upstream_bridge_distance_warn(struct pci_dev *provider, * * For now, "compatible" means the provider and the clients are all behind * the same PCI root port. This cuts out cases that may work but is safest - * for the user. Future work can expand this to white-list root complexes that - * can safely forward between each ports. + * for the user. */ int pci_p2pdma_distance_many(struct pci_dev *provider, struct device **clients, int num_clients, bool verbose)
Presently, there is no path to DMA map P2PDMA memory, so if a TLP targeting this memory hits the root complex and an IOMMU is present, the IOMMU will reject the transaction, even if the RC would support P2PDMA. So until the kernel knows to map these DMA addresses in the IOMMU, we should not enable the whitelist when an IOMMU is present. While we are at it, remove the comment mentioning future work to add a white list. Fixes: 0f97da831026 ("PCI/P2PDMA: Allow P2P DMA between any devices under AMD ZEN Root Complex") Signed-off-by: Logan Gunthorpe <logang@deltatee.com> Cc: Christian König <christian.koenig@amd.com> Cc: Bjorn Helgaas <bhelgaas@google.com> --- drivers/pci/p2pdma.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) Hey, I realized recently that I missed this issue between the IOMMU and the whitelist when reviewing Christian's patch. Unless there are any objections, I think this should be squashed with the commit marked in the Fixes tag (from pci-v5.2-changes). Thanks, Logan -- 2.20.1