| Message ID | 20190916135435.5017-3-jonathan.derrick@intel.com (mailing list archive) |
|---|---|
| State | Mainlined, archived |
| Commit | a1a30170138c9c5157bd514ccd4d76b47060f29b |
| Headers | show |
| Series | VMD fixes for v5.4 | expand |
On Mon, Sep 16, 2019 at 07:54:35AM -0600, Jon Derrick wrote: > The shadow offset scratchpad was moved to 0x2000-0x2010. Update the > location to get the correct shadow offset. Hi Jon, what does "was moved" mean ? Would this code still work on previous HW ? We must make sure that the address move is managed seamlessly by the kernel. For the time being I have to drop this fix and it is extremely tight to get it in v5.4, I can't send stable changes that we may have to revert. Thanks, Lorenzo > Cc: stable@vger.kernel.org # v5.2+ > Fixes: 6788958e ("PCI: vmd: Assign membar addresses from shadow registers") > Signed-off-by: Jon Derrick <jonathan.derrick@intel.com> > --- > drivers/pci/controller/vmd.c | 9 ++++++--- > 1 file changed, 6 insertions(+), 3 deletions(-) > > diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c > index 2e4da3f51d6b..a35d3f3996d7 100644 > --- a/drivers/pci/controller/vmd.c > +++ b/drivers/pci/controller/vmd.c > @@ -31,6 +31,9 @@ > #define PCI_REG_VMLOCK 0x70 > #define MB2_SHADOW_EN(vmlock) (vmlock & 0x2) > > +#define MB2_SHADOW_OFFSET 0x2000 > +#define MB2_SHADOW_SIZE 16 > + > enum vmd_features { > /* > * Device may contain registers which hint the physical location of the > @@ -578,7 +581,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) > u32 vmlock; > int ret; > > - membar2_offset = 0x2018; > + membar2_offset = MB2_SHADOW_OFFSET + MB2_SHADOW_SIZE; > ret = pci_read_config_dword(vmd->dev, PCI_REG_VMLOCK, &vmlock); > if (ret || vmlock == ~0) > return -ENODEV; > @@ -590,9 +593,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) > if (!membar2) > return -ENOMEM; > offset[0] = vmd->dev->resource[VMD_MEMBAR1].start - > - readq(membar2 + 0x2008); > + readq(membar2 + MB2_SHADOW_OFFSET); > offset[1] = vmd->dev->resource[VMD_MEMBAR2].start - > - readq(membar2 + 0x2010); > + readq(membar2 + MB2_SHADOW_OFFSET + 8); > pci_iounmap(vmd->dev, membar2); > } > } > -- > 2.20.1 >
On Tue, 2019-09-17 at 11:41 +0100, Lorenzo Pieralisi wrote: > On Mon, Sep 16, 2019 at 07:54:35AM -0600, Jon Derrick wrote: > > The shadow offset scratchpad was moved to 0x2000-0x2010. Update the > > location to get the correct shadow offset. > > Hi Jon, > > what does "was moved" mean ? Would this code still work on previous HW ? > The previous code won't work on (not yet released) hw. Guests using the domain will see the wrong offset and enumerate the domain incorrectly. > We must make sure that the address move is managed seamlessly by the > kernel. If we need to avoid changing addressing within stable, then that's fine. But otherwise is there an issue merging with 5.4? > > For the time being I have to drop this fix and it is extremely > tight to get it in v5.4, I can't send stable changes that we may > have to revert. Aren't we in the beginning of the merge window? > > Thanks, > Lorenzo > > > Cc: stable@vger.kernel.org # v5.2+ > > Fixes: 6788958e ("PCI: vmd: Assign membar addresses from shadow registers") > > Signed-off-by: Jon Derrick <jonathan.derrick@intel.com> > > --- > > drivers/pci/controller/vmd.c | 9 ++++++--- > > 1 file changed, 6 insertions(+), 3 deletions(-) > > > > diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c > > index 2e4da3f51d6b..a35d3f3996d7 100644 > > --- a/drivers/pci/controller/vmd.c > > +++ b/drivers/pci/controller/vmd.c > > @@ -31,6 +31,9 @@ > > #define PCI_REG_VMLOCK 0x70 > > #define MB2_SHADOW_EN(vmlock) (vmlock & 0x2) > > > > +#define MB2_SHADOW_OFFSET 0x2000 > > +#define MB2_SHADOW_SIZE 16 > > + > > enum vmd_features { > > /* > > * Device may contain registers which hint the physical location of the > > @@ -578,7 +581,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) > > u32 vmlock; > > int ret; > > > > - membar2_offset = 0x2018; > > + membar2_offset = MB2_SHADOW_OFFSET + MB2_SHADOW_SIZE; > > ret = pci_read_config_dword(vmd->dev, PCI_REG_VMLOCK, &vmlock); > > if (ret || vmlock == ~0) > > return -ENODEV; > > @@ -590,9 +593,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) > > if (!membar2) > > return -ENOMEM; > > offset[0] = vmd->dev->resource[VMD_MEMBAR1].start - > > - readq(membar2 + 0x2008); > > + readq(membar2 + MB2_SHADOW_OFFSET); > > offset[1] = vmd->dev->resource[VMD_MEMBAR2].start - > > - readq(membar2 + 0x2010); > > + readq(membar2 + MB2_SHADOW_OFFSET + 8); > > pci_iounmap(vmd->dev, membar2); > > } > > } > > -- > > 2.20.1 > >
On Tue, Sep 17, 2019 at 01:55:59PM +0000, Derrick, Jonathan wrote: > On Tue, 2019-09-17 at 11:41 +0100, Lorenzo Pieralisi wrote: > > On Mon, Sep 16, 2019 at 07:54:35AM -0600, Jon Derrick wrote: > > > The shadow offset scratchpad was moved to 0x2000-0x2010. Update the > > > location to get the correct shadow offset. > > > > Hi Jon, > > > > what does "was moved" mean ? Would this code still work on previous HW ? > > > The previous code won't work on (not yet released) hw. Guests using the > domain will see the wrong offset and enumerate the domain incorrectly. That's true also for new kernels on _current_ hardware, isn't it ? What I am saying is that there must be a way to detect the right offset from HW probing or firmware otherwise things will break one way of another. > > We must make sure that the address move is managed seamlessly by the > > kernel. > If we need to avoid changing addressing within stable, then that's > fine. But otherwise is there an issue merging with 5.4? See above. Would 5.4 with this patch applied work on systems where the offset is the same as the _current_ one without this patch applied ? > > For the time being I have to drop this fix and it is extremely > > tight to get it in v5.4, I can't send stable changes that we may > > have to revert. > Aren't we in the beginning of the merge window? Yes and that's the problem, patches for v5.4 should have already being queued a while ago, I do not know when Bjorn will send the PR for v5.4 but that's going to happen shortly, I am making an exception to squeeze these patches in since it is vmd only code but still it is very very tight. Thanks, Lorenzo > > Thanks, > > Lorenzo > > > > > Cc: stable@vger.kernel.org # v5.2+ > > > Fixes: 6788958e ("PCI: vmd: Assign membar addresses from shadow registers") > > > Signed-off-by: Jon Derrick <jonathan.derrick@intel.com> > > > --- > > > drivers/pci/controller/vmd.c | 9 ++++++--- > > > 1 file changed, 6 insertions(+), 3 deletions(-) > > > > > > diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c > > > index 2e4da3f51d6b..a35d3f3996d7 100644 > > > --- a/drivers/pci/controller/vmd.c > > > +++ b/drivers/pci/controller/vmd.c > > > @@ -31,6 +31,9 @@ > > > #define PCI_REG_VMLOCK 0x70 > > > #define MB2_SHADOW_EN(vmlock) (vmlock & 0x2) > > > > > > +#define MB2_SHADOW_OFFSET 0x2000 > > > +#define MB2_SHADOW_SIZE 16 > > > + > > > enum vmd_features { > > > /* > > > * Device may contain registers which hint the physical location of the > > > @@ -578,7 +581,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) > > > u32 vmlock; > > > int ret; > > > > > > - membar2_offset = 0x2018; > > > + membar2_offset = MB2_SHADOW_OFFSET + MB2_SHADOW_SIZE; > > > ret = pci_read_config_dword(vmd->dev, PCI_REG_VMLOCK, &vmlock); > > > if (ret || vmlock == ~0) > > > return -ENODEV; > > > @@ -590,9 +593,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) > > > if (!membar2) > > > return -ENOMEM; > > > offset[0] = vmd->dev->resource[VMD_MEMBAR1].start - > > > - readq(membar2 + 0x2008); > > > + readq(membar2 + MB2_SHADOW_OFFSET); > > > offset[1] = vmd->dev->resource[VMD_MEMBAR2].start - > > > - readq(membar2 + 0x2010); > > > + readq(membar2 + MB2_SHADOW_OFFSET + 8); > > > pci_iounmap(vmd->dev, membar2); > > > } > > > } > > > -- > > > 2.20.1 > > >
On Tue, 2019-09-17 at 15:05 +0100, Lorenzo Pieralisi wrote: > On Tue, Sep 17, 2019 at 01:55:59PM +0000, Derrick, Jonathan wrote: > > On Tue, 2019-09-17 at 11:41 +0100, Lorenzo Pieralisi wrote: > > > On Mon, Sep 16, 2019 at 07:54:35AM -0600, Jon Derrick wrote: > > > > The shadow offset scratchpad was moved to 0x2000-0x2010. Update the > > > > location to get the correct shadow offset. > > > > > > Hi Jon, > > > > > > what does "was moved" mean ? Would this code still work on previous HW ? > > > > > The previous code won't work on (not yet released) hw. Guests using the > > domain will see the wrong offset and enumerate the domain incorrectly. > > That's true also for new kernels on _current_ hardware, isn't it ? > > What I am saying is that there must be a way to detect the right > offset from HW probing or firmware otherwise things will break > one way of another. > I think this is basically that, but the spec changed which register addresses contained the offset. The offset was still discoverable either way, but is now within 0x2000-0x2010, with 0x2010-0x2090 as oob interface. > > > We must make sure that the address move is managed seamlessly by the > > > kernel. > > If we need to avoid changing addressing within stable, then that's > > fine. But otherwise is there an issue merging with 5.4? > > See above. Would 5.4 with this patch applied work on systems where > the offset is the same as the _current_ one without this patch > applied ? I understand your concern, but these systems with wrong addressing won't exist because the hardware isn't released yet. In the future, the hardware will be released and users will inevitably load some unfixed kernel, and we would like to point to stable for the fix. > > > > For the time being I have to drop this fix and it is extremely > > > tight to get it in v5.4, I can't send stable changes that we may > > > have to revert. > > Aren't we in the beginning of the merge window? > > Yes and that's the problem, patches for v5.4 should have already > being queued a while ago, I do not know when Bjorn will send the > PR for v5.4 but that's going to happen shortly, I am making an > exception to squeeze these patches in since it is vmd only code > but still it is very very tight. > If you feel there's a risk, then I think it can be staged for v5.5. Hardware will not be available for some time. Best regards, Jon > Thanks, > Lorenzo > > > > Thanks, > > > Lorenzo > > > > > > > Cc: stable@vger.kernel.org # v5.2+ > > > > Fixes: 6788958e ("PCI: vmd: Assign membar addresses from shadow registers") > > > > Signed-off-by: Jon Derrick <jonathan.derrick@intel.com> > > > > --- > > > > drivers/pci/controller/vmd.c | 9 ++++++--- > > > > 1 file changed, 6 insertions(+), 3 deletions(-) > > > > > > > > diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c > > > > index 2e4da3f51d6b..a35d3f3996d7 100644 > > > > --- a/drivers/pci/controller/vmd.c > > > > +++ b/drivers/pci/controller/vmd.c > > > > @@ -31,6 +31,9 @@ > > > > #define PCI_REG_VMLOCK 0x70 > > > > #define MB2_SHADOW_EN(vmlock) (vmlock & 0x2) > > > > > > > > +#define MB2_SHADOW_OFFSET 0x2000 > > > > +#define MB2_SHADOW_SIZE 16 > > > > + > > > > enum vmd_features { > > > > /* > > > > * Device may contain registers which hint the physical location of the > > > > @@ -578,7 +581,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) > > > > u32 vmlock; > > > > int ret; > > > > > > > > - membar2_offset = 0x2018; > > > > + membar2_offset = MB2_SHADOW_OFFSET + MB2_SHADOW_SIZE; > > > > ret = pci_read_config_dword(vmd->dev, PCI_REG_VMLOCK, &vmlock); > > > > if (ret || vmlock == ~0) > > > > return -ENODEV; > > > > @@ -590,9 +593,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) > > > > if (!membar2) > > > > return -ENOMEM; > > > > offset[0] = vmd->dev->resource[VMD_MEMBAR1].start - > > > > - readq(membar2 + 0x2008); > > > > + readq(membar2 + MB2_SHADOW_OFFSET); > > > > offset[1] = vmd->dev->resource[VMD_MEMBAR2].start - > > > > - readq(membar2 + 0x2010); > > > > + readq(membar2 + MB2_SHADOW_OFFSET + 8); > > > > pci_iounmap(vmd->dev, membar2); > > > > } > > > > } > > > > -- > > > > 2.20.1 > > > >
On Tue, Sep 17, 2019 at 02:45:03PM +0000, Derrick, Jonathan wrote: > On Tue, 2019-09-17 at 15:05 +0100, Lorenzo Pieralisi wrote: > > On Tue, Sep 17, 2019 at 01:55:59PM +0000, Derrick, Jonathan wrote: > > > On Tue, 2019-09-17 at 11:41 +0100, Lorenzo Pieralisi wrote: > > > > On Mon, Sep 16, 2019 at 07:54:35AM -0600, Jon Derrick wrote: > > > > > The shadow offset scratchpad was moved to 0x2000-0x2010. Update the > > > > > location to get the correct shadow offset. > > > > > > > > Hi Jon, > > > > > > > > what does "was moved" mean ? Would this code still work on previous HW ? > > > > > > > The previous code won't work on (not yet released) hw. Guests using the > > > domain will see the wrong offset and enumerate the domain incorrectly. > > > > That's true also for new kernels on _current_ hardware, isn't it ? > > > > What I am saying is that there must be a way to detect the right > > offset from HW probing or firmware otherwise things will break > > one way of another. > > > I think this is basically that, but the spec changed which register > addresses contained the offset. The offset was still discoverable > either way, but is now within 0x2000-0x2010, with 0x2010-0x2090 as oob > interface. > > > > > > > We must make sure that the address move is managed seamlessly by the > > > > kernel. > > > If we need to avoid changing addressing within stable, then that's > > > fine. But otherwise is there an issue merging with 5.4? > > > > See above. Would 5.4 with this patch applied work on systems where > > the offset is the same as the _current_ one without this patch > > applied ? > I understand your concern, but these systems with wrong addressing > won't exist because the hardware isn't released yet. > > In the future, the hardware will be released and users will inevitably > load some unfixed kernel, and we would like to point to stable for the > fix. I am sorry for being blunt but I need to understand. If we apply this patch, are you telling me that the _current_ HW would fail ? I assume the current HW+kernel set-up is working, maybe that's what I got wrong. Reworded: on existing HW, is this patch fixing anything ? This patch when it hits the mainline will trickle into stable kernel unchanged. > > > > For the time being I have to drop this fix and it is extremely > > > > tight to get it in v5.4, I can't send stable changes that we may > > > > have to revert. > > > Aren't we in the beginning of the merge window? > > > > Yes and that's the problem, patches for v5.4 should have already > > being queued a while ago, I do not know when Bjorn will send the > > PR for v5.4 but that's going to happen shortly, I am making an > > exception to squeeze these patches in since it is vmd only code > > but still it is very very tight. > > > If you feel there's a risk, then I think it can be staged for v5.5. > Hardware will not be available for some time. I do not feel it is risky, I feel it would be much better if the scratchpad address could be detected at runtime through versioning of sorts either HW or firmware based. If we can't probe it inevitably we will have systems where kernels would break and that's something we should avoid. Lorenzo
On Tue, 2019-09-17 at 16:15 +0100, Lorenzo Pieralisi wrote: > On Tue, Sep 17, 2019 at 02:45:03PM +0000, Derrick, Jonathan wrote: > > On Tue, 2019-09-17 at 15:05 +0100, Lorenzo Pieralisi wrote: > > > On Tue, Sep 17, 2019 at 01:55:59PM +0000, Derrick, Jonathan wrote: > > > > On Tue, 2019-09-17 at 11:41 +0100, Lorenzo Pieralisi wrote: > > > > > On Mon, Sep 16, 2019 at 07:54:35AM -0600, Jon Derrick wrote: > > > > > > The shadow offset scratchpad was moved to 0x2000-0x2010. Update the > > > > > > location to get the correct shadow offset. > > > > > > > > > > Hi Jon, > > > > > > > > > > what does "was moved" mean ? Would this code still work on previous HW ? > > > > > > > > > The previous code won't work on (not yet released) hw. Guests using the > > > > domain will see the wrong offset and enumerate the domain incorrectly. > > > > > > That's true also for new kernels on _current_ hardware, isn't it ? > > > > > > What I am saying is that there must be a way to detect the right > > > offset from HW probing or firmware otherwise things will break > > > one way of another. > > > > > I think this is basically that, but the spec changed which register > > addresses contained the offset. The offset was still discoverable > > either way, but is now within 0x2000-0x2010, with 0x2010-0x2090 as oob > > interface. > > > > > > > > > > > We must make sure that the address move is managed seamlessly by the > > > > > kernel. > > > > If we need to avoid changing addressing within stable, then that's > > > > fine. But otherwise is there an issue merging with 5.4? > > > > > > See above. Would 5.4 with this patch applied work on systems where > > > the offset is the same as the _current_ one without this patch > > > applied ? > > I understand your concern, but these systems with wrong addressing > > won't exist because the hardware isn't released yet. > > > > In the future, the hardware will be released and users will inevitably > > load some unfixed kernel, and we would like to point to stable for the > > fix. > > I am sorry for being blunt but I need to understand. If we apply > this patch, are you telling me that the _current_ HW would fail ? > > I assume the current HW+kernel set-up is working, maybe that's > what I got wrong. > > Reworded: on existing HW, is this patch fixing anything ? > > This patch when it hits the mainline will trickle into stable > kernel unchanged. Sorry for the confusion. These changes only affect systems with VMD devices with 8086:28C0 device IDs, but these won't be production hardware for some time. Systems with VMD devices exist in the wild with 8086:201D device IDs. These don't support the guest passthrough mode and this code won't break anything with them. Additionally, patch 1/2 (bus numbering) only affects 8086:28C0. So on existing HW, these patches won't affect anything > > > > > > For the time being I have to drop this fix and it is extremely > > > > > tight to get it in v5.4, I can't send stable changes that we may > > > > > have to revert. > > > > Aren't we in the beginning of the merge window? > > > > > > Yes and that's the problem, patches for v5.4 should have already > > > being queued a while ago, I do not know when Bjorn will send the > > > PR for v5.4 but that's going to happen shortly, I am making an > > > exception to squeeze these patches in since it is vmd only code > > > but still it is very very tight. > > > > > If you feel there's a risk, then I think it can be staged for v5.5. > > Hardware will not be available for some time. > > I do not feel it is risky, I feel it would be much better if the > scratchpad address could be detected at runtime through versioning > of sorts either HW or firmware based. > > If we can't probe it inevitably we will have systems where kernels > would break and that's something we should avoid. > I agree that it might have been nicer if it were an ACPI/EFI var, but I think there were some complexities with teaching hypervisors to expose it to the guests for use when enumerating the domain from the passed- through endpoint. The method that exists in 8086:28C0 hardware divorces the firmware descriptors from the device so that the guest driver only needs to read the host-to-guest physical offset from the device itself. Best regards, Jon > Lorenzo
On Tue, Sep 17, 2019 at 03:51:39PM +0000, Derrick, Jonathan wrote: [...] > Sorry for the confusion. > > These changes only affect systems with VMD devices with 8086:28C0 > device IDs, but these won't be production hardware for some time. > > Systems with VMD devices exist in the wild with 8086:201D device IDs. > These don't support the guest passthrough mode and this code won't > break anything with them. Additionally, patch 1/2 (bus numbering) only > affects 8086:28C0. > > So on existing HW, these patches won't affect anything It is me who created confusion, apologies. I read the code properly and I understand that both patches are fixes for HW that is still not available (and they can't create an issue with current kernel because HAS_MEMBAR_SHADOW and HAS_BUS_RESTRICTIONS features are not implemented on 8086:201D), we should take these patches and trickle them to stable kernels as soon as possible so that when HW _is_ available mainline and stable kernels are fixed. Correct ? Lorenzo
On Tue, 2019-09-17 at 17:37 +0100, Lorenzo Pieralisi wrote: > On Tue, Sep 17, 2019 at 03:51:39PM +0000, Derrick, Jonathan wrote: > > [...] > > > Sorry for the confusion. > > > > These changes only affect systems with VMD devices with 8086:28C0 > > device IDs, but these won't be production hardware for some time. > > > > Systems with VMD devices exist in the wild with 8086:201D device IDs. > > These don't support the guest passthrough mode and this code won't > > break anything with them. Additionally, patch 1/2 (bus numbering) only > > affects 8086:28C0. > > > > So on existing HW, these patches won't affect anything > > It is me who created confusion, apologies. I read the code properly and > I understand that both patches are fixes for HW that is still not > available (and they can't create an issue with current kernel because > HAS_MEMBAR_SHADOW and HAS_BUS_RESTRICTIONS features are not implemented > on 8086:201D), we should take these patches and trickle them to stable > kernels as soon as possible so that when HW _is_ available mainline and > stable kernels are fixed. > > Correct ? > > Lorenzo That's correct. It will apply to 5.2 stable but is missing a few deps for 4.19 that I wouldn't consider as qualifying as stable. I can backport to 4.19 fairly easily. Thanks, Jon
diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index 2e4da3f51d6b..a35d3f3996d7 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -31,6 +31,9 @@ #define PCI_REG_VMLOCK 0x70 #define MB2_SHADOW_EN(vmlock) (vmlock & 0x2) +#define MB2_SHADOW_OFFSET 0x2000 +#define MB2_SHADOW_SIZE 16 + enum vmd_features { /* * Device may contain registers which hint the physical location of the @@ -578,7 +581,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) u32 vmlock; int ret; - membar2_offset = 0x2018; + membar2_offset = MB2_SHADOW_OFFSET + MB2_SHADOW_SIZE; ret = pci_read_config_dword(vmd->dev, PCI_REG_VMLOCK, &vmlock); if (ret || vmlock == ~0) return -ENODEV; @@ -590,9 +593,9 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) if (!membar2) return -ENOMEM; offset[0] = vmd->dev->resource[VMD_MEMBAR1].start - - readq(membar2 + 0x2008); + readq(membar2 + MB2_SHADOW_OFFSET); offset[1] = vmd->dev->resource[VMD_MEMBAR2].start - - readq(membar2 + 0x2010); + readq(membar2 + MB2_SHADOW_OFFSET + 8); pci_iounmap(vmd->dev, membar2); } }
The shadow offset scratchpad was moved to 0x2000-0x2010. Update the location to get the correct shadow offset. Cc: stable@vger.kernel.org # v5.2+ Fixes: 6788958e ("PCI: vmd: Assign membar addresses from shadow registers") Signed-off-by: Jon Derrick <jonathan.derrick@intel.com> --- drivers/pci/controller/vmd.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)