diff mbox series

[2/2] vfio/pci: Remove console drivers

Message ID 165453800875.3592816.12944011921352366695.stgit@omen (mailing list archive)
State New, archived
Headers show
Series Improve vfio-pci primary GPU assignment behavior | expand

Commit Message

Alex Williamson June 6, 2022, 5:53 p.m. UTC
Console drivers can create conflicts with PCI resources resulting in
userspace getting mmap failures to memory BARs.  This is especially evident
when trying to re-use the system primary console for userspace drivers.
Attempt to remove all nature of conflicting drivers as part of our VGA
initialization.

Reported-by: Laszlo Ersek <lersek@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Suggested-by: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
---
 drivers/vfio/pci/vfio_pci_core.c |   17 +++++++++++++++++
 1 file changed, 17 insertions(+)

Comments

Thomas Zimmermann June 8, 2022, 11:11 a.m. UTC | #1
Hi Alex

Am 06.06.22 um 19:53 schrieb Alex Williamson:
> Console drivers can create conflicts with PCI resources resulting in
> userspace getting mmap failures to memory BARs.  This is especially evident
> when trying to re-use the system primary console for userspace drivers.
> Attempt to remove all nature of conflicting drivers as part of our VGA
> initialization.

First a dumb question about your use case.  You want to assign a PCI 
graphics card to a virtual machine and need to remove the generic driver 
from the framebuffer?

> 
> Reported-by: Laszlo Ersek <lersek@redhat.com>
> Tested-by: Laszlo Ersek <lersek@redhat.com>
> Suggested-by: Gerd Hoffmann <kraxel@redhat.com>
> Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> ---
>   drivers/vfio/pci/vfio_pci_core.c |   17 +++++++++++++++++
>   1 file changed, 17 insertions(+)
> 
> diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
> index a0d69ddaf90d..e0cbcbc2aee1 100644
> --- a/drivers/vfio/pci/vfio_pci_core.c
> +++ b/drivers/vfio/pci/vfio_pci_core.c
> @@ -13,6 +13,7 @@
>   #include <linux/device.h>
>   #include <linux/eventfd.h>
>   #include <linux/file.h>
> +#include <linux/fb.h>
>   #include <linux/interrupt.h>
>   #include <linux/iommu.h>
>   #include <linux/module.h>
> @@ -29,6 +30,8 @@
>   
>   #include <linux/vfio_pci_core.h>
>   
> +#include <drm/drm_aperture.h>
> +
>   #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
>   #define DRIVER_DESC "core driver for VFIO based PCI devices"
>   
> @@ -1793,6 +1796,20 @@ static int vfio_pci_vga_init(struct vfio_pci_core_device *vdev)
>   	if (!vfio_pci_is_vga(pdev))
>   		return 0;
>   
> +#if IS_REACHABLE(CONFIG_DRM)
> +	drm_aperture_detach_platform_drivers(pdev);
> +#endif
> +
> +#if IS_REACHABLE(CONFIG_FB)
> +	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
> +	if (ret)
> +		return ret;
> +#endif
> +
> +	ret = vga_remove_vgacon(pdev);
> +	if (ret)
> +		return ret;
> +

You shouldn't have to copy any of the implementation of the aperture 
helpers.

If you call drm_aperture_remove_conflicting_pci_framebuffers() it should 
work correctly. The only reason why it requires a DRM driver structure 
as second argument is for the driver's name. [1] And that name is only 
used for printing an info message. [2]

The plan forward would be to drop patch 1 entirely.

For patch 2, the most trivial workaround is to instanciate struct 
drm_driver here and set the name field to 'vdev->vdev.ops->name'. In the 
longer term, the aperture helpers will be moved out of DRM and into a 
more prominent location. That workaround will be cleaned up then.

Alternatively, drm_aperture_remove_conflicting_pci_framebuffers() could 
be changed to accept the name string as second argument, but that's 
quite a bit of churn within the DRM code.

Best regards
Thomas

[1] 
https://elixir.bootlin.com/linux/v5.18.2/source/drivers/gpu/drm/drm_aperture.c#L347
[2] 
https://elixir.bootlin.com/linux/v5.18.2/source/drivers/video/fbdev/core/fbmem.c#L1570


>   	ret = vga_client_register(pdev, vfio_pci_set_decode);
>   	if (ret)
>   		return ret;
> 
>
Alex Williamson June 8, 2022, 2:04 p.m. UTC | #2
Hi Thomas,

On Wed, 8 Jun 2022 13:11:21 +0200
Thomas Zimmermann <tzimmermann@suse.de> wrote:

> Hi Alex
> 
> Am 06.06.22 um 19:53 schrieb Alex Williamson:
> > Console drivers can create conflicts with PCI resources resulting in
> > userspace getting mmap failures to memory BARs.  This is especially evident
> > when trying to re-use the system primary console for userspace drivers.
> > Attempt to remove all nature of conflicting drivers as part of our VGA
> > initialization.  
> 
> First a dumb question about your use case.  You want to assign a PCI 
> graphics card to a virtual machine and need to remove the generic driver 
> from the framebuffer?

Exactly.
 
> > Reported-by: Laszlo Ersek <lersek@redhat.com>
> > Tested-by: Laszlo Ersek <lersek@redhat.com>
> > Suggested-by: Gerd Hoffmann <kraxel@redhat.com>
> > Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> > ---
> >   drivers/vfio/pci/vfio_pci_core.c |   17 +++++++++++++++++
> >   1 file changed, 17 insertions(+)
> > 
> > diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
> > index a0d69ddaf90d..e0cbcbc2aee1 100644
> > --- a/drivers/vfio/pci/vfio_pci_core.c
> > +++ b/drivers/vfio/pci/vfio_pci_core.c
> > @@ -13,6 +13,7 @@
> >   #include <linux/device.h>
> >   #include <linux/eventfd.h>
> >   #include <linux/file.h>
> > +#include <linux/fb.h>
> >   #include <linux/interrupt.h>
> >   #include <linux/iommu.h>
> >   #include <linux/module.h>
> > @@ -29,6 +30,8 @@
> >   
> >   #include <linux/vfio_pci_core.h>
> >   
> > +#include <drm/drm_aperture.h>
> > +
> >   #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
> >   #define DRIVER_DESC "core driver for VFIO based PCI devices"
> >   
> > @@ -1793,6 +1796,20 @@ static int vfio_pci_vga_init(struct vfio_pci_core_device *vdev)
> >   	if (!vfio_pci_is_vga(pdev))
> >   		return 0;
> >   
> > +#if IS_REACHABLE(CONFIG_DRM)
> > +	drm_aperture_detach_platform_drivers(pdev);
> > +#endif
> > +
> > +#if IS_REACHABLE(CONFIG_FB)
> > +	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
> > +	if (ret)
> > +		return ret;
> > +#endif
> > +
> > +	ret = vga_remove_vgacon(pdev);
> > +	if (ret)
> > +		return ret;
> > +  
> 
> You shouldn't have to copy any of the implementation of the aperture 
> helpers.
> 
> If you call drm_aperture_remove_conflicting_pci_framebuffers() it should 
> work correctly. The only reason why it requires a DRM driver structure 
> as second argument is for the driver's name. [1] And that name is only 
> used for printing an info message. [2]

vfio-pci is not dependent on CONFIG_DRM, therefore we need to open code
this regardless.  The only difference if we were to use the existing
function would be something like:

#if IS_REACHABLE(CONFIG_DRM)
	ret = drm_aperture_remove_conflicting_pci_framebuffers(pdev, &dummy_drm_driver);
	if (ret)
		return ret;
#else
#if IS_REACHABLE(CONFIG_FB)
	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
	if (ret)
		return ret;
#endif
	ret = vga_remove_vgacon(pdev);
	if (ret)
		return ret;
#endif

It's also bad practice to create a dummy DRM driver struct with some
assumption which fields are used.  If the usage is later expanded, we'd
only discover it by users getting segfaults.  If DRM wanted to make
such an API guarantee, then we shouldn't have commit 97c9bfe3f660
("drm/aperture: Pass DRM driver structure instead of driver name").

> The plan forward would be to drop patch 1 entirely.
> 
> For patch 2, the most trivial workaround is to instanciate struct 
> drm_driver here and set the name field to 'vdev->vdev.ops->name'. In the 
> longer term, the aperture helpers will be moved out of DRM and into a 
> more prominent location. That workaround will be cleaned up then.

Trivial in execution, but as above, this is poor practice and should be
avoided.

> Alternatively, drm_aperture_remove_conflicting_pci_framebuffers() could 
> be changed to accept the name string as second argument, but that's 
> quite a bit of churn within the DRM code.

The series as presented was exactly meant to provide the most correct
solution with the least churn to the DRM code.  The above referenced
commit precludes us from calling the existing DRM function directly
without resorting to poor practices of assuming the usage of the DRM
driver struct.  Using the existing DRM function also does not prevent
us from open coding the remainder of the function to avoid a CONFIG_DRM
dependency.  Thanks,

Alex
Gerd Hoffmann June 8, 2022, 3:37 p.m. UTC | #3
Hi,

> You shouldn't have to copy any of the implementation of the aperture
> helpers.

That comes from the aperture helpers being part of drm ...

> For patch 2, the most trivial workaround is to instanciate struct drm_driver
> here and set the name field to 'vdev->vdev.ops->name'. In the longer term,
> the aperture helpers will be moved out of DRM and into a more prominent
> location. That workaround will be cleaned up then.

... but if the long-term plan is to clean that up properly anyway I
don't see the point in bike shedding too much on the details of some
temporary solution.

> Alternatively, drm_aperture_remove_conflicting_pci_framebuffers() could be
> changed to accept the name string as second argument, but that's quite a bit
> of churn within the DRM code.

Also pointless churn because you'll have the very same churn again when
moving the aperture helpers out of drm.

take care,
  Gerd
Thomas Zimmermann June 9, 2022, 9:13 a.m. UTC | #4
Hi

Am 08.06.22 um 16:04 schrieb Alex Williamson:
>>
>> You shouldn't have to copy any of the implementation of the aperture
>> helpers.
>>
>> If you call drm_aperture_remove_conflicting_pci_framebuffers() it should
>> work correctly. The only reason why it requires a DRM driver structure
>> as second argument is for the driver's name. [1] And that name is only
>> used for printing an info message. [2]
> 
> vfio-pci is not dependent on CONFIG_DRM, therefore we need to open code
> this regardless.  The only difference if we were to use the existing
> function would be something like:
> 
> #if IS_REACHABLE(CONFIG_DRM)
> 	ret = drm_aperture_remove_conflicting_pci_framebuffers(pdev, &dummy_drm_driver);
> 	if (ret)
> 		return ret;
> #else
> #if IS_REACHABLE(CONFIG_FB)
> 	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
> 	if (ret)
> 		return ret;
> #endif
> 	ret = vga_remove_vgacon(pdev);
> 	if (ret)
> 		return ret;
> #endif
> 
> It's also bad practice to create a dummy DRM driver struct with some
> assumption which fields are used.  If the usage is later expanded, we'd
> only discover it by users getting segfaults.  If DRM wanted to make
> such an API guarantee, then we shouldn't have commit 97c9bfe3f660
> ("drm/aperture: Pass DRM driver structure instead of driver name").

What you're open coding within vfio is legacy code and we want to remove 
it as much as possible from the aperture helpers.

Tying the helpers to DRM was in mistake in retrospective. We wanted 
something tailored to the needs of DRM. Now that we've seen quite a few 
corner cases in the interaction among graphics subsystems, we need 
something else. The order of creating devices and loading driver modules 
is crucial to making the hand-over between drivers work reliably. So 
far, this luckily has only been a problem in theory, but not practice.

> 
>> The plan forward would be to drop patch 1 entirely.
>>
>> For patch 2, the most trivial workaround is to instanciate struct
>> drm_driver here and set the name field to 'vdev->vdev.ops->name'. In the
>> longer term, the aperture helpers will be moved out of DRM and into a
>> more prominent location. That workaround will be cleaned up then.
> 
> Trivial in execution, but as above, this is poor practice and should be
> avoided.
> 
>> Alternatively, drm_aperture_remove_conflicting_pci_framebuffers() could
>> be changed to accept the name string as second argument, but that's
>> quite a bit of churn within the DRM code.
> 
> The series as presented was exactly meant to provide the most correct
> solution with the least churn to the DRM code.  The above referenced
> commit precludes us from calling the existing DRM function directly
> without resorting to poor practices of assuming the usage of the DRM
> driver struct.  Using the existing DRM function also does not prevent
> us from open coding the remainder of the function to avoid a CONFIG_DRM
> dependency.  Thanks,

Please have a look at the attached patch. It moves the aperture helpers 
to a location common to the various possible users (DRM, fbdev, vfio). 
The DRM interfaces remain untouched for now.  The patch should provide 
what you need in vfio and also serve our future use cases for graphics 
drivers. If possible, please create your patch on top of it.

Best regards
Thomas

> 
> Alex
>
Alex Williamson June 9, 2022, 9:41 p.m. UTC | #5
On Thu, 9 Jun 2022 11:13:22 +0200
Thomas Zimmermann <tzimmermann@suse.de> wrote:
> 
> Please have a look at the attached patch. It moves the aperture helpers 
> to a location common to the various possible users (DRM, fbdev, vfio). 
> The DRM interfaces remain untouched for now.  The patch should provide 
> what you need in vfio and also serve our future use cases for graphics 
> drivers. If possible, please create your patch on top of it.

Looks good to me, this of course makes the vfio change quite trivial.
One change I'd request:

diff --git a/drivers/video/console/Kconfig b/drivers/video/console/Kconfig
index 40c50fa2dd70..7f3c44e1538b 100644
--- a/drivers/video/console/Kconfig
+++ b/drivers/video/console/Kconfig
@@ -10,6 +10,7 @@ config VGA_CONSOLE
 	depends on !4xx && !PPC_8xx && !SPARC && !M68K && !PARISC &&  !SUPERH && \
 		(!ARM || ARCH_FOOTBRIDGE || ARCH_INTEGRATOR || ARCH_NETWINDER) && \
 		!ARM64 && !ARC && !MICROBLAZE && !OPENRISC && !S390 && !UML
+	select APERTURE_HELPERS if (DRM || FB || VFIO_PCI)
 	default y
 	help
 	  Saying Y here will allow you to use Linux in text mode through a

This should be VFIO_PCI_CORE.  Thanks,

Alex
Alex Williamson June 9, 2022, 9:44 p.m. UTC | #6
On Thu, 9 Jun 2022 15:41:02 -0600
Alex Williamson <alex.williamson@redhat.com> wrote:

> On Thu, 9 Jun 2022 11:13:22 +0200
> Thomas Zimmermann <tzimmermann@suse.de> wrote:
> > 
> > Please have a look at the attached patch. It moves the aperture helpers 
> > to a location common to the various possible users (DRM, fbdev, vfio). 
> > The DRM interfaces remain untouched for now.  The patch should provide 
> > what you need in vfio and also serve our future use cases for graphics 
> > drivers. If possible, please create your patch on top of it.  
> 
> Looks good to me, this of course makes the vfio change quite trivial.
> One change I'd request:
> 
> diff --git a/drivers/video/console/Kconfig b/drivers/video/console/Kconfig
> index 40c50fa2dd70..7f3c44e1538b 100644
> --- a/drivers/video/console/Kconfig
> +++ b/drivers/video/console/Kconfig
> @@ -10,6 +10,7 @@ config VGA_CONSOLE
>  	depends on !4xx && !PPC_8xx && !SPARC && !M68K && !PARISC &&  !SUPERH && \
>  		(!ARM || ARCH_FOOTBRIDGE || ARCH_INTEGRATOR || ARCH_NETWINDER) && \
>  		!ARM64 && !ARC && !MICROBLAZE && !OPENRISC && !S390 && !UML
> +	select APERTURE_HELPERS if (DRM || FB || VFIO_PCI)
>  	default y
>  	help
>  	  Saying Y here will allow you to use Linux in text mode through a
> 
> This should be VFIO_PCI_CORE.  Thanks,

Also, whatever tree this lands in, I'd appreciate a topic branch being
made available so I can more easily get the vfio change in on the same
release.  Thanks,

Alex
Thomas Zimmermann June 10, 2022, 7:03 a.m. UTC | #7
Hi

Am 09.06.22 um 23:44 schrieb Alex Williamson:
> On Thu, 9 Jun 2022 15:41:02 -0600
> Alex Williamson <alex.williamson@redhat.com> wrote:
> 
>> On Thu, 9 Jun 2022 11:13:22 +0200
>> Thomas Zimmermann <tzimmermann@suse.de> wrote:
>>>
>>> Please have a look at the attached patch. It moves the aperture helpers
>>> to a location common to the various possible users (DRM, fbdev, vfio).
>>> The DRM interfaces remain untouched for now.  The patch should provide
>>> what you need in vfio and also serve our future use cases for graphics
>>> drivers. If possible, please create your patch on top of it.
>>
>> Looks good to me, this of course makes the vfio change quite trivial.
>> One change I'd request:
>>
>> diff --git a/drivers/video/console/Kconfig b/drivers/video/console/Kconfig
>> index 40c50fa2dd70..7f3c44e1538b 100644
>> --- a/drivers/video/console/Kconfig
>> +++ b/drivers/video/console/Kconfig
>> @@ -10,6 +10,7 @@ config VGA_CONSOLE
>>   	depends on !4xx && !PPC_8xx && !SPARC && !M68K && !PARISC &&  !SUPERH && \
>>   		(!ARM || ARCH_FOOTBRIDGE || ARCH_INTEGRATOR || ARCH_NETWINDER) && \
>>   		!ARM64 && !ARC && !MICROBLAZE && !OPENRISC && !S390 && !UML
>> +	select APERTURE_HELPERS if (DRM || FB || VFIO_PCI)
>>   	default y
>>   	help
>>   	  Saying Y here will allow you to use Linux in text mode through a
>>
>> This should be VFIO_PCI_CORE.  Thanks,

I attached an updated patch to this email.

> 
> Also, whatever tree this lands in, I'd appreciate a topic branch being
> made available so I can more easily get the vfio change in on the same
> release.  Thanks,

You can add my patch to your series and merge it through vfio. You'd 
only have to cc dri-devel for the patch's review. I guess it's more 
important for vfio than DRM. We have no hurry on the DRM side, but v5.20 
would be nice.

Best regards
Thomas

> 
> Alex
>
Alex Williamson June 10, 2022, 2:30 p.m. UTC | #8
On Fri, 10 Jun 2022 09:03:15 +0200
Thomas Zimmermann <tzimmermann@suse.de> wrote:

> Hi
> 
> Am 09.06.22 um 23:44 schrieb Alex Williamson:
> > On Thu, 9 Jun 2022 15:41:02 -0600
> > Alex Williamson <alex.williamson@redhat.com> wrote:
> >   
> >> On Thu, 9 Jun 2022 11:13:22 +0200
> >> Thomas Zimmermann <tzimmermann@suse.de> wrote:  
> >>>
> >>> Please have a look at the attached patch. It moves the aperture helpers
> >>> to a location common to the various possible users (DRM, fbdev, vfio).
> >>> The DRM interfaces remain untouched for now.  The patch should provide
> >>> what you need in vfio and also serve our future use cases for graphics
> >>> drivers. If possible, please create your patch on top of it.  
> >>
> >> Looks good to me, this of course makes the vfio change quite trivial.
> >> One change I'd request:
> >>
> >> diff --git a/drivers/video/console/Kconfig b/drivers/video/console/Kconfig
> >> index 40c50fa2dd70..7f3c44e1538b 100644
> >> --- a/drivers/video/console/Kconfig
> >> +++ b/drivers/video/console/Kconfig
> >> @@ -10,6 +10,7 @@ config VGA_CONSOLE
> >>   	depends on !4xx && !PPC_8xx && !SPARC && !M68K && !PARISC &&  !SUPERH && \
> >>   		(!ARM || ARCH_FOOTBRIDGE || ARCH_INTEGRATOR || ARCH_NETWINDER) && \
> >>   		!ARM64 && !ARC && !MICROBLAZE && !OPENRISC && !S390 && !UML
> >> +	select APERTURE_HELPERS if (DRM || FB || VFIO_PCI)
> >>   	default y
> >>   	help
> >>   	  Saying Y here will allow you to use Linux in text mode through a
> >>
> >> This should be VFIO_PCI_CORE.  Thanks,  
> 
> I attached an updated patch to this email.
> 
> > 
> > Also, whatever tree this lands in, I'd appreciate a topic branch being
> > made available so I can more easily get the vfio change in on the same
> > release.  Thanks,  
> 
> You can add my patch to your series and merge it through vfio. You'd 
> only have to cc dri-devel for the patch's review. I guess it's more 
> important for vfio than DRM. We have no hurry on the DRM side, but v5.20 
> would be nice.

Ok, I didn't realize you were offering the patch for me to post and
merge.  I'll do that.  Thanks!

Alex
diff mbox series

Patch

diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index a0d69ddaf90d..e0cbcbc2aee1 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -13,6 +13,7 @@ 
 #include <linux/device.h>
 #include <linux/eventfd.h>
 #include <linux/file.h>
+#include <linux/fb.h>
 #include <linux/interrupt.h>
 #include <linux/iommu.h>
 #include <linux/module.h>
@@ -29,6 +30,8 @@ 
 
 #include <linux/vfio_pci_core.h>
 
+#include <drm/drm_aperture.h>
+
 #define DRIVER_AUTHOR   "Alex Williamson <alex.williamson@redhat.com>"
 #define DRIVER_DESC "core driver for VFIO based PCI devices"
 
@@ -1793,6 +1796,20 @@  static int vfio_pci_vga_init(struct vfio_pci_core_device *vdev)
 	if (!vfio_pci_is_vga(pdev))
 		return 0;
 
+#if IS_REACHABLE(CONFIG_DRM)
+	drm_aperture_detach_platform_drivers(pdev);
+#endif
+
+#if IS_REACHABLE(CONFIG_FB)
+	ret = remove_conflicting_pci_framebuffers(pdev, vdev->vdev.ops->name);
+	if (ret)
+		return ret;
+#endif
+
+	ret = vga_remove_vgacon(pdev);
+	if (ret)
+		return ret;
+
 	ret = vga_client_register(pdev, vfio_pci_set_decode);
 	if (ret)
 		return ret;