[03/51] drm: add managed resources tied to drm_device
diff mbox series

Message ID 20200221210319.2245170-4-daniel.vetter@ffwll.ch
State New
Headers show
Series
  • drm managed resources, v2
Related show

Commit Message

Daniel Vetter Feb. 21, 2020, 9:02 p.m. UTC
We have lots of these. And the cleanup code tends to be of dubious
quality. The biggest wrong pattern is that developers use devm_, which
ties the release action to the underlying struct device, whereas
all the userspace visible stuff attached to a drm_device can long
outlive that one (e.g. after a hotunplug while userspace has open
files and mmap'ed buffers). Give people what they want, but with more
correctness.

Mostly copied from devres.c, with types adjusted to fit drm_device and
a few simplifications - I didn't (yet) copy over everything. Since
the types don't match code sharing looked like a hopeless endeavour.

For now it's only super simplified, no groups, you can't remove
actions (but kfree exists, we'll need that soon). Plus all specific to
drm_device ofc, including the logging. Which I didn't bother to make
compile-time optional, since none of the other drm logging is compile
time optional either.

One tricky bit here is the chicken&egg between allocating your
drm_device structure and initiliazing it with drm_dev_init. For
perfect onion unwinding we'd need to have the action to kfree the
allocation registered before drm_dev_init registers any of its own
release handlers. But drm_dev_init doesn't know where exactly the
drm_device is emebedded into the overall structure, and by the time it
returns it'll all be too late. And forcing drivers to be able clean up
everything except the one kzalloc is silly.

Work around this by having a very special final_kfree pointer. This
also avoids troubles with the list head possibly disappearing from
underneath us when we release all resources attached to the
drm_device.

v2: Do all the kerneldoc at the end, to avoid lots of fairly pointless
shuffling while getting everything into shape.

v3: Add static to add/del_dr (Neil)
Move typo fix to the right patch (Neil)

v4: Enforce contract for drmm_add_final_kfree:

Use ksize() to check that the drm_device is indeed contained somewhere
in the final kfree(). Because we need that or the entire managed
release logic blows up in a pile of use-after-frees. Motivated by a
discussion with Laurent.

v5: Review from Laurent:
- %zu instead of casting size_t
- header guards
- sorting of includes
- guarding of data assignment if we didn't allocate it for a NULL
  pointer
- delete spurious newline
- cast void* data parameter correctly in ->release call, no idea how
  this even worked before

Cc: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Cc: Neil Armstrong <narmstrong@baylibre.com
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
---
 Documentation/gpu/drm-internals.rst |   6 +
 drivers/gpu/drm/Makefile            |   3 +-
 drivers/gpu/drm/drm_drv.c           |  13 ++-
 drivers/gpu/drm/drm_internal.h      |   3 +
 drivers/gpu/drm/drm_managed.c       | 175 ++++++++++++++++++++++++++++
 include/drm/drm_device.h            |  12 ++
 include/drm/drm_managed.h           |  30 +++++
 include/drm/drm_print.h             |   6 +
 8 files changed, 246 insertions(+), 2 deletions(-)
 create mode 100644 drivers/gpu/drm/drm_managed.c
 create mode 100644 include/drm/drm_managed.h

Comments

Andrzej Hajda Feb. 25, 2020, 10:27 a.m. UTC | #1
Hi Daniel,


The patchset looks interesting.


On 21.02.2020 22:02, Daniel Vetter wrote:
> We have lots of these. And the cleanup code tends to be of dubious
> quality. The biggest wrong pattern is that developers use devm_, which
> ties the release action to the underlying struct device, whereas
> all the userspace visible stuff attached to a drm_device can long
> outlive that one (e.g. after a hotunplug while userspace has open
> files and mmap'ed buffers). Give people what they want, but with more
> correctness.


I am not familiar with this stuff, so forgive me stupid questions.

Is it documented how uapi should behave in such case?

I guess the general rule is to return errors on most ioctls (ENODEV,
EIO?), and wait until userspace releases everything, as there is not
much more to do.

If that is true what is the point of keeping these structs anyway -
trivial functions with small context data should do the job.

I suspect I am missing something but I do not know what :)


>
> Mostly copied from devres.c, with types adjusted to fit drm_device and
> a few simplifications - I didn't (yet) copy over everything. Since
> the types don't match code sharing looked like a hopeless endeavour.
>
> For now it's only super simplified, no groups, you can't remove
> actions (but kfree exists, we'll need that soon). Plus all specific to
> drm_device ofc, including the logging. Which I didn't bother to make
> compile-time optional, since none of the other drm logging is compile
> time optional either.


I saw in v1 thread that copy/paste is OK and merging back devres and
drmres can be done later, but experience shows that after short time
things get de-synchronized and merging process becomes quite painful.

On the other side I guess it shouldn't be difficult to split devres into
consumer agnostic core and "struct device" helpers and then use the core
in drm.

For example currently devres uses two fields from struct device:

    spinlock_t        devres_lock;
    struct list_head    devres_head;

Lets put it into separate struct:

struct devres {

    spinlock_t        lock;
    struct list_head    head;

};

And embed this struct into "struct device".

Then convert all core devres functions to take "struct devres *"
argument instead of "struct device *" and then these core functions can
be usable in drm.

Looks quite simple separation of abstraction (devres) and its consumer
(struct device).

After such split one could think about changing name devres to something
more reliable.


Regards

Andrzej
Daniel Vetter Feb. 25, 2020, 3:03 p.m. UTC | #2
On Tue, Feb 25, 2020 at 11:27 AM Andrzej Hajda <a.hajda@samsung.com> wrote:
>
> Hi Daniel,
>
>
> The patchset looks interesting.
>
>
> On 21.02.2020 22:02, Daniel Vetter wrote:
> > We have lots of these. And the cleanup code tends to be of dubious
> > quality. The biggest wrong pattern is that developers use devm_, which
> > ties the release action to the underlying struct device, whereas
> > all the userspace visible stuff attached to a drm_device can long
> > outlive that one (e.g. after a hotunplug while userspace has open
> > files and mmap'ed buffers). Give people what they want, but with more
> > correctness.
>
>
> I am not familiar with this stuff, so forgive me stupid questions.
>
> Is it documented how uapi should behave in such case?
>
> I guess the general rule is to return errors on most ioctls (ENODEV,
> EIO?), and wait until userspace releases everything, as there is not
> much more to do.
>
> If that is true what is the point of keeping these structs anyway -
> trivial functions with small context data should do the job.
>
> I suspect I am missing something but I do not know what :)

We could do the above (also needs unmapping of all mmaps, so userspace
then gets SIGSEGV everywhere) and watch userspace crash&burn.
Essentially if the kernel can't do this properly, then there's no hope
that userspace will be any better.

Hence the idea is that we keep everything userspace facing still
around, except it doesn't do much anymore. So connectors still there,
but they look disconnected. Userspace can then hopefully eventually
get around to processing the sysfs hotunplug event and remove the
device from all its list. So the long-term idea is that a lot of stuff
keeps working, except the driver doesn't talk to the hardware anymore.
And we just sit around waiting for userspace to clean things up.

I guess once we have a bunch of the panel/usb drivers converted over
we could indeed document how this is all supposed to work from an uapi
pov. But right now a lot of this is all rather aspirational, I think
only the recent simple display pipe based drivers implement this as
described above.

> > Mostly copied from devres.c, with types adjusted to fit drm_device and
> > a few simplifications - I didn't (yet) copy over everything. Since
> > the types don't match code sharing looked like a hopeless endeavour.
> >
> > For now it's only super simplified, no groups, you can't remove
> > actions (but kfree exists, we'll need that soon). Plus all specific to
> > drm_device ofc, including the logging. Which I didn't bother to make
> > compile-time optional, since none of the other drm logging is compile
> > time optional either.
>
>
> I saw in v1 thread that copy/paste is OK and merging back devres and
> drmres can be done later, but experience shows that after short time
> things get de-synchronized and merging process becomes quite painful.
>
> On the other side I guess it shouldn't be difficult to split devres into
> consumer agnostic core and "struct device" helpers and then use the core
> in drm.
>
> For example currently devres uses two fields from struct device:
>
>     spinlock_t        devres_lock;
>     struct list_head    devres_head;
>
> Lets put it into separate struct:
>
> struct devres {
>
>     spinlock_t        lock;
>     struct list_head    head;
>
> };
>
> And embed this struct into "struct device".
>
> Then convert all core devres functions to take "struct devres *"
> argument instead of "struct device *" and then these core functions can
> be usable in drm.
>
> Looks quite simple separation of abstraction (devres) and its consumer
> (struct device).
>
> After such split one could think about changing name devres to something
> more reliable.

There was a long discussion on v1 exactly about this, Greg's
suggestion was to "just share a struct device". So we're not going to
do this here, and the struct device seems like slight overkill and not
a good enough fit here.
-Daniel
Andrzej Hajda Feb. 26, 2020, 9:21 a.m. UTC | #3
On 25.02.2020 16:03, Daniel Vetter wrote:
> On Tue, Feb 25, 2020 at 11:27 AM Andrzej Hajda <a.hajda@samsung.com> wrote:
>> Hi Daniel,
>>
>>
>> The patchset looks interesting.
>>
>>
>> On 21.02.2020 22:02, Daniel Vetter wrote:
>>> We have lots of these. And the cleanup code tends to be of dubious
>>> quality. The biggest wrong pattern is that developers use devm_, which
>>> ties the release action to the underlying struct device, whereas
>>> all the userspace visible stuff attached to a drm_device can long
>>> outlive that one (e.g. after a hotunplug while userspace has open
>>> files and mmap'ed buffers). Give people what they want, but with more
>>> correctness.
>>
>> I am not familiar with this stuff, so forgive me stupid questions.
>>
>> Is it documented how uapi should behave in such case?
>>
>> I guess the general rule is to return errors on most ioctls (ENODEV,
>> EIO?), and wait until userspace releases everything, as there is not
>> much more to do.
>>
>> If that is true what is the point of keeping these structs anyway -
>> trivial functions with small context data should do the job.
>>
>> I suspect I am missing something but I do not know what :)
> We could do the above (also needs unmapping of all mmaps, so userspace
> then gets SIGSEGV everywhere) and watch userspace crash&burn.
> Essentially if the kernel can't do this properly, then there's no hope
> that userspace will be any better.


We do not want to crash userspace. We just need to tell userspace that
the kernel objects userspace has references to are not valid.

For this two mechanism should be enough:

- signal hot-unplug,

- report error (ENODEV for example) on any userspace requests (ioctls)
on invalid objects.

Expecting from userspace properly handling ioctl errors seems to be fair.

Regarding mmap I am not sure how to properly handle disappearing
devices, but this is common problem regardless which solution we use.


>
> Hence the idea is that we keep everything userspace facing still
> around, except it doesn't do much anymore. So connectors still there,
> but they look disconnected.


It looks like lying to userspace that physical connectors still exists.
If we want to lie we need good reason for that. What is that reason?

Why not just tell connectors are gone?


> Userspace can then hopefully eventually
> get around to processing the sysfs hotunplug event and remove the
> device from all its list. So the long-term idea is that a lot of stuff
> keeps working, except the driver doesn't talk to the hardware anymore.
> And we just sit around waiting for userspace to clean things up.


What does it mean "lot of stuff keeps working"? What drm driver can do
without hardware? Could you show some examples?


>
> I guess once we have a bunch of the panel/usb drivers converted over
> we could indeed document how this is all supposed to work from an uapi
> pov. But right now a lot of this is all rather aspirational, I think
> only the recent simple display pipe based drivers implement this as
> described above.
>
>>> Mostly copied from devres.c, with types adjusted to fit drm_device and
>>> a few simplifications - I didn't (yet) copy over everything. Since
>>> the types don't match code sharing looked like a hopeless endeavour.
>>>
>>> For now it's only super simplified, no groups, you can't remove
>>> actions (but kfree exists, we'll need that soon). Plus all specific to
>>> drm_device ofc, including the logging. Which I didn't bother to make
>>> compile-time optional, since none of the other drm logging is compile
>>> time optional either.
>>
>> I saw in v1 thread that copy/paste is OK and merging back devres and
>> drmres can be done later, but experience shows that after short time
>> things get de-synchronized and merging process becomes quite painful.
>>
>> On the other side I guess it shouldn't be difficult to split devres into
>> consumer agnostic core and "struct device" helpers and then use the core
>> in drm.
>>
>> For example currently devres uses two fields from struct device:
>>
>>     spinlock_t        devres_lock;
>>     struct list_head    devres_head;
>>
>> Lets put it into separate struct:
>>
>> struct devres {
>>
>>     spinlock_t        lock;
>>     struct list_head    head;
>>
>> };
>>
>> And embed this struct into "struct device".
>>
>> Then convert all core devres functions to take "struct devres *"
>> argument instead of "struct device *" and then these core functions can
>> be usable in drm.
>>
>> Looks quite simple separation of abstraction (devres) and its consumer
>> (struct device).
>>
>> After such split one could think about changing name devres to something
>> more reliable.
> There was a long discussion on v1 exactly about this, Greg's
> suggestion was to "just share a struct device". So we're not going to
> do this here, and the struct device seems like slight overkill and not
> a good enough fit here.


But my proposition is different, I want to get rid of "struct device"
from devres core - devres has nothing to do with device, it was bound to
it probably because it was convenient as device was the only client of
devres (I guess). Now if we want to have more devres clients abstracting
out devres from device seems quite natural. This way we will have proper
abstractions without code duplication.

Examples of devres related code according to my proposition:

// devres core

void devres_add(struct devres_head *dh, void *res)
{

   struct devres *dr = container_of(res, struct devres, data);

    unsigned long flags;

    spin_lock_irqsave(&dh->lock, flags);
    add_dr(dev, &dr->node);
    spin_unlock_irqrestore(&dh->lock, flags);
}

// device devres helper (non core)

struct clk *devm_clk_get(struct device *dev, const char *id)
{
    struct clk **ptr, *clk;

    ptr = devres_alloc(devm_clk_release, sizeof(*ptr), GFP_KERNEL);
    if (!ptr)
        return ERR_PTR(-ENOMEM);

    clk = clk_get(dev, id);
    if (!IS_ERR(clk)) {
        *ptr = clk;
        devres_add(&dev->devres, ptr);
    } else {
        devres_free(ptr);
    }

    return clk;
}


Changes are cosmetic. But then you can easily add devres to drmdev:

struct drm_device {

   ...

+   struct devres_head devres;

};

// then copy/modify from your patch:

+void *drmm_kmalloc(struct drm_device *dev, size_t size, gfp_t gfp)
+{
+	struct drmres *dr;
+
+	dr = alloc_dr(NULL, size, gfp, dev_to_node(dev->dev));
+	if (!dr)
+		return NULL;
+	dr->node.name = "kmalloc";
+
+	devres_add(&dev->devres, dr); // the only change is here
+
+	return dr->data;
+}


Btw, reimplemented add_dr is different of original add_dr and is similar
to original devres_add, so your implementation differs already from
original one, merging back these two will be painfull :)


Regards

Andrzej
Daniel Vetter Feb. 26, 2020, 10:21 a.m. UTC | #4
On Wed, Feb 26, 2020 at 10:21:17AM +0100, Andrzej Hajda wrote:
> On 25.02.2020 16:03, Daniel Vetter wrote:
> > On Tue, Feb 25, 2020 at 11:27 AM Andrzej Hajda <a.hajda@samsung.com> wrote:
> >> Hi Daniel,
> >>
> >>
> >> The patchset looks interesting.
> >>
> >>
> >> On 21.02.2020 22:02, Daniel Vetter wrote:
> >>> We have lots of these. And the cleanup code tends to be of dubious
> >>> quality. The biggest wrong pattern is that developers use devm_, which
> >>> ties the release action to the underlying struct device, whereas
> >>> all the userspace visible stuff attached to a drm_device can long
> >>> outlive that one (e.g. after a hotunplug while userspace has open
> >>> files and mmap'ed buffers). Give people what they want, but with more
> >>> correctness.
> >>
> >> I am not familiar with this stuff, so forgive me stupid questions.
> >>
> >> Is it documented how uapi should behave in such case?
> >>
> >> I guess the general rule is to return errors on most ioctls (ENODEV,
> >> EIO?), and wait until userspace releases everything, as there is not
> >> much more to do.
> >>
> >> If that is true what is the point of keeping these structs anyway -
> >> trivial functions with small context data should do the job.
> >>
> >> I suspect I am missing something but I do not know what :)
> > We could do the above (also needs unmapping of all mmaps, so userspace
> > then gets SIGSEGV everywhere) and watch userspace crash&burn.
> > Essentially if the kernel can't do this properly, then there's no hope
> > that userspace will be any better.
> 
> 
> We do not want to crash userspace. We just need to tell userspace that
> the kernel objects userspace has references to are not valid.
> 
> For this two mechanism should be enough:
> 
> - signal hot-unplug,
> 
> - report error (ENODEV for example) on any userspace requests (ioctls)
> on invalid objects.
> 
> Expecting from userspace properly handling ioctl errors seems to be fair.

The trouble is that maybe it's fair, practice says it's just not going to
happen.

> Regarding mmap I am not sure how to properly handle disappearing
> devices, but this is common problem regardless which solution we use.

signal handler wrapped around every mmap access. Which doesn't compose
across libraries, so is essentially impossible.

Note that e.g. GL's robustness extensions works exactly like this here
too: GPU dies, kernel kills all your objects and contexts and everything.
But the driver keeps "working". The only way to get information that
everything is actually dead is by querying the robustness extension, which
then will tell you what's happened.

Again this is because it's impossible to make sure userspace actually
checks error codes every where. It's also prohibitively expensive. vk goes
as far as outright removing all error validation (at least as much as
possible).

> > Hence the idea is that we keep everything userspace facing still
> > around, except it doesn't do much anymore. So connectors still there,
> > but they look disconnected.
> 
> 
> It looks like lying to userspace that physical connectors still exists.
> If we want to lie we need good reason for that. What is that reason?
> 
> Why not just tell connectors are gone?

Userspace sucks at handling hotunplugged connectors. Most of it is special
case code for DP MST connectors only.

> > Userspace can then hopefully eventually
> > get around to processing the sysfs hotunplug event and remove the
> > device from all its list. So the long-term idea is that a lot of stuff
> > keeps working, except the driver doesn't talk to the hardware anymore.
> > And we just sit around waiting for userspace to clean things up.
> 
> 
> What does it mean "lot of stuff keeps working"? What drm driver can do
> without hardware? Could you show some examples?

Nothing will "work", the goal is simply for userspace to not explode in
fire and take the entire desktop down with it.

> > I guess once we have a bunch of the panel/usb drivers converted over
> > we could indeed document how this is all supposed to work from an uapi
> > pov. But right now a lot of this is all rather aspirational, I think
> > only the recent simple display pipe based drivers implement this as
> > described above.
> >
> >>> Mostly copied from devres.c, with types adjusted to fit drm_device and
> >>> a few simplifications - I didn't (yet) copy over everything. Since
> >>> the types don't match code sharing looked like a hopeless endeavour.
> >>>
> >>> For now it's only super simplified, no groups, you can't remove
> >>> actions (but kfree exists, we'll need that soon). Plus all specific to
> >>> drm_device ofc, including the logging. Which I didn't bother to make
> >>> compile-time optional, since none of the other drm logging is compile
> >>> time optional either.
> >>
> >> I saw in v1 thread that copy/paste is OK and merging back devres and
> >> drmres can be done later, but experience shows that after short time
> >> things get de-synchronized and merging process becomes quite painful.
> >>
> >> On the other side I guess it shouldn't be difficult to split devres into
> >> consumer agnostic core and "struct device" helpers and then use the core
> >> in drm.
> >>
> >> For example currently devres uses two fields from struct device:
> >>
> >>     spinlock_t        devres_lock;
> >>     struct list_head    devres_head;
> >>
> >> Lets put it into separate struct:
> >>
> >> struct devres {
> >>
> >>     spinlock_t        lock;
> >>     struct list_head    head;
> >>
> >> };
> >>
> >> And embed this struct into "struct device".
> >>
> >> Then convert all core devres functions to take "struct devres *"
> >> argument instead of "struct device *" and then these core functions can
> >> be usable in drm.
> >>
> >> Looks quite simple separation of abstraction (devres) and its consumer
> >> (struct device).
> >>
> >> After such split one could think about changing name devres to something
> >> more reliable.
> > There was a long discussion on v1 exactly about this, Greg's
> > suggestion was to "just share a struct device". So we're not going to
> > do this here, and the struct device seems like slight overkill and not
> > a good enough fit here.
> 
> 
> But my proposition is different, I want to get rid of "struct device"
> from devres core - devres has nothing to do with device, it was bound to
> it probably because it was convenient as device was the only client of
> devres (I guess). Now if we want to have more devres clients abstracting
> out devres from device seems quite natural. This way we will have proper
> abstractions without code duplication.
> 
> Examples of devres related code according to my proposition:
> 
> // devres core
> 
> void devres_add(struct devres_head *dh, void *res)
> {
> 
>    struct devres *dr = container_of(res, struct devres, data);
> 
>     unsigned long flags;
> 
>     spin_lock_irqsave(&dh->lock, flags);
>     add_dr(dev, &dr->node);
>     spin_unlock_irqrestore(&dh->lock, flags);
> }
> 
> // device devres helper (non core)
> 
> struct clk *devm_clk_get(struct device *dev, const char *id)
> {
>     struct clk **ptr, *clk;
> 
>     ptr = devres_alloc(devm_clk_release, sizeof(*ptr), GFP_KERNEL);
>     if (!ptr)
>         return ERR_PTR(-ENOMEM);
> 
>     clk = clk_get(dev, id);
>     if (!IS_ERR(clk)) {
>         *ptr = clk;
>         devres_add(&dev->devres, ptr);
>     } else {
>         devres_free(ptr);
>     }
> 
>     return clk;
> }
> 
> 
> Changes are cosmetic. But then you can easily add devres to drmdev:
> 
> struct drm_device {
> 
>    ...
> 
> +   struct devres_head devres;
> 
> };
> 
> // then copy/modify from your patch:
> 
> +void *drmm_kmalloc(struct drm_device *dev, size_t size, gfp_t gfp)
> +{
> +	struct drmres *dr;
> +
> +	dr = alloc_dr(NULL, size, gfp, dev_to_node(dev->dev));
> +	if (!dr)
> +		return NULL;
> +	dr->node.name = "kmalloc";
> +
> +	devres_add(&dev->devres, dr); // the only change is here
> +
> +	return dr->data;
> +}
> 
> 
> Btw, reimplemented add_dr is different of original add_dr and is similar
> to original devres_add, so your implementation differs already from
> original one, merging back these two will be painfull :)

Oh I know, I guess I could go more into details about why exactly. One
reason is that I want type-checking, so struct drm_device * instead of
something else. At least for the userspace callbacks. That's going to be
tough with your approach - kmalloc is easy, it's the _add_action which
gets nasty with the type checking.

The other is that we can use drm debugging, which gives us some nice
consistency within drm at least.
-Daniel
Andrzej Hajda Feb. 26, 2020, 2:38 p.m. UTC | #5
On 26.02.2020 11:21, Daniel Vetter wrote:
> On Wed, Feb 26, 2020 at 10:21:17AM +0100, Andrzej Hajda wrote:
>> On 25.02.2020 16:03, Daniel Vetter wrote:
>>> On Tue, Feb 25, 2020 at 11:27 AM Andrzej Hajda <a.hajda@samsung.com> wrote:
>>>> Hi Daniel,
>>>>
>>>>
>>>> The patchset looks interesting.
>>>>
>>>>
>>>> On 21.02.2020 22:02, Daniel Vetter wrote:
>>>>> We have lots of these. And the cleanup code tends to be of dubious
>>>>> quality. The biggest wrong pattern is that developers use devm_, which
>>>>> ties the release action to the underlying struct device, whereas
>>>>> all the userspace visible stuff attached to a drm_device can long
>>>>> outlive that one (e.g. after a hotunplug while userspace has open
>>>>> files and mmap'ed buffers). Give people what they want, but with more
>>>>> correctness.
>>>> I am not familiar with this stuff, so forgive me stupid questions.
>>>>
>>>> Is it documented how uapi should behave in such case?
>>>>
>>>> I guess the general rule is to return errors on most ioctls (ENODEV,
>>>> EIO?), and wait until userspace releases everything, as there is not
>>>> much more to do.
>>>>
>>>> If that is true what is the point of keeping these structs anyway -
>>>> trivial functions with small context data should do the job.
>>>>
>>>> I suspect I am missing something but I do not know what :)
>>> We could do the above (also needs unmapping of all mmaps, so userspace
>>> then gets SIGSEGV everywhere) and watch userspace crash&burn.
>>> Essentially if the kernel can't do this properly, then there's no hope
>>> that userspace will be any better.
>>
>> We do not want to crash userspace. We just need to tell userspace that
>> the kernel objects userspace has references to are not valid.
>>
>> For this two mechanism should be enough:
>>
>> - signal hot-unplug,
>>
>> - report error (ENODEV for example) on any userspace requests (ioctls)
>> on invalid objects.
>>
>> Expecting from userspace properly handling ioctl errors seems to be fair.
> The trouble is that maybe it's fair, practice says it's just not going to
> happen.


So what? Bad API usage causes bad things, crashes will force developers
to fix it, if not we can assume it is not so harmful.

The gain is that kernel side is simpler and don't need to lie :)


>> Regarding mmap I am not sure how to properly handle disappearing
>> devices, but this is common problem regardless which solution we use.
> signal handler wrapped around every mmap access. Which doesn't compose
> across libraries, so is essentially impossible.
>
> Note that e.g. GL's robustness extensions works exactly like this here
> too: GPU dies, kernel kills all your objects and contexts and everything.
> But the driver keeps "working". The only way to get information that
> everything is actually dead is by querying the robustness extension, which
> then will tell you what's happened.
>
> Again this is because it's impossible to make sure userspace actually
> checks error codes every where. It's also prohibitively expensive. vk goes
> as far as outright removing all error validation (at least as much as
> possible).


vk is different story, and is for me counter-example - it has clear
policy - user should take care of proper API handling otherwise it risks
undefined behavior/crash. In your proposition I see opposition: lets
baby-sit user and protect him from his mistakes.


>
>>> Hence the idea is that we keep everything userspace facing still
>>> around, except it doesn't do much anymore. So connectors still there,
>>> but they look disconnected.
>>
>> It looks like lying to userspace that physical connectors still exists.
>> If we want to lie we need good reason for that. What is that reason?
>>
>> Why not just tell connectors are gone?
> Userspace sucks at handling hotunplugged connectors. Most of it is special
> case code for DP MST connectors only.
>
>>> Userspace can then hopefully eventually
>>> get around to processing the sysfs hotunplug event and remove the
>>> device from all its list. So the long-term idea is that a lot of stuff
>>> keeps working, except the driver doesn't talk to the hardware anymore.
>>> And we just sit around waiting for userspace to clean things up.
>>
>> What does it mean "lot of stuff keeps working"? What drm driver can do
>> without hardware? Could you show some examples?
> Nothing will "work", the goal is simply for userspace to not explode in
> fire and take the entire desktop down with it.


And why do we need to keep whole drm device for this task? What exactly
causes userspace explosion?


>
>>> I guess once we have a bunch of the panel/usb drivers converted over
>>> we could indeed document how this is all supposed to work from an uapi
>>> pov. But right now a lot of this is all rather aspirational, I think
>>> only the recent simple display pipe based drivers implement this as
>>> described above.
>>>
>>>>> Mostly copied from devres.c, with types adjusted to fit drm_device and
>>>>> a few simplifications - I didn't (yet) copy over everything. Since
>>>>> the types don't match code sharing looked like a hopeless endeavour.
>>>>>
>>>>> For now it's only super simplified, no groups, you can't remove
>>>>> actions (but kfree exists, we'll need that soon). Plus all specific to
>>>>> drm_device ofc, including the logging. Which I didn't bother to make
>>>>> compile-time optional, since none of the other drm logging is compile
>>>>> time optional either.
>>>> I saw in v1 thread that copy/paste is OK and merging back devres and
>>>> drmres can be done later, but experience shows that after short time
>>>> things get de-synchronized and merging process becomes quite painful.
>>>>
>>>> On the other side I guess it shouldn't be difficult to split devres into
>>>> consumer agnostic core and "struct device" helpers and then use the core
>>>> in drm.
>>>>
>>>> For example currently devres uses two fields from struct device:
>>>>
>>>>     spinlock_t        devres_lock;
>>>>     struct list_head    devres_head;
>>>>
>>>> Lets put it into separate struct:
>>>>
>>>> struct devres {
>>>>
>>>>     spinlock_t        lock;
>>>>     struct list_head    head;
>>>>
>>>> };
>>>>
>>>> And embed this struct into "struct device".
>>>>
>>>> Then convert all core devres functions to take "struct devres *"
>>>> argument instead of "struct device *" and then these core functions can
>>>> be usable in drm.
>>>>
>>>> Looks quite simple separation of abstraction (devres) and its consumer
>>>> (struct device).
>>>>
>>>> After such split one could think about changing name devres to something
>>>> more reliable.
>>> There was a long discussion on v1 exactly about this, Greg's
>>> suggestion was to "just share a struct device". So we're not going to
>>> do this here, and the struct device seems like slight overkill and not
>>> a good enough fit here.
>>
>> But my proposition is different, I want to get rid of "struct device"
>> from devres core - devres has nothing to do with device, it was bound to
>> it probably because it was convenient as device was the only client of
>> devres (I guess). Now if we want to have more devres clients abstracting
>> out devres from device seems quite natural. This way we will have proper
>> abstractions without code duplication.
>>
>> Examples of devres related code according to my proposition:
>>
>> // devres core
>>
>> void devres_add(struct devres_head *dh, void *res)
>> {
>>
>>    struct devres *dr = container_of(res, struct devres, data);
>>
>>     unsigned long flags;
>>
>>     spin_lock_irqsave(&dh->lock, flags);
>>     add_dr(dev, &dr->node);
>>     spin_unlock_irqrestore(&dh->lock, flags);
>> }
>>
>> // device devres helper (non core)
>>
>> struct clk *devm_clk_get(struct device *dev, const char *id)
>> {
>>     struct clk **ptr, *clk;
>>
>>     ptr = devres_alloc(devm_clk_release, sizeof(*ptr), GFP_KERNEL);
>>     if (!ptr)
>>         return ERR_PTR(-ENOMEM);
>>
>>     clk = clk_get(dev, id);
>>     if (!IS_ERR(clk)) {
>>         *ptr = clk;
>>         devres_add(&dev->devres, ptr);
>>     } else {
>>         devres_free(ptr);
>>     }
>>
>>     return clk;
>> }
>>
>>
>> Changes are cosmetic. But then you can easily add devres to drmdev:
>>
>> struct drm_device {
>>
>>    ...
>>
>> +   struct devres_head devres;
>>
>> };
>>
>> // then copy/modify from your patch:
>>
>> +void *drmm_kmalloc(struct drm_device *dev, size_t size, gfp_t gfp)
>> +{
>> +	struct drmres *dr;
>> +
>> +	dr = alloc_dr(NULL, size, gfp, dev_to_node(dev->dev));
>> +	if (!dr)
>> +		return NULL;
>> +	dr->node.name = "kmalloc";
>> +
>> +	devres_add(&dev->devres, dr); // the only change is here
>> +
>> +	return dr->data;
>> +}
>>
>>
>> Btw, reimplemented add_dr is different of original add_dr and is similar
>> to original devres_add, so your implementation differs already from
>> original one, merging back these two will be painfull :)
> Oh I know, I guess I could go more into details about why exactly. One
> reason is that I want type-checking, so struct drm_device * instead of
> something else. At least for the userspace callbacks. That's going to be
> tough with your approach - kmalloc is easy, it's the _add_action which
> gets nasty with the type checking.

Sth like this:


+static void drmm_action_release(struct devres_head *dh, void *res)
+{
+	struct drm_action_devres *devres = res;
+       struct drm_device *dev = container_of(dh, struct drm_device, devres);
+
+	devres->action(dev, devres->data);
+}
+
+int __drmm_add_action(struct drm_device *dev,
+		      drmres_release_t action,
+		      void *data, const char *name)
+{
+	struct action_devres *devres;
+
+	devres = devres_alloc(drmm_action_release,
+			      sizeof(struct drm_action_devres), GFP_KERNEL);
+	if (!devres)
+		return -ENOMEM;
+
+	devres->data = data;
+	devres->action = action;
+
+	devres_add(dev, devres);
+	return 0;
+}


Regards

Andrzej

Patch
diff mbox series

diff --git a/Documentation/gpu/drm-internals.rst b/Documentation/gpu/drm-internals.rst
index a73320576ca9..a6b6145fda78 100644
--- a/Documentation/gpu/drm-internals.rst
+++ b/Documentation/gpu/drm-internals.rst
@@ -132,6 +132,12 @@  be unmapped; on many devices, the ROM address decoder is shared with
 other BARs, so leaving it mapped could cause undesired behaviour like
 hangs or memory corruption.
 
+Managed Resources
+-----------------
+
+.. kernel-doc:: drivers/gpu/drm/drm_managed.c
+   :doc: managed resources
+
 Bus-specific Device Registration and PCI Support
 ------------------------------------------------
 
diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile
index ca0ca775d37f..53d8fa170143 100644
--- a/drivers/gpu/drm/Makefile
+++ b/drivers/gpu/drm/Makefile
@@ -17,7 +17,8 @@  drm-y       :=	drm_auth.o drm_cache.o \
 		drm_plane.o drm_color_mgmt.o drm_print.o \
 		drm_dumb_buffers.o drm_mode_config.o drm_vblank.o \
 		drm_syncobj.o drm_lease.o drm_writeback.o drm_client.o \
-		drm_client_modeset.o drm_atomic_uapi.o drm_hdcp.o
+		drm_client_modeset.o drm_atomic_uapi.o drm_hdcp.o \
+		drm_managed.o
 
 drm-$(CONFIG_DRM_LEGACY) += drm_legacy_misc.o drm_bufs.o drm_context.o drm_dma.o drm_scatter.o drm_lock.o
 drm-$(CONFIG_DRM_LIB_RANDOM) += lib/drm_random.o
diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
index 9fcd6ab3c154..3e5627d6eba6 100644
--- a/drivers/gpu/drm/drm_drv.c
+++ b/drivers/gpu/drm/drm_drv.c
@@ -629,6 +629,9 @@  int drm_dev_init(struct drm_device *dev,
 	dev->dev = get_device(parent);
 	dev->driver = driver;
 
+	INIT_LIST_HEAD(&dev->managed.resources);
+	spin_lock_init(&dev->managed.lock);
+
 	/* no per-device feature limits by default */
 	dev->driver_features = ~0u;
 
@@ -828,8 +831,16 @@  static void drm_dev_release(struct kref *ref)
 		dev->driver->release(dev);
 	} else {
 		drm_dev_fini(dev);
-		kfree(dev);
+		if (!dev->managed.final_kfree) {
+			WARN_ON(!list_empty(&dev->managed.resources));
+			kfree(dev);
+		}
 	}
+
+	drm_managed_release(dev);
+
+	if (dev->managed.final_kfree)
+		kfree(dev->managed.final_kfree);
 }
 
 /**
diff --git a/drivers/gpu/drm/drm_internal.h b/drivers/gpu/drm/drm_internal.h
index aeec2e68d772..8c2628dfc6c7 100644
--- a/drivers/gpu/drm/drm_internal.h
+++ b/drivers/gpu/drm/drm_internal.h
@@ -89,6 +89,9 @@  void drm_prime_remove_buf_handle_locked(struct drm_prime_file_private *prime_fpr
 struct drm_minor *drm_minor_acquire(unsigned int minor_id);
 void drm_minor_release(struct drm_minor *minor);
 
+/* drm_managed.c */
+void drm_managed_release(struct drm_device *dev);
+
 /* drm_vblank.c */
 void drm_vblank_disable_and_save(struct drm_device *dev, unsigned int pipe);
 void drm_vblank_cleanup(struct drm_device *dev);
diff --git a/drivers/gpu/drm/drm_managed.c b/drivers/gpu/drm/drm_managed.c
new file mode 100644
index 000000000000..a36d4604ee18
--- /dev/null
+++ b/drivers/gpu/drm/drm_managed.c
@@ -0,0 +1,175 @@ 
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2020 Intel
+ *
+ * Based on drivers/base/devres.c
+ */
+
+#include <drm/drm_managed.h>
+
+#include <linux/list.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+
+#include <drm/drm_device.h>
+#include <drm/drm_print.h>
+
+/**
+ * DOC: managed resources
+ *
+ * Inspired by struct &device managed resources, but tied to the lifetime of
+ * struct &drm_device, which can outlive the underlying physical device, usually
+ * when userspace has some open files and other handles to resources still open.
+ */
+struct drmres_node {
+	struct list_head		entry;
+	drmres_release_t		release;
+	const char			*name;
+	size_t				size;
+};
+
+struct drmres {
+	struct drmres_node		node;
+	/*
+	 * Some archs want to perform DMA into kmalloc caches
+	 * and need a guaranteed alignment larger than
+	 * the alignment of a 64-bit integer.
+	 * Thus we use ARCH_KMALLOC_MINALIGN here and get exactly the same
+	 * buffer alignment as if it was allocated by plain kmalloc().
+	 */
+	u8 __aligned(ARCH_KMALLOC_MINALIGN) data[];
+};
+
+void drm_managed_release(struct drm_device *dev)
+{
+	struct drmres *dr, *tmp;
+
+	drm_dbg_drmres(dev, "drmres release begin\n");
+	list_for_each_entry_safe(dr, tmp, &dev->managed.resources, node.entry) {
+		drm_dbg_drmres(dev, "REL %p %s (%zu bytes)\n",
+			       dr, dr->node.name, dr->node.size);
+
+		if (dr->node.release)
+			dr->node.release(dev, dr->node.size ? *(void **)&dr->data : NULL);
+
+		list_del(&dr->node.entry);
+		kfree(dr);
+	}
+	drm_dbg_drmres(dev, "drmres release end\n");
+}
+
+static __always_inline struct drmres * alloc_dr(drmres_release_t release,
+						size_t size, gfp_t gfp, int nid)
+{
+	size_t tot_size;
+	struct drmres *dr;
+
+	/* We must catch any near-SIZE_MAX cases that could overflow. */
+	if (unlikely(check_add_overflow(sizeof(*dr), size, &tot_size)))
+		return NULL;
+
+	dr = kmalloc_node_track_caller(tot_size, gfp, nid);
+	if (unlikely(!dr))
+		return NULL;
+
+	memset(dr, 0, offsetof(struct drmres, data));
+
+	INIT_LIST_HEAD(&dr->node.entry);
+	dr->node.release = release;
+	dr->node.size = size;
+
+	return dr;
+}
+
+static void del_dr(struct drm_device *dev, struct drmres *dr)
+{
+	list_del_init(&dr->node.entry);
+
+	drm_dbg_drmres(dev, "DEL %p %s (%lu bytes)\n",
+		       dr, dr->node.name, (unsigned long) dr->node.size);
+}
+
+static void add_dr(struct drm_device *dev, struct drmres *dr)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&dev->managed.lock, flags);
+	list_add(&dr->node.entry, &dev->managed.resources);
+	spin_unlock_irqrestore(&dev->managed.lock, flags);
+
+	drm_dbg_drmres(dev, "ADD %p %s (%lu bytes)\n",
+		       dr, dr->node.name, (unsigned long) dr->node.size);
+}
+
+void drmm_add_final_kfree(struct drm_device *dev, void *parent)
+{
+	WARN_ON(dev->managed.final_kfree);
+	WARN_ON(dev < (struct drm_device *) parent);
+	WARN_ON(dev + 1 >= (struct drm_device *) (parent + ksize(parent)));
+	dev->managed.final_kfree = parent;
+}
+EXPORT_SYMBOL(drmm_add_final_kfree);
+
+int __drmm_add_action(struct drm_device *dev,
+		      drmres_release_t action,
+		      void *data, const char *name)
+{
+	struct drmres *dr;
+	void **void_ptr;
+
+	dr = alloc_dr(action, data ? sizeof(void*) : 0,
+		      GFP_KERNEL | __GFP_ZERO,
+		      dev_to_node(dev->dev));
+	if (!dr)
+		return -ENOMEM;
+	dr->node.name = name;
+	if (data) {
+		void_ptr = (void **)&dr->data;
+		*void_ptr = data;
+	}
+
+	add_dr(dev, dr);
+
+	return 0;
+}
+EXPORT_SYMBOL(__drmm_add_action);
+
+void *drmm_kmalloc(struct drm_device *dev, size_t size, gfp_t gfp)
+{
+	struct drmres *dr;
+
+	dr = alloc_dr(NULL, size, gfp, dev_to_node(dev->dev));
+	if (!dr)
+		return NULL;
+	dr->node.name = "kmalloc";
+
+	add_dr(dev, dr);
+
+	return dr->data;
+}
+EXPORT_SYMBOL(drmm_kmalloc);
+
+void drmm_kfree(struct drm_device *dev, void *data)
+{
+	struct drmres *dr_match = NULL, *dr;
+	unsigned long flags;
+
+	if (!data)
+		return;
+
+	spin_lock_irqsave(&dev->managed.lock, flags);
+	list_for_each_entry(dr, &dev->managed.resources, node.entry) {
+		if (dr->data == data) {
+			dr_match = dr;
+			del_dr(dev, dr_match);
+			break;
+		}
+	}
+	spin_unlock_irqrestore(&dev->managed.lock, flags);
+
+	if (WARN_ON(!dr_match))
+		return;
+
+	kfree(dr_match);
+}
+EXPORT_SYMBOL(drmm_kfree);
diff --git a/include/drm/drm_device.h b/include/drm/drm_device.h
index bb60a949f416..2790c9ed614e 100644
--- a/include/drm/drm_device.h
+++ b/include/drm/drm_device.h
@@ -67,6 +67,18 @@  struct drm_device {
 	/** @dev: Device structure of bus-device */
 	struct device *dev;
 
+	/**
+	 * @managed:
+	 *
+	 * Managed resources linked to the lifetime of this &drm_device as
+	 * tracked by @ref.
+	 */
+	struct {
+		struct list_head resources;
+		void *final_kfree;
+		spinlock_t lock;
+	} managed;
+
 	/** @driver: DRM driver managing the device */
 	struct drm_driver *driver;
 
diff --git a/include/drm/drm_managed.h b/include/drm/drm_managed.h
new file mode 100644
index 000000000000..7b5df7d09b19
--- /dev/null
+++ b/include/drm/drm_managed.h
@@ -0,0 +1,30 @@ 
+// SPDX-License-Identifier: GPL-2.0
+
+#ifndef _DRM_MANAGED_H_
+#define _DRM_MANAGED_H_
+
+#include <linux/gfp.h>
+#include <linux/types.h>
+
+struct drm_device;
+
+typedef void (*drmres_release_t)(struct drm_device *dev, void *res);
+
+#define drmm_add_action(dev, action, data) \
+	__drmm_add_action(dev, action, data, #action)
+
+int __must_check __drmm_add_action(struct drm_device *dev,
+				   drmres_release_t action,
+				   void *data, const char *name);
+
+void drmm_add_final_kfree(struct drm_device *dev, void *parent);
+
+void *drmm_kmalloc(struct drm_device *dev, size_t size, gfp_t gfp) __malloc;
+static inline void *drmm_kzalloc(struct drm_device *dev, size_t size, gfp_t gfp)
+{
+	return drmm_kmalloc(dev, size, gfp | __GFP_ZERO);
+}
+
+void drmm_kfree(struct drm_device *dev, void *data);
+
+#endif
diff --git a/include/drm/drm_print.h b/include/drm/drm_print.h
index ca7cee8e728a..1c9417430d08 100644
--- a/include/drm/drm_print.h
+++ b/include/drm/drm_print.h
@@ -313,6 +313,10 @@  enum drm_debug_category {
 	 * @DRM_UT_DP: Used in the DP code.
 	 */
 	DRM_UT_DP		= 0x100,
+	/**
+	 * @DRM_UT_DRMRES: Used in the drm managed resources code.
+	 */
+	DRM_UT_DRMRES		= 0x200,
 };
 
 static inline bool drm_debug_enabled(enum drm_debug_category category)
@@ -442,6 +446,8 @@  void drm_dev_dbg(const struct device *dev, enum drm_debug_category category,
 	drm_dev_dbg((drm)->dev, DRM_UT_LEASE, fmt, ##__VA_ARGS__)
 #define drm_dbg_dp(drm, fmt, ...)					\
 	drm_dev_dbg((drm)->dev, DRM_UT_DP, fmt, ##__VA_ARGS__)
+#define drm_dbg_drmres(drm, fmt, ...)					\
+	drm_dev_dbg((drm)->dev, DRM_UT_DRMRES, fmt, ##__VA_ARGS__)
 
 
 /*