Proposal for RandR version 1.6, Leases and EDID-based output grabs

Commit Message

Keith Packard April 1, 2017, 10:58 p.m. UTC

As a part of the DRM leasing work, we need a way to have the X server
create a lease and pass it back to an X client. Here's a proposal for
the RandR specification changes necessary for that. The basic plan is
pretty simple:

 1. Expose the ability to create a lease for a set of CRTCs and
    OUTPUTs. The X server will have to pick a suitable encoder as that's
    not visible via RandR.

 2. Provide a way to hide some monitors from other clients using EDID
    manufacturer ids and product codes. Outputs with EDID properties
    matching the grab will report 'disconnected' to all clients other
    than the grabbing client. This way, the desktop environment never
    knows that an HMD has been plugged in, so there's no transient
    flicker of desktop being presented to it.

I wanted to make it possible for the X server to set the mode on the
leased outputs, but I'm not sure how -- I don't want to display the X
screen on them, and there's no other frame buffer I can name over the
wire. For now, that means the leasing client will need to set a mode
itself. If that fails, it can go whack the X configuration over RandR
and try again, which is all the X server would do anyways.

Comments are welcome; I'll go try to write the code in the next few
days; it all looks pretty easy at this point.

Comments

Daniel Vetter April 2, 2017, 3:43 p.m. UTC | #1

On Sat, Apr 01, 2017 at 03:58:58PM -0700, "Keith Packard" wrote:
> 
> As a part of the DRM leasing work, we need a way to have the X server
> create a lease and pass it back to an X client. Here's a proposal for
> the RandR specification changes necessary for that. The basic plan is
> pretty simple:
> 
>  1. Expose the ability to create a lease for a set of CRTCs and
>     OUTPUTs. The X server will have to pick a suitable encoder as that's
>     not visible via RandR.

On that, I think we could just unconditionally hand leases all encoders.
Encoders turned out to be a bit an uapi mistake. Neither setcrtc nor
atomic use it, the kernel always selects the right encoder for you. It's
only exposed to give userspace some hints wrt routing (and it's pretty bad
at describing modern restrictions, which often means you get a 1:1
encoder/connector mapping). Unconditionally exposing all encoders for all
lessees would fix all these troubles.

Note that there's also no properties on encoders, those only exist on
crtc, connector and planes.

Kinda more a comment on the kernel side than for xrandr.
-Daniel

>  2. Provide a way to hide some monitors from other clients using EDID
>     manufacturer ids and product codes. Outputs with EDID properties
>     matching the grab will report 'disconnected' to all clients other
>     than the grabbing client. This way, the desktop environment never
>     knows that an HMD has been plugged in, so there's no transient
>     flicker of desktop being presented to it.
> 
> I wanted to make it possible for the X server to set the mode on the
> leased outputs, but I'm not sure how -- I don't want to display the X
> screen on them, and there's no other frame buffer I can name over the
> wire. For now, that means the leasing client will need to set a mode
> itself. If that fails, it can go whack the X configuration over RandR
> and try again, which is all the X server would do anyways.
> 
> Comments are welcome; I'll go try to write the code in the next few
> days; it all looks pretty easy at this point.
> 
> diff --git a/randrproto.txt b/randrproto.txt
> index 74b7c36..8dded63 100644
> --- a/randrproto.txt
> +++ b/randrproto.txt
> @@ -1,6 +1,6 @@
>  	       The X Resize, Rotate and Reflect Extension
> -			     Version 1.5.0
> -			       2015-03-14
> +			     Version 1.6.0
> +			       2017-04-01
>  
>  			      Jim Gettys
>  			   [e-mail]
> @@ -9,9 +9,7 @@
>  			Hewlett Packard Company
>  
>  			     Keith Packard
> -			[e-mail]
> -		     Open Source Technology Center
> -			   Intel Corporation
> +			  [e-mail]
>  
>  1. Introduction
>  
> @@ -186,6 +184,24 @@ consider as single viewable areas.
>  Xinerama's information now comes from the Monitors instead of directly
>  from the CRTCs. The Monitor marked as Primary will be listed first.
>  
> +1.6. Introduction to version 1.6 of the extension
> +
> +Version 1.6 adds resource leasing.
> +
> + • A 'Lease' is a collection of crtcs and outputs which are made
> +   available to a client for direct access via kernel KMS and DRM
> +   APIs. This is done by passing a suitable file descriptor back to
> +   the client which has access to those resources. While leased, those
> +   resources aren't used by the X server.
> +
> +Version 1.6 adds EDID-based output 'grabbing'.
> +
> + • An 'Output Grab' matches a set of EDID Manufacturer ID and product
> +   codes. For outputs with matching EDID values, the connected status
> +   of the output is only visible to the grabbing client. Other clients
> +   will see the output as disconnected. Attempts to configure the
> +   grabbed output by other clients will fail.
> +
>  1.99 Acknowledgments
>  
>  Our thanks to the contributors to the design found on the xpert mailing
> @@ -273,6 +289,10 @@ Mode
>  Provider
>  	A value for a PROVIDER argument does not name a defined PROVIDER.
>  
> +OutputGrab
> +	A value for an OUTPUTGRAB argument does not name a defined
> +	OUTPUTGRAB
> +
>  			      ❧❧❧❧❧❧❧❧❧❧❧
>  
>  5. Protocol Types
> @@ -419,6 +439,23 @@ MONITORINFO { name: ATOM
>  
>  			      ❧❧❧❧❧❧❧❧❧❧❧
>  
> +5.7. Protocol Types added in version 1.6 of the extension
> +
> +OUTPUTGRAB { XID }
> +
> +EDIDMATCH { id: CARD16
> +	     code-min: CARD16
> +	     code-max: CARD16 }
> +
> +	These values come from the EDID specification. 'id' is the
> +	Manufacturer ID value which is bytes 8 and 9 in the EDID
> +	packet, stored in big endian order (MSB first). 'code-min' and
> +	'code-max' define a closed-interval of Manufacturer product
> +	codes, which is byte 10 and 11 of the EDID packet, stored in
> +	little endian order (LSB first).
> +
> +			      ❧❧❧❧❧❧❧❧❧❧❧
> +
>  6. Extension Initialization
>  
>  The name of this extension is "RANDR".
> @@ -1666,6 +1703,67 @@ dynamic changes in the display environment.
>  	window of the screen.
>  
>  			      ❧❧❧❧❧❧❧❧❧❧❧
> +
> +7.6. Extension Requests added in version 1.6 of the extension.
> +
> +┌───
> +    RRCreateLease
> +	window : WINDOW
> +	crtcs: LISTofCRTC
> +	outputs: LISTofOUTPUT
> +     ▶
> +	nfd: CARD8
> +	lease: FD
> +└───
> +	Errors: Window, Access, Value, CRTC, Output
> +
> +	Creates a new Lease for the specified crtcs and outputs from
> +	the screen defined by 'window'. Returns a KMS/DRM file
> +	descriptor which can control the leased objects directly
> +	through the kernel. While leased, all resources will appear to
> +	be 'useless' to clients other than the leasing client as
> +	follows:
> +
> +	• Crtcs are reported as having no 'possible-outputs' and all
> +	  other values reported as if the crtc were disabled.
> +
> +	• Outputs are reported as having no crtcs they can be
> +	  connected to, no clones they can share a crtc with, will
> +	  report a connection status of Disconnected, and will show
> +	  the current crtc as Disabled.
> +
> +	The lease remains in effect until the file descriptor is
> +	closed, even if the client holding the lease disconnects from
> +	the X server.
> +
> +	Returns an Access error if any of the named resources are in
> +	use or already leased to another client.
> +
> +	Returns a Match error if any of the named resources are in use
> +	by the X server.
> +
> +┌───
> +    RRCreateOutputGrab
> +	window : WINDOW
> +	outputgrab: OUTPUTGRAB
> +	matches: LISTofEDIDMATCH
> +└───
> +	Errors: Window, Access, Value, CRTC, Output
> +
> +	Creates an Output Grab with the specified ID on the screen
> +	associated with 'window'. Any output containing an EDID
> +	property matching the grab will appear to be disconnected to
> +	all clients other than the grabbing client.
> +
> +┌───
> +    RRDestroyOutputGrab
> +	outputgrab: OUTPUTGRAB
> +└───
> +	Errors: OutputGrab
> +
> +	Destroys the named OUTPUTGRAB.
> +
> +			      ❧❧❧❧❧❧❧❧❧❧❧
>  8. Extension Events
>  
>  Clients MAY select for ConfigureNotify on the root window to be
> 
> 
> -- 
> -keith



> _______________________________________________
> dri-devel mailing list
> [e-mail]
> https://lists.freedesktop.org/mailman/listinfo/dri-devel

Keith Packard April 2, 2017, 7:58 p.m. UTC | #2

Daniel Vetter <[e-mail]> writes:

> On that, I think we could just unconditionally hand leases all encoders.
> Encoders turned out to be a bit an uapi mistake.

Well, given the complexity of hardware these days, even crtcs would
probably best have been hidden...

> Neither setcrtc nor atomic use it, the kernel always selects the right
> encoder for you. It's only exposed to give userspace some hints wrt
> routing (and it's pretty bad at describing modern restrictions, which
> often means you get a 1:1 encoder/connector mapping). Unconditionally
> exposing all encoders for all lessees would fix all these troubles.

Yeah, I can't find encoders passed into any kernel api, other than the
getencoder call. It seems like we can leave them as shared objects not
subject to leasing as they are query-only. I'll go ahead and do
that. The encoder crtc set still needs to be filtered in the query
operation so that the client knows which crtc to use for each output.

Now as to how we report the kernel objects that have been leased, we
have two options:

 1) Report them back via the X protocol

 2) Have the client query the resulting lease for its contents

I already suggested that the drm query API should be changed to report
both type and id for each leased object, that would make it sufficient
here. Instead of duplicating that over the X protocol, I suggest we just
use the adjusted kernel API.

> Note that there's also no properties on encoders, those only exist on
> crtc, connector and planes.

Any thoughts on access control for properties? Right now, the set
property ioctl checks for access to the containing object, but there's
no check when querying properties. This means that you don't need to add
leases on properties.

> Kinda more a comment on the kernel side than for xrandr.

It's all tied together :-)

Daniel Vetter April 3, 2017, 7:45 a.m. UTC | #3

On Sun, Apr 02, 2017 at 12:58:33PM -0700, Keith Packard wrote:
> Daniel Vetter <[e-mail]> writes:
> 
> > On that, I think we could just unconditionally hand leases all encoders.
> > Encoders turned out to be a bit an uapi mistake.
> 
> Well, given the complexity of hardware these days, even crtcs would
> probably best have been hidden...
> 
> > Neither setcrtc nor atomic use it, the kernel always selects the right
> > encoder for you. It's only exposed to give userspace some hints wrt
> > routing (and it's pretty bad at describing modern restrictions, which
> > often means you get a 1:1 encoder/connector mapping). Unconditionally
> > exposing all encoders for all lessees would fix all these troubles.
> 
> Yeah, I can't find encoders passed into any kernel api, other than the
> getencoder call. It seems like we can leave them as shared objects not
> subject to leasing as they are query-only. I'll go ahead and do
> that. The encoder crtc set still needs to be filtered in the query
> operation so that the client knows which crtc to use for each output.
> 
> Now as to how we report the kernel objects that have been leased, we
> have two options:
> 
>  1) Report them back via the X protocol
> 
>  2) Have the client query the resulting lease for its contents
> 
> I already suggested that the drm query API should be changed to report
> both type and id for each leased object, that would make it sufficient
> here. Instead of duplicating that over the X protocol, I suggest we just
> use the adjusted kernel API.

Hm, if you restrict getresources and getplanes, you'll get your leased
objects query api. Iirc that part was missing in your kernel patch. And it
gives you exaclty what you want: per-type list of object ids.
> 
> > Note that there's also no properties on encoders, those only exist on
> > crtc, connector and planes.
> 
> Any thoughts on access control for properties? Right now, the set
> property ioctl checks for access to the containing object, but there's
> no check when querying properties. This means that you don't need to add
> leases on properties.

This changes with atomic - without properties you can't change anything
using atomic. Otoh maybe we just want to add checks on the containing
object for now, some of the finer semantic points like "don't modeset" or
"don't steal shared resources at all" can't be expressed (in a generic way
at least) by restricting to specific properties anyway. E.g. on many chips
changing the fb is only ok if you keep the same stride, pixel format and
modifier (~ tiling).

Keith Packard April 3, 2017, 4:41 p.m. UTC | #4

Daniel Vetter <[e-mail]> writes:

> Hm, if you restrict getresources and getplanes, you'll get your leased
> objects query api. Iirc that part was missing in your kernel patch. And it
> gives you exaclty what you want: per-type list of object ids.

Hrm. I think that's one Dave didn't want to restrict so that
applications don't see resources disappear and get confused. But,
perhaps a simple variant of those instead of a completely different API.

> This changes with atomic - without properties you can't change anything
> using atomic. Otoh maybe we just want to add checks on the containing
> object for now,

That's what is already in place -- property setting is always relative
to the containing resource, so we don't need any specific controls on
the properties themselves unless we wanted to restrict queries, which
doesn't seem necessary.

Daniel Vetter April 3, 2017, 10:07 p.m. UTC | #5

On Mon, Apr 03, 2017 at 09:41:34AM -0700, Keith Packard wrote:
> Daniel Vetter <[e-mail]> writes:
> 
> > Hm, if you restrict getresources and getplanes, you'll get your leased
> > objects query api. Iirc that part was missing in your kernel patch. And it
> > gives you exaclty what you want: per-type list of object ids.
> 
> Hrm. I think that's one Dave didn't want to restrict so that
> applications don't see resources disappear and get confused. But,
> perhaps a simple variant of those instead of a completely different API.

On the X side we don't want to restrict. But the VR client better be able
to deal with not seeing or being able to use everything.

Also if this confuses VR, then another reason why we want to make leases
invariant and only allow pure revoke, not changing the list.
-Daniel

Keith Packard April 3, 2017, 10:50 p.m. UTC | #6

Daniel Vetter <[e-mail]> writes:

> Also if this confuses VR, then another reason why we want to make leases
> invariant and only allow pure revoke, not changing the list.

I'm not sure why you want this to be asymmetrical, nor why you would
expect lessees to be any more competent at dealing with hotplug than the
lessor.

One use case already proposed for this API was to allow for multi-seat,
where the lessee would be an existing window system, which we already
accept as being incompetent at dealing with resource hotplug. I imagine
that in this case, a newly plugged monitor would be detected by the
multi-seat manager (logind?) and added to one of the existing leases,
along with a suitable CRTC resource. For this to work, the lessee will
need to already know about those resources and only have their
connectivity status hidden.

Daniel Vetter April 4, 2017, 7:02 a.m. UTC | #7

On Mon, Apr 03, 2017 at 03:50:33PM -0700, Keith Packard wrote:
> Daniel Vetter <[e-mail]> writes:
> 
> > Also if this confuses VR, then another reason why we want to make leases
> > invariant and only allow pure revoke, not changing the list.
> 
> I'm not sure why you want this to be asymmetrical, nor why you would
> expect lessees to be any more competent at dealing with hotplug than the
> lessor.
> 
> One use case already proposed for this API was to allow for multi-seat,
> where the lessee would be an existing window system, which we already
> accept as being incompetent at dealing with resource hotplug. I imagine
> that in this case, a newly plugged monitor would be detected by the
> multi-seat manager (logind?) and added to one of the existing leases,
> along with a suitable CRTC resource. For this to work, the lessee will
> need to already know about those resources and only have their
> connectivity status hidden.

The multi-seat thing sounds like vapourware, I think we should care about
the vr use-case for now, and only that one. And for that restricting stuff
is perfectly fine. Of course we can design the entire thing in a way that
doesn't draw us into a corner in the future right away, but that's mostly
on the implementation side. For VR itself I'd go as far as saying that
probably our "create lease" ioctl should have only the semantics we need
to pass one crtc+primary plane for pageflipping in a VR compositor,
expressed in a flag. All the details about additional corner cases are
just so unclear to me (and there's not even a clear use case that will
materialize) that I don't think having the uapi is worth it. Too close to
the "I'll regret this immediately" bucket :-)

Cheers, Daniel

Keith Packard April 4, 2017, 3:53 p.m. UTC | #8

Daniel Vetter <[e-mail]> writes:

> The multi-seat thing sounds like vapourware, I think we should care about
> the vr use-case for now, and only that one.

Ok, I can live with that, even if I like the idea of a slightly more
general solution.

> For VR itself I'd go as far as saying that probably our "create lease"
> ioctl should have only the semantics we need to pass one crtc+primary
> plane for pageflipping in a VR compositor, expressed in a flag.

Yeah, we can't express planes through X anyways. I'll leave the kernel
API with multiple planes as that's actually simpler than having it
validate that only a single plane is in the lease.

> All the details about additional corner cases are just so unclear to
> me (and there's not even a clear use case that will materialize) that
> I don't think having the uapi is worth it. Too close to the "I'll
> regret this immediately" bucket :-)

Removing the 'ChangeLease' ioctl eliminates a bunch of complexity in the
code, and means I don't even have to think about sending events. I'll
also go ahead and remove the ability to hide resources from the lessor.

Thanks, as always, for your thoughtful review.

ps -- Any thoughts on whether the X request should include the mode to
use?  Doing that would let us restrict the lessee from setting modes,
and avoid potential resource issues with the window system. However, it
would also require providing a scanout buffer in the request.

Daniel Vetter April 4, 2017, 3:59 p.m. UTC | #9

On Tue, Apr 04, 2017 at 08:53:45AM -0700, Keith Packard wrote:
> Daniel Vetter <[e-mail]> writes:
> 
> > The multi-seat thing sounds like vapourware, I think we should care about
> > the vr use-case for now, and only that one.
> 
> Ok, I can live with that, even if I like the idea of a slightly more
> general solution.
> 
> > For VR itself I'd go as far as saying that probably our "create lease"
> > ioctl should have only the semantics we need to pass one crtc+primary
> > plane for pageflipping in a VR compositor, expressed in a flag.
> 
> Yeah, we can't express planes through X anyways. I'll leave the kernel
> API with multiple planes as that's actually simpler than having it
> validate that only a single plane is in the lease.
> 
> > All the details about additional corner cases are just so unclear to
> > me (and there's not even a clear use case that will materialize) that
> > I don't think having the uapi is worth it. Too close to the "I'll
> > regret this immediately" bucket :-)
> 
> Removing the 'ChangeLease' ioctl eliminates a bunch of complexity in the
> code, and means I don't even have to think about sending events. I'll
> also go ahead and remove the ability to hide resources from the lessor.
> 
> Thanks, as always, for your thoughtful review.
> 
> ps -- Any thoughts on whether the X request should include the mode to
> use?  Doing that would let us restrict the lessee from setting modes,
> and avoid potential resource issues with the window system. However, it
> would also require providing a scanout buffer in the request.

Yeah I think that's a pretty neat idea to reduce the lease complexity even
more. If the VR compositor is unhappy and wants a different mode, it can
simply nuke the lease and as for a new one. Forgot to say that :-)
-Daniel

Keith Packard April 4, 2017, 4:48 p.m. UTC | #10

Daniel Vetter <[e-mail]> writes:

> Yeah I think that's a pretty neat idea to reduce the lease complexity even
> more. If the VR compositor is unhappy and wants a different mode, it can
> simply nuke the lease and as for a new one. Forgot to say that :-)

Not sure it changes the lease complexity, but it reduces the potential
interference with the X server after the lease is created.

Hrm. Thinking about the impact on X a bit more, this seems hard - you
can't just display the root window in the HMD, so you need a frame
buffer to use. The VR compositor can construct this knowing the planned
X mode, but, we then have to wire it through the whole X mode set
infrastructure, which is not exactly set up to do that.

I'll go look at the code in more detail, but I suspect the easiest
plan will be to have the VR compositor set its own mode. That may fail
if X is consuming too many display resources, but that doesn't seem
significantly different from having the lease fail.

This doesn't change the kernel API at all, so we can figure out the X
bits separately from the kernel bits.

Michel Dänzer April 7, 2017, 12:17 a.m. UTC | #11

On 02/04/17 07:58 AM, Keith Packard wrote:
> 
>  2. Provide a way to hide some monitors from other clients using EDID
>     manufacturer ids and product codes. Outputs with EDID properties
>     matching the grab will report 'disconnected' to all clients other
>     than the grabbing client. This way, the desktop environment never
>     knows that an HMD has been plugged in, so there's no transient
>     flicker of desktop being presented to it.

When no such special client (Steam?) is running, the desktop environment
will try to use the HMD anyway, right? So the expected use case would be
for the user to start a special client first and only plug in the HMD
afterwards? That seems a bit inconvenient.

Keith Packard April 7, 2017, 3:02 a.m. UTC | #12

Michel Dänzer <[e-mail]> writes:

> When no such special client (Steam?) is running, the desktop environment
> will try to use the HMD anyway, right? So the expected use case would be
> for the user to start a special client first and only plug in the HMD
> afterwards? That seems a bit inconvenient.

I'd love a better alternative, but this is the best I've come up
with.

Of course, it needn't be the actual VR client, it could be a stub
application that popped open a 'which app would you like to run on the
HMD' chooser of some kind, or even hooks in the desktop system that
managed that.

Suggestions very much encouraged.

Pekka Paalanen April 7, 2017, 7:56 a.m. UTC | #13

On Thu, 06 Apr 2017 20:02:23 -0700
Keith Packard <[e-mail]> wrote:

> Michel Dänzer <[e-mail]> writes:
> 
> > When no such special client (Steam?) is running, the desktop environment
> > will try to use the HMD anyway, right? So the expected use case would be
> > for the user to start a special client first and only plug in the HMD
> > afterwards? That seems a bit inconvenient.  
> 
> I'd love a better alternative, but this is the best I've come up
> with.
> 
> Of course, it needn't be the actual VR client, it could be a stub
> application that popped open a 'which app would you like to run on the
> HMD' chooser of some kind, or even hooks in the desktop system that
> managed that.
> 
> Suggestions very much encouraged.

Hi,

we need some kind of a database to recognize HMDs in any case, right?
It would be best if the database was shared, so that everyone could
recognize all HMDs, at least as far as one can do that based on EDID.

If we had such a database, perhaps interfaced with a library, when how
about Xorg using that library to automatically recognize and hide HMDs?
That library would of course be also used by all HMD-supporting Wayland
servers.

If there was such a library, maybe it could also handle EDID parsing
once and for all... a libinput for outputs?

Btw. I was also told at #openhmd that some HMDs do not appear as
connected outputs until you specifically turn it on via USB. So there
is going to be a hotplug, and you'd want to avoid sending that to
normal X11 clients so that they won't race with the special VR client
grabbing it.


Thanks,
pq

Keith Packard April 9, 2017, 5:27 p.m. UTC | #14

Pekka Paalanen <[e-mail]> writes:

> we need some kind of a database to recognize HMDs in any case, right?
> It would be best if the database was shared, so that everyone could
> recognize all HMDs, at least as far as one can do that based on EDID.

Yeah, I think you've got some good ideas here. Here are some questions
which this raises:

 * What should we do with an HMD which is in the database but for which
   no application is installed on the host?

 * Entries in the database should probably have names an versions, so
   that clients could have some way to identify whether they support
   what has been connected. Would providing the manufacturer ID and
   product codes be sufficient here?

 * Should the database be visible and managed over X? Or should it be
   hidden from clients and only visible to the X server?

 * We want to generate events when the HMD is plugged in; how should
   clients select for these?

> If we had such a database, perhaps interfaced with a library, when how
> about Xorg using that library to automatically recognize and hide HMDs?
> That library would of course be also used by all HMD-supporting Wayland
> servers.

I'm not sure a library is necessary, if the database was a directory
full of files.

> If there was such a library, maybe it could also handle EDID parsing
> once and for all... a libinput for outputs?

Yeah, not a terrible idea, but possibly more yak shaving than we really
need to get into here.

> Btw. I was also told at #openhmd that some HMDs do not appear as
> connected outputs until you specifically turn it on via USB. So there
> is going to be a hotplug, and you'd want to avoid sending that to
> normal X11 clients so that they won't race with the special VR client
> grabbing it.

It'd still get 'hot plugged' in the X server when enabled over USB, so
we'll want to hide that from regular clients in some way.

Pekka Paalanen April 10, 2017, 11:35 a.m. UTC | #15

On Sun, 09 Apr 2017 10:27:31 -0700
Keith Packard <[e-mail]> wrote:

> Pekka Paalanen <[e-mail]> writes:
> 
> > we need some kind of a database to recognize HMDs in any case, right?
> > It would be best if the database was shared, so that everyone could
> > recognize all HMDs, at least as far as one can do that based on EDID.  
> 
> Yeah, I think you've got some good ideas here. Here are some questions
> which this raises:
> 
>  * What should we do with an HMD which is in the database but for which
>    no application is installed on the host?

IMHO, if nothing is providing a picture intended for the HMD, the HMD
should be off. There is no use in showing an arbitrary image that does
not look right, is there?

For the rest of the questions I can't really say anything, except that
even if the database was just a bunch of files, you still need code to
access it, and probably something to ensure the entries are
well-formed, so that everyone will agree on how to parse them. One
should probably decide whether the database will only answer the
question "is it a HMD?" or can it provide other information as well.


Thanks,
pq

Mario Kleiner April 10, 2017, 5:47 p.m. UTC | #16

On 04/04/2017 06:48 PM, Keith Packard wrote:
> Daniel Vetter <[e-mail]> writes:
>
>> Yeah I think that's a pretty neat idea to reduce the lease complexity even
>> more. If the VR compositor is unhappy and wants a different mode, it can
>> simply nuke the lease and as for a new one. Forgot to say that :-)
>
> Not sure it changes the lease complexity, but it reduces the potential
> interference with the X server after the lease is created.
>
> Hrm. Thinking about the impact on X a bit more, this seems hard - you
> can't just display the root window in the HMD, so you need a frame
> buffer to use. The VR compositor can construct this knowing the planned
> X mode, but, we then have to wire it through the whole X mode set
> infrastructure, which is not exactly set up to do that.
>
> I'll go look at the code in more detail, but I suspect the easiest
> plan will be to have the VR compositor set its own mode. That may fail
> if X is consuming too many display resources, but that doesn't seem
> significantly different from having the lease fail.
>
> This doesn't change the kernel API at all, so we can figure out the X
> bits separately from the kernel bits.
>

Hi,

as input from a highly interested future user of such api's:

An additional use case beyond VR compositors, at least highly valuable 
for my kind of apps (= neuroscience / vision science / medical research 
toolkits) would be to get fullscreen exclusive control over regular 
outputs / displays. My use cases run about 98% of the time in fullscreen 
exclusive mode and want as little interference from the windowing system 
/ desktop environment as possible, with as much low level control as 
possible. It still needs windowed mode for same cases and needs a 
windowing system up and running.

Atm. under X i have to hope that fullscreen unredirection works to get 
me page flipping for single display monoscopic stimulation and 
dual-display stereoscopic stuff. And then there's the popular "Regular 
desktop GUI for interaction" + separate displays for "fullscreen + page 
flipping for controlled presentation" case.

Atm. i have to use custom xorg.confs + dual/multi-x-screen + 
ZaphodHeads, a static configuration with all the logout/login dance / no 
display hotplug for configuration change. Workable under X (minus the 
occasional ZaphodHeads corner case bugs), but somewhat clunky.

So if this would give me a plug & play dynamic replacement for 
ZaphodHeads and xorg.conf fiddling that would be _very_ valuable.

The RRCreateLease requests looks as if i could get that for regular 
non-HMD video outputs as well?

And the RRCreateOutputGrab would be mostly to avoid flicker when 
plugging HMD's or other special purpose devices, but wouldn't be 
strictly needed for a regular X-client to get a lease on a set of outputs?

As far as controllable properties on a lease go, i'd find it very useful 
if i could have control over framebuffer formats, e.g., being able to 
select 10 bit scanout formats would be very useful, but is afaik 
something that X currently doesn't expose with most drivers, especially 
not for regular desktop mode.

Set/Get Gamma tables, dithering properties, other output properties, 
modesetting would be also important. On X i can do that via RandR, but 
in the Wayland world, much of this stuff is afaik often restricted to 
privileged clients or proprietary per-compositor protocols. That's a big 
upcoming problem for me in the Wayland world, and lease support could be 
a very good solution for me.

If the underlying DRM leases allow me to control this stuff, and Wayland 
would gain similar extensions to lease outputs for fullscreen exclusive 
control, i could have one drm/kms backend that can be mostly agnostic of 
X vs. Wayland / different Wayland compositor flavors.

Basically my vote to expose as much flexility in modesetting / 
properties for the exposed leases as possible on the kernel and X side.

thanks,
-mario

>
>
> _______________________________________________
> [e-mail]: X.Org development
> Archives: http://lists.x.org/archives/xorg-devel
> Info: https://lists.x.org/mailman/listinfo/xorg-devel
>

Keith Packard April 10, 2017, 6:11 p.m. UTC | #17

Mario Kleiner <[e-mail]> writes:

> as input from a highly interested future user of such api's:

Thanks much for taking a look at this.

> My use cases run about 98% of the time in fullscreen 
> exclusive mode and want as little interference from the windowing system 
> / desktop environment as possible, with as much low level control as 
> possible. It still needs windowed mode for same cases and needs a 
> windowing system up and running.

The new APIs can give you exclusive access to the display resources,
bypassing the window system entirely.

> Atm. under X i have to hope that fullscreen unredirection works to get 
> me page flipping for single display monoscopic stimulation and 
> dual-display stereoscopic stuff. And then there's the popular "Regular 
> desktop GUI for interaction" + separate displays for "fullscreen + page 
> flipping for controlled presentation" case.

You're still depending on the window system server for timely page flips
though; the main reason we're doing the leasing work is to get that
variable out of the equation, eliminating any window system scheduling
jitter from the regular screen updates on the HMD.

> The RRCreateLease requests looks as if i could get that for regular 
> non-HMD video outputs as well?

Yes, there's almost no way we could restrict it even if we wanted
to. I'm doing testing with two standard monitors, but any hardware at
all will work.

> And the RRCreateOutputGrab would be mostly to avoid flicker when 
> plugging HMD's or other special purpose devices, but wouldn't be 
> strictly needed for a regular X-client to get a lease on a set of
> outputs?

Yes, just to avoid having the desktop extend itself onto the HMD, even
briefly. The two components of the RandR changes are logically separate,
but should be useful in combination when using HMD displays.

> As far as controllable properties on a lease go, i'd find it very useful 
> if i could have control over framebuffer formats, e.g., being able to 
> select 10 bit scanout formats would be very useful, but is afaik 
> something that X currently doesn't expose with most drivers, especially 
> not for regular desktop mode.

You have the full KMS api at your disposal; do whatever you like :-)

> If the underlying DRM leases allow me to control this stuff, and Wayland 
> would gain similar extensions to lease outputs for fullscreen exclusive 
> control, i could have one drm/kms backend that can be mostly agnostic of 
> X vs. Wayland / different Wayland compositor flavors.

Right, that's another benefit here -- allowing HMD applications to be largely
window system independent, except for acquiring the initial lease.

> Basically my vote to expose as much flexility in modesetting / 
> properties for the exposed leases as possible on the kernel and X
> side.

I'll have a second cut of the kernel API changes ready in another day or
so; that will eliminate the ability to change an existing lease, so
you'll have to acquire all of the resources you need all at
once. Otherwise, it will look quite similar at the user API level.

The RandR protocol changes will also need another spin; it sounds like
we want to configure the set of 'special' monitors and have those never
reported as connected via the current API. I don't think that will
affect your use case at all, and the other part of the protocol for
creating a lease won't change.

Alex Deucher April 10, 2017, 6:45 p.m. UTC | #18

On Tue, Apr 4, 2017 at 3:02 AM, Daniel Vetter <[e-mail]> wrote:
> On Mon, Apr 03, 2017 at 03:50:33PM -0700, Keith Packard wrote:
>> Daniel Vetter <[e-mail]> writes:
>>
>> > Also if this confuses VR, then another reason why we want to make leases
>> > invariant and only allow pure revoke, not changing the list.
>>
>> I'm not sure why you want this to be asymmetrical, nor why you would
>> expect lessees to be any more competent at dealing with hotplug than the
>> lessor.
>>
>> One use case already proposed for this API was to allow for multi-seat,
>> where the lessee would be an existing window system, which we already
>> accept as being incompetent at dealing with resource hotplug. I imagine
>> that in this case, a newly plugged monitor would be detected by the
>> multi-seat manager (logind?) and added to one of the existing leases,
>> along with a suitable CRTC resource. For this to work, the lessee will
>> need to already know about those resources and only have their
>> connectivity status hidden.
>
> The multi-seat thing sounds like vapourware, I think we should care about
> the vr use-case for now, and only that one. And for that restricting stuff
> is perfectly fine. Of course we can design the entire thing in a way that
> doesn't draw us into a corner in the future right away, but that's mostly
> on the implementation side. For VR itself I'd go as far as saying that
> probably our "create lease" ioctl should have only the semantics we need
> to pass one crtc+primary plane for pageflipping in a VR compositor,
> expressed in a flag. All the details about additional corner cases are
> just so unclear to me (and there's not even a clear use case that will
> materialize) that I don't think having the uapi is worth it. Too close to
> the "I'll regret this immediately" bucket :-)
>

I don't know about vaprware.  There were a number of attempts to
provide static allocation if display resources over the years to solve
things like custom zaphod configs and users wanting to use multiple
heads for separate users (which currently ends up in various zephyr
hacks IRC).  Lots of patches were proposed, none landed.  I think
there is a definite use case there.  Why do we need to make X aware of
the lease stuff?  What about having some pre-X configuration that
decides how to split up the display resources.  It could be user
defined static or dynamic based on what is attached.  You can just
start X on the device nodes with whatever resources they are assigned.
In the multi-user case, you can statically assign resources to each
node; in the VR case, you can detect the HMD and automatically assign
it's resources to a separate node or override if necessary.

Alex

Keith Packard April 10, 2017, 7:39 p.m. UTC | #19

Alex Deucher <[e-mail]> writes:

> I think there is a definite use case there.

I agree. What we're missing right now is someone interested in driving
an implementation of that use case to help define the right
interfaces. Lacking that, we're not likely to come up with a good
solution on our own. I think that's what Daniel is concerned about --
specifying something in the absence of an implementation using it.

I took a stab at this and added the ability to change leases on the fly,
and to create 'sub-leases' from an existing lease. He's pushing back on
those features, and I think his reasons are sound.

> Why do we need to make X aware of the lease stuff? What about having
> some pre-X configuration that decides how to split up the display
> resources.

For multi-user, this makes a lot of sense; you'd want the system to
allocate resources between users to allow them to operate fairly
independently.

For single-user with a hot-plugged HMD, I'd suggest that having X
involved makes a lot of sense -- you may have to interact with the user
to reduce resource consumption so that the HMD can be driven
successfully, and that will involve poking at the X configuration.

> It could be user defined static or dynamic based on what is attached.
> You can just start X on the device nodes with whatever resources they
> are assigned.  In the multi-user case, you can statically assign
> resources to each node; in the VR case, you can detect the HMD and
> automatically assign it's resources to a separate node or override if
> necessary.

I don't think we've precluded this for a multi-user environment, and I
think it's a good plan in the abstract.

I will, however, suggest that asking for VR applications to wait for the
desktop environment to be re-architected so that display resources can
be centrally allocated by a new 'display resource manager' seems like a
rather steep requirement.

What I do want to ensure is that these two use cases can both be
supported by the kernel interfaces we define, and that we can work in
user space on either design going forward.

If you'd like to start exploring the design of such a central service,
that'd be awesome.

Mario Kleiner April 10, 2017, 8:05 p.m. UTC | #20

On 04/10/2017 08:11 PM, Keith Packard wrote:
> Mario Kleiner <[e-mail]> writes:
>
>> as input from a highly interested future user of such api's:
>
> Thanks much for taking a look at this.
>
>> My use cases run about 98% of the time in fullscreen
>> exclusive mode and want as little interference from the windowing system
>> / desktop environment as possible, with as much low level control as
>> possible. It still needs windowed mode for same cases and needs a
>> windowing system up and running.
>
> The new APIs can give you exclusive access to the display resources,
> bypassing the window system entirely.
>
>> Atm. under X i have to hope that fullscreen unredirection works to get
>> me page flipping for single display monoscopic stimulation and
>> dual-display stereoscopic stuff. And then there's the popular "Regular
>> desktop GUI for interaction" + separate displays for "fullscreen + page
>> flipping for controlled presentation" case.
>
> You're still depending on the window system server for timely page flips
> though; the main reason we're doing the leasing work is to get that
> variable out of the equation, eliminating any window system scheduling
> jitter from the regular screen updates on the HMD.
>

Yes, and that will be an advantage for me as well, especially for some 
more exotic situations. That said, in my experience good old X is 
holding up rather well for page-flipped windows if one gets unredirected 
fullscreen with no DE interference. I have users which runs a 
1920x1080@240 Hz display at 240 fps stable update rates without skipping 
frames and proper frame accurate timing on AMD gfx with the open-source 
graphics stack on a standard Ubuntu 16.04 + Server 1.18 :)
In fact, it still beats current standard Wayland compositors by a large 
margin, mostly due to how display update scheduling is done in the 
current incarnations and because Wayland doesn't have a full DRI/Present 
or OML_sync_control equivalent yet.

>> The RRCreateLease requests looks as if i could get that for regular
>> non-HMD video outputs as well?
>
> Yes, there's almost no way we could restrict it even if we wanted
> to. I'm doing testing with two standard monitors, but any hardware at
> all will work.
>
>> And the RRCreateOutputGrab would be mostly to avoid flicker when
>> plugging HMD's or other special purpose devices, but wouldn't be
>> strictly needed for a regular X-client to get a lease on a set of
>> outputs?
>
> Yes, just to avoid having the desktop extend itself onto the HMD, even
> briefly. The two components of the RandR changes are logically separate,
> but should be useful in combination when using HMD displays.
>
>> As far as controllable properties on a lease go, i'd find it very useful
>> if i could have control over framebuffer formats, e.g., being able to
>> select 10 bit scanout formats would be very useful, but is afaik
>> something that X currently doesn't expose with most drivers, especially
>> not for regular desktop mode.
>
> You have the full KMS api at your disposal; do whatever you like :-)
>
>> If the underlying DRM leases allow me to control this stuff, and Wayland
>> would gain similar extensions to lease outputs for fullscreen exclusive
>> control, i could have one drm/kms backend that can be mostly agnostic of
>> X vs. Wayland / different Wayland compositor flavors.
>
> Right, that's another benefit here -- allowing HMD applications to be largely
> window system independent, except for acquiring the initial lease.
>

Great! Haven't looked at your patches yet, only at this thread and your 
blog, but this sounds all very promising.

>> Basically my vote to expose as much flexility in modesetting /
>> properties for the exposed leases as possible on the kernel and X
>> side.
>
> I'll have a second cut of the kernel API changes ready in another day or
> so; that will eliminate the ability to change an existing lease, so
> you'll have to acquire all of the resources you need all at
> once. Otherwise, it will look quite similar at the user API level.
>
> The RandR protocol changes will also need another spin; it sounds like
> we want to configure the set of 'special' monitors and have those never
> reported as connected via the current API. I don't think that will
> affect your use case at all, and the other part of the protocol for
> creating a lease won't change.
>

Yes, sounds like i could deal with that.
-mario

Keith Packard April 10, 2017, 9:16 p.m. UTC | #21

Mario Kleiner <[e-mail]> writes:

> Great! Haven't looked at your patches yet, only at this thread and your 
> blog, but this sounds all very promising.

I'll write up another blog post when I finish with the first round of
review. That should describe the kernel API at least. I think the X API
will be pretty simple though -- tell me which output and  crtc and I'll
hand you a lease for those.

Keith Packard April 28, 2017, 10:03 p.m. UTC | #22

Pekka Paalanen <[e-mail]> writes:

> IMHO, if nothing is providing a picture intended for the HMD, the HMD
> should be off. There is no use in showing an arbitrary image that does
> not look right, is there?

Well, if the HMD is being worn and the application crashes, then
what you want is something that keeps responding to your motion so you
can get the HMD off without falling or running into stuff...

But, yeah, in general, if you don't have an HMD application running, the
HMD can't usefully show anything from a bare window system. The trick is
to make sure existing desktop applications don't try to use it by mistake.

> even if the database was just a bunch of files, you still need code to
> access it, and probably something to ensure the entries are
> well-formed, so that everyone will agree on how to parse them. One
> should probably decide whether the database will only answer the
> question "is it a HMD?" or can it provide other information as well.

Yup, we need a spec for the files that is reasonably sane, and also
extensible so that if we want to add new data, we can. I discussed this
with Eric Anholt at breakfast this morning and we came up with a couple
of ideas:

 1) A directory full of files, each file can contain one or more monitor
    entries

 2) Monitors should be identified (at a minimum) using the EDID
    Manufacturer ID and Manufacturer product code. I can imagine
    also allowing the use of a serial number and/or date code if we end
    up using this for more stuff later.

 3) Window system independent. We obviously need this for X, but
    other window systems should share the same data.

 4) Use an existing format. Both of us would rather use JSON, but
    there's already XML in the DRM world, so that might make
    sense. These are functionally equivalent, but XML syntax is rough on
    the eyes.

 5) Allow multiple definitions in each file, but allow for multiple
    files too.

Here's a JSON-formatted example:

{
        "monitors": [
                {
                        "Manufacturer" : "LGD",
                        "Product" : 437,
                        "desktop" : true
                }
        ]
        "copyright" : "Copyright © 2017, Keith Packard",
        "license" : "GPLv3"
}

One can imagine the same done in XML, but I'm too lazy to type that
here. In any case, as you can see above, I've added a "desktop" field;
if false, the monitor would be hidden to 'normal' desktop applications
and only made visible to others.

Questions:

 Q) Where should the directory live.

 A) /usr/share/monitors for distribution-provided files, /etc/monitors
    for application-provided files.

 Q) How about RandR protocol.

 A) I'm thinking of just creating a new randr request like
    RRGetOutputInfo but which will return even "hidden" outputs with
    non-spoofed 'connection' value.

 Q) What file names should the entries use.

 A) How about just the manufacturer and product of the first entry?

 Q) Ranges of product ids?

 A) Yeah, we might want this to avoid a lot of duplicate entries.

Pekka Paalanen May 2, 2017, 7:39 a.m. UTC | #23

On Fri, 28 Apr 2017 15:03:17 -0700
Keith Packard <[e-mail]> wrote:

> Pekka Paalanen <[e-mail]> writes:
> 
> > IMHO, if nothing is providing a picture intended for the HMD, the HMD
> > should be off. There is no use in showing an arbitrary image that does
> > not look right, is there?  
> 
> Well, if the HMD is being worn and the application crashes, then
> what you want is something that keeps responding to your motion so you
> can get the HMD off without falling or running into stuff...

Hi,

I was under the impression that if you have a VR application running on
the HMD, it necessarily also implies that you have a VR compositor
running, which means that there is always some process providing a valid
image to the HMD. (At least in end-user setups.)

Here the assumption is that the application may crash or stall, but the
VR compositor never will. Obviously that's not exactly true, but if we
design for also the VR compositor crashing or stalling, then we have a
requirement for a tertiary process that is essentially just like the
VR compositor. And so the turtles pile up... which is also why I don't
think a backup plan for the VR compositor is necessary.

Or did you have an idea of something extremely simple that could still
provide a reasonable image for the HMD even when the compositors have
crashed and the GPU is frozen?

However, your original question was:

>  * What should we do with an HMD which is in the database but for which
>    no application is installed on the host?  

I assumed that means there are no VR apps nor a VR compositor that
could handle the HMD. In that case I think the HMD should be always off.

> But, yeah, in general, if you don't have an HMD application running, the
> HMD can't usefully show anything from a bare window system. The trick is
> to make sure existing desktop applications don't try to use it by mistake.

Sure.

> > even if the database was just a bunch of files, you still need code to
> > access it, and probably something to ensure the entries are
> > well-formed, so that everyone will agree on how to parse them. One
> > should probably decide whether the database will only answer the
> > question "is it a HMD?" or can it provide other information as well.  
> 
> Yup, we need a spec for the files that is reasonably sane, and also
> extensible so that if we want to add new data, we can. I discussed this
> with Eric Anholt at breakfast this morning and we came up with a couple
> of ideas:
> 
>  1) A directory full of files, each file can contain one or more monitor
>     entries
> 
>  2) Monitors should be identified (at a minimum) using the EDID
>     Manufacturer ID and Manufacturer product code. I can imagine
>     also allowing the use of a serial number and/or date code if we end
>     up using this for more stuff later.
> 
>  3) Window system independent. We obviously need this for X, but
>     other window systems should share the same data.
> 
>  4) Use an existing format. Both of us would rather use JSON, but
>     there's already XML in the DRM world, so that might make
>     sense. These are functionally equivalent, but XML syntax is rough on
>     the eyes.
> 
>  5) Allow multiple definitions in each file, but allow for multiple
>     files too.
> 
> Here's a JSON-formatted example:
> 
> {
>         "monitors": [
>                 {
>                         "Manufacturer" : "LGD",
>                         "Product" : 437,
>                         "desktop" : true
>                 }
>         ]
>         "copyright" : "Copyright © 2017, Keith Packard",
>         "license" : "GPLv3"
> }
> 
> One can imagine the same done in XML, but I'm too lazy to type that
> here. In any case, as you can see above, I've added a "desktop" field;
> if false, the monitor would be hidden to 'normal' desktop applications
> and only made visible to others.

I presume that if "desktop" is set to "true", it implies that the HMD
is capable of showing a simple 2D canvas in stereo without any special
rendering and with the default video mode. That is, creating a sort of
a virtual 2D monitor. That would be nice.

> Questions:
> 
>  Q) Where should the directory live.
> 
>  A) /usr/share/monitors for distribution-provided files, /etc/monitors
>     for application-provided files.
> 
>  Q) How about RandR protocol.
> 
>  A) I'm thinking of just creating a new randr request like
>     RRGetOutputInfo but which will return even "hidden" outputs with
>     non-spoofed 'connection' value.
> 
>  Q) What file names should the entries use.
> 
>  A) How about just the manufacturer and product of the first entry?
> 
>  Q) Ranges of product ids?
> 
>  A) Yeah, we might want this to avoid a lot of duplicate entries.
> 

FWIW, it all sounds good to me!

I don't really have an opinion on XML vs. JSON, I'm just happy it's not
another ad hoc format.


Thanks,
pq

Keith Packard May 2, 2017, 2:45 p.m. UTC | #24

Pekka Paalanen <[e-mail]> writes:

> I was under the impression that if you have a VR application running on
> the HMD, it necessarily also implies that you have a VR compositor
> running, which means that there is always some process providing a valid
> image to the HMD. (At least in end-user setups.)

Yes, I think the point I was trying to make was that X should never
attempt to talk to the HMD and provide an image.

> I assumed that means there are no VR apps nor a VR compositor that
> could handle the HMD. In that case I think the HMD should be always
> off.

Agreed.

> I presume that if "desktop" is set to "true", it implies that the HMD
> is capable of showing a simple 2D canvas in stereo without any special
> rendering and with the default video mode. That is, creating a sort of
> a virtual 2D monitor. That would be nice.

I was thinking that 'desktop' would be true for non-HMD displays that
didn't need the VR compositor. If you've got a VR compositor and want to
paint the desktop inside the VR environment, then I think you'd want to
create a synthetic monitor and hand images from that to the VR
compositor each frame.

> FWIW, it all sounds good to me!
>
> I don't really have an opinion on XML vs. JSON, I'm just happy it's not
> another ad hoc format.

Yeah, I think we're done with ad-hoc formats. I've done both XML and
JSON, and JSON is way easier to hand write, but we already use XML in a
bunch of places, so the necessary libaries are already linked into the
server...

Pekka Paalanen May 3, 2017, 7:08 a.m. UTC | #25

On Tue, 02 May 2017 07:45:25 -0700
Keith Packard <[e-mail]> wrote:

> Pekka Paalanen <[e-mail]> writes:

> > I presume that if "desktop" is set to "true", it implies that the HMD
> > is capable of showing a simple 2D canvas in stereo without any special
> > rendering and with the default video mode. That is, creating a sort of
> > a virtual 2D monitor. That would be nice.  
> 
> I was thinking that 'desktop' would be true for non-HMD displays that
> didn't need the VR compositor. If you've got a VR compositor and want to
> paint the desktop inside the VR environment, then I think you'd want to
> create a synthetic monitor and hand images from that to the VR
> compositor each frame.

Hi Keith,

do you mean to list all kinds of display devices in the database? I was
assuming it would list only HMDs, so not in database would imply it's a
normal display and good for extending the desktop to.

Or did you mean it for exceptions? As in, define a range of HMDs, but
the vendor put a few normal displays in the middle of the range, so one
needs to be able to exclude those?

The reason I mentioned "virtual 2D display" was that I recall hearing
that actually exists in some HMD hardware. If you don't do anything to
enable a 3D mode, the HMD will process the signal to produce a virtual
2D display in front the user. In such case, there is no need for a VR
compositor, the plain old 2D image signal will be shown correctly on a
plane in the virtual space by the HMD hardware itself.

I asked on #openhmd and was pointed to "Cinematic mode" on PSVR, and
found e.g. this:
http://www.psu.com/feature/31372/How-to-use-PlayStation-VR-cinematic-mode-effectively

Mind the note towards the bottom: you don't actually need a PS4 to use
it - so it must be something built into the HMD. However, reading more
details from
https://blog.us.playstation.com/2016/10/03/playstation-vr-the-ultimate-faq/
reveals that there is actually a separate processor box providing the
cinematic mode. Sounds like it's your VR compositor as a middle-man
hardware device rather than just a program. :-)


Thanks,
pq

Keith Packard May 4, 2017, 2:04 a.m. UTC | #26

Pekka Paalanen <[e-mail]> writes:

> do you mean to list all kinds of display devices in the database? I was
> assuming it would list only HMDs, so not in database would imply it's a
> normal display and good for extending the desktop to.

I intended for it to be a general database to which we could add almost
anything; you can imagine using this to replace broken EDID values,
configure alternate preferred modes or whatever.

The 'desktop' boolean says whether the desktop should be expected to use
the device or not; that's all I need for the HMD case.

> Or did you mean it for exceptions? As in, define a range of HMDs, but
> the vendor put a few normal displays in the middle of the range, so one
> needs to be able to exclude those?

Oh, that's a great thought; I hadn't considered what we would do with
conflicting entries that mapped the same device. I'd like to make that
invalid, and potentially spit out a warning message somewhere...

> The reason I mentioned "virtual 2D display" was that I recall hearing
> that actually exists in some HMD hardware. If you don't do anything to
> enable a 3D mode, the HMD will process the signal to produce a virtual
> 2D display in front the user. In such case, there is no need for a VR
> compositor, the plain old 2D image signal will be shown correctly on a
> plane in the virtual space by the HMD hardware itself.

Oh, cool! That doesn't help as it means the user will want to pick if
this happens or not. Maybe just don't include it in the database and let
the VR application turn off the X output before creating a lease?

> Mind the note towards the bottom: you don't actually need a PS4 to use
> it - so it must be something built into the HMD. However, reading more
> details from
> https://blog.us.playstation.com/2016/10/03/playstation-vr-the-ultimate-faq/
> reveals that there is actually a separate processor box providing the
> cinematic mode. Sounds like it's your VR compositor as a middle-man
> hardware device rather than just a program. :-)

Interesting. I guess it's a way to make it work without hacking the
desktop environment at all?

Thanks for your suggestions; I hope we're getting closer to some kind of
prototype at least...

Pekka Paalanen May 4, 2017, 8:13 a.m. UTC | #27

On Wed, 03 May 2017 19:04:38 -0700
Keith Packard <[e-mail]> wrote:

> Pekka Paalanen <[e-mail]> writes:
> 
> > do you mean to list all kinds of display devices in the database? I was
> > assuming it would list only HMDs, so not in database would imply it's a
> > normal display and good for extending the desktop to.  
> 
> I intended for it to be a general database to which we could add almost
> anything; you can imagine using this to replace broken EDID values,
> configure alternate preferred modes or whatever.

Ooh, a much much larger scope than I assumed. Nice.

That means you need an explicit key to denote HMDs. More below.

We could probably use it to recognize projectors and such, too.... which
makes me think that there should probably be an easy way to add more
entries: you plug in a projector, it's not recognized as such, you tell
your DE it is a projector, the DE goes and saves the match in the
database. Should be pretty easy with the directory full of files
approach you had in mind, if there just is write access.

> The 'desktop' boolean says whether the desktop should be expected to use
> the device or not; that's all I need for the HMD case.
> 
> > Or did you mean it for exceptions? As in, define a range of HMDs, but
> > the vendor put a few normal displays in the middle of the range, so one
> > needs to be able to exclude those?  
> 
> Oh, that's a great thought; I hadn't considered what we would do with
> conflicting entries that mapped the same device. I'd like to make that
> invalid, and potentially spit out a warning message somewhere...

I'm not sure... disfavouring overlapping entries at least would save us
from inventing heuristics on which one of the overlapping entries
should be picked for a device. And if those heuristics would need to be
implemented in each display server, more room for inconsistent
behaviour.

I think your idea is the better one.

> > The reason I mentioned "virtual 2D display" was that I recall hearing
> > that actually exists in some HMD hardware. If you don't do anything to
> > enable a 3D mode, the HMD will process the signal to produce a virtual
> > 2D display in front the user. In such case, there is no need for a VR
> > compositor, the plain old 2D image signal will be shown correctly on a
> > plane in the virtual space by the HMD hardware itself.  
> 
> Oh, cool! That doesn't help as it means the user will want to pick if
> this happens or not. Maybe just don't include it in the database and let
> the VR application turn off the X output before creating a lease?

Or, list it in the database as both "desktop" and "HMD" for the same
effect?

That way the desktop would extend there automatically, but it would
also be advertised as a HMD to clients. If it's not listed at all,
would it be possible to advertise it as a HMD and DRM-leased out etc.?

At least for Wayland, I'd like to not advertise "everything" as
possible HMDs but only those that really are.

> > Mind the note towards the bottom: you don't actually need a PS4 to use
> > it - so it must be something built into the HMD. However, reading more
> > details from
> > https://blog.us.playstation.com/2016/10/03/playstation-vr-the-ultimate-faq/
> > reveals that there is actually a separate processor box providing the
> > cinematic mode. Sounds like it's your VR compositor as a middle-man
> > hardware device rather than just a program. :-)  
> 
> Interesting. I guess it's a way to make it work without hacking the
> desktop environment at all?

It seems it should be capable of displaying the desktop out of the box,
but I don't think there is a separate HDMI-in for actual VR content, so
you'd still need to cooperate with the display server to turn it to
VR mode and feed the VR video stream.

I also read some discussions about 3D TV modes, but couldn't really get
a conclusion if they are supposed to render like if you were in a 3D
cinema theatre or not. Apparently that was enabled in a firmware
upgrade and maybe had some extra requirements.

I would guess the real purpose behind the cinematic mode is to get more
content workable on the HMD earlier, as it allows you to play almost
all existing games and movies that were not made for VR/HMD to begin
with.

> Thanks for your suggestions; I hope we're getting closer to some kind of
> prototype at least...
> 

Thanks,
pq

Keith Packard May 4, 2017, 6:02 p.m. UTC | #28

Pekka Paalanen <[e-mail]> writes:

> Ooh, a much much larger scope than I assumed. Nice.

Well, it's more out of a sense of fear than future planning. If all we
ever use it for is as a list of monitors that the desktop should ignore,
that'd be fine.

> That means you need an explicit key to denote HMDs. More below.

I don't think so. Presumably the VR system will have a known list of
HMDs it works with, and so it will probably have a list of EDID
information; I expect that's where the data for monitors-to-ignore will
come from in many cases. After all, the goal is to prevent the desktop
From shifting onto a display only to get kicked off as the VR app takes
over the HMD.

> We could probably use it to recognize projectors and such, too.... which
> makes me think that there should probably be an easy way to add more
> entries: you plug in a projector, it's not recognized as such, you tell
> your DE it is a projector, the DE goes and saves the match in the
> database. Should be pretty easy with the directory full of files
> approach you had in mind, if there just is write access.

Yeah, one can imagine an application designed to manage that. One can
also imagine having an additional database which the desktop itself
would read on behalf of the user; I'd hate to have the user directly
providing data for the window system server...

> I think your idea is the better one.

Thanks, it seems simpler and more predictable to me as well.

> Or, list it in the database as both "desktop" and "HMD" for the same
> effect?

All that the database needs to do today is to hide entries from the
desktop; with that, the VR app can go find the monitors and hook them up
without needing additional data.

> That way the desktop would extend there automatically, but it would
> also be advertised as a HMD to clients. If it's not listed at all,
> would it be possible to advertise it as a HMD and DRM-leased out etc.?

I think all we'd do is offer a new RandR request which listed "all"
monitors, including those hidden from the desktop, and expect that the
monitor-specific applications would use that to detect the presence of
their magic outputs and use them. For an HMD, you'll also have other
devices connected, so we just need some way of positively associating
the various input devices with the specific monitor (in case there is
more than one).

> At least for Wayland, I'd like to not advertise "everything" as
> possible HMDs but only those that really are.

I don't think the window system needs to know that the displays are HMD,
only that it shouldn't present them as part of the desktop. Everything
else about them can be left outside of this spec.

> It seems it should be capable of displaying the desktop out of the box,
> but I don't think there is a separate HDMI-in for actual VR content, so
> you'd still need to cooperate with the display server to turn it to
> VR mode and feed the VR video stream.

Sounds like it might be worth exploring how those work, and whether we
might want to provide some additional RandR protocol to make a specific
monitor 'visible' so that the desktop would extend onto it with a
command from a client. I think we can safely ignore this for now and
plan on coming up with a solution that extends on this basic work.

> I also read some discussions about 3D TV modes, but couldn't really get
> a conclusion if they are supposed to render like if you were in a 3D
> cinema theatre or not. Apparently that was enabled in a firmware
> upgrade and maybe had some extra requirements.

Hard to imagine this being relevant for the Linux desktop at least...

> I would guess the real purpose behind the cinematic mode is to get more
> content workable on the HMD earlier, as it allows you to play almost
> all existing games and movies that were not made for VR/HMD to begin
> with.

Seems like a safe assumption. Once you install a VR app and the
associated 'ignore this monitor' files for the window system, I'd expect
that to be the primary use case.

Thanks again; I'm sensing that a simple 'ignore this monitor for the
desktop' might suffice for now, but that we'll end up wanting something
more complicated in the future. I think the key is to try and avoid
making that harder by what we do now, but I don't think we should be
trying to implement a larger solution too soon.

Pekka Paalanen May 5, 2017, 8:20 a.m. UTC | #29

On Thu, 04 May 2017 11:02:44 -0700
Keith Packard <[e-mail]> wrote:

> Pekka Paalanen <[e-mail]> writes:
> 
> > That means you need an explicit key to denote HMDs. More below.  
> 
> I don't think so. Presumably the VR system will have a known list of
> HMDs it works with, and so it will probably have a list of EDID
> information; I expect that's where the data for monitors-to-ignore will
> come from in many cases. After all, the goal is to prevent the desktop
> From shifting onto a display only to get kicked off as the VR app takes
> over the HMD.

Hi,

I disagree on the details, more below.

> > That way the desktop would extend there automatically, but it would
> > also be advertised as a HMD to clients. If it's not listed at all,
> > would it be possible to advertise it as a HMD and DRM-leased out etc.?  
> 
> I think all we'd do is offer a new RandR request which listed "all"
> monitors, including those hidden from the desktop, and expect that the
> monitor-specific applications would use that to detect the presence of
> their magic outputs and use them. For an HMD, you'll also have other
> devices connected, so we just need some way of positively associating
> the various input devices with the specific monitor (in case there is
> more than one).

Such a RandR request is something I would not like to have to replicate
on Wayland. The display server contains the policy, it should not just
expose everything up for grabs. This is a fundamental difference
between X11 and Wayland architectures, and I think the output
information database should support both ways.

> > At least for Wayland, I'd like to not advertise "everything" as
> > possible HMDs but only those that really are.  
> 
> I don't think the window system needs to know that the displays are HMD,
> only that it shouldn't present them as part of the desktop. Everything
> else about them can be left outside of this spec.

I disagree. Wayland will likely have a special protocol extension
exclusively for advertising HMDs, so the display server will need to
know which ones are HMDs and which one are not, regardless whether the
desktop is extended there or not. This extension could also offer the
DRM leasing capabilities.

I gave the PSVR as an example of hardware where one both can
meaningfully extend the desktop there and offer it as a HMD. When that
output is taken into HMD use, the display will automatically stop
putting the desktop there - but only temporarily. All windows that were
on that extended output should still be there when the VR
app/compositor finishes. This is window management policy, so the
window manager must know what is going on.

(That is similar to the fullscreening with video mode changing
mechanism, where the video mode requested by an application is
temporary, and normality will be restored automatically when e.g. the
window disappears by client crash or the user switches to another
window.)

Having the window manager know what is a HMD and which client is active
on it will let it make better management decisions and even allow
switching between VR apps, or temporarily switching from VR app to the
desktop when supported, and back.

> > I also read some discussions about 3D TV modes, but couldn't really get
> > a conclusion if they are supposed to render like if you were in a 3D
> > cinema theatre or not. Apparently that was enabled in a firmware
> > upgrade and maybe had some extra requirements.  
> 
> Hard to imagine this being relevant for the Linux desktop at least...

Depends on how a video player will deliver the 3D movie content. Can it
be done as a normal GUI app providing stereo buffers, or does one need
to turn the video player into a VR app and reimplement the cinematic
mode in software. The latter option works for any HMD, but the former
option needs less power from the computer as head tracking etc. is
outsourced to the "HMD" itself.

> Thanks again; I'm sensing that a simple 'ignore this monitor for the
> desktop' might suffice for now, but that we'll end up wanting something
> more complicated in the future. I think the key is to try and avoid
> making that harder by what we do now, but I don't think we should be
> trying to implement a larger solution too soon.


Yes, that I agree with.

Ultimately I would envision an output device database describing
what kind of a device it is (a normal monitor, a video projector, a
HMD, a TV, ...) and then the software that implements the desktop or UI
policy will decide how that will be used. Some policy examples:

- A projector: do not extend desktop UI, but have the output normally
  available, so one can put regular windows there (presentation
  software). Turned on by default.

- A HMD: Do not extend desktop, do not expose to normal apps, keep it
  off until special request.

- A HMD with cinematic mode support: Extend desktop, turn on by
  default, allow special HMD requests.

- A TV: Try to disable overscan or try to compensate for it by default.

Also, a normal monitor and maybe a TV might have physical image size
relevant, while for a projector and a HMD the physical size is mostly
irrelevant.

So, how can we start simple while avoiding the need to break everything
if we later want to extend support for all the fancy stuff...

I would suggest to have a "output device type" attribute in the
database, and support only one value for now: "HMD". Then all display
servers can encode the policy to hide all HMDs by default.
Implementationwise it is no different from your idea, but separating
device description from usage policy would be a good thing in my
opinion. You can still document suggested policies in the spec if you
wish, only the wording will be more of a recommendation than a
requirement.


Thanks,
pq

Keith Packard May 5, 2017, 2:25 p.m. UTC | #30

Pekka Paalanen <[e-mail]> writes:

> I disagree on the details, more below.

> Such a RandR request is something I would not like to have to replicate
> on Wayland. The display server contains the policy, it should not just
> expose everything up for grabs. This is a fundamental difference
> between X11 and Wayland architectures, and I think the output
> information database should support both ways.

Sounds like a good idea; if you want to work on how this should appear
in Wayland, that would be great as it would provide two implementations,
rather than just one.

> I disagree. Wayland will likely have a special protocol extension
> exclusively for advertising HMDs, so the display server will need to
> know which ones are HMDs and which one are not, regardless whether the
> desktop is extended there or not. This extension could also offer the
> DRM leasing capabilities.

What I was thinking that a display which the window system cannot drive
natively should probably just be ignored entirely. An HMD which can
natively handle a desktop (such as the PSVR system) might well be
advertised as 'desktop capable', even though it is an HMD.

However, I also think you're right -- while the window system can't deal
with it *today*, that doesn't mean the window system won't be able to do
that in the future.

Hrm. I think the important distinction here is that the user installed
an application which is designed to talk directly to this display. From
that, we should probably infer that the user would like to use that
application with the display.

> Having the window manager know what is a HMD and which client is active
> on it will let it make better management decisions and even allow
> switching between VR apps, or temporarily switching from VR app to the
> desktop when supported, and back.

The window system will know when outputs are leased to another client,
so it can tell when to bring them back to the desktop. In the PSVR
instance, you'd list the device as 'desktop', but still allow
leasing. When no lease was active, the desktop could extend into the
PSVR environment. When the custom application started, it would request
a lease at which time the desktop would move off of the PSVR.

>> Thanks again; I'm sensing that a simple 'ignore this monitor for the
>> desktop' might suffice for now, but that we'll end up wanting something
>> more complicated in the future. I think the key is to try and avoid
>> making that harder by what we do now, but I don't think we should be
>> trying to implement a larger solution too soon.
>
> Yes, that I agree with.

I guess that's the question -- is a simple command to ignore a monitor
for purposes of the desktop sufficient for now? Or do we need to worry
about a possible future in which the window system implements native HMD
support?

> Ultimately I would envision an output device database describing
> what kind of a device it is (a normal monitor, a video projector, a
> HMD, a TV, ...) and then the software that implements the desktop or UI
> policy will decide how that will be used. Some policy examples:
>
> - A projector: do not extend desktop UI, but have the output normally
>   available, so one can put regular windows there (presentation
>   software). Turned on by default.
>
> - A HMD: Do not extend desktop, do not expose to normal apps, keep it
>   off until special request.
>
> - A HMD with cinematic mode support: Extend desktop, turn on by
>   default, allow special HMD requests.
>
> - A TV: Try to disable overscan or try to compensate for it by
> default.

Those all sound useful. I wonder how much we might be able to guess from
EDID info and whether we want to programmatically do some of this
(especially the TV thing; that's really annoying :-). In any case, a
problem for the future.

> I would suggest to have a "output device type" attribute in the
> database, and support only one value for now: "HMD". Then all display
> servers can encode the policy to hide all HMDs by default.
> Implementationwise it is no different from your idea, but separating
> device description from usage policy would be a good thing in my
> opinion. You can still document suggested policies in the spec if you
> wish, only the wording will be more of a recommendation than a
> requirement.

The only issue here is that now we're encoding policy in code, which is
hard to change for the average user, rather than in a configuration
file, which is easy to change. However, as we've defined it, these files
are installed by the system and it would be nice if they weren't
expected to be overridden by the user, so encoding policy there is
almost worse than in the code.

Hrm. How about we adopt your design and encode the device type in the
file, provide a fixed policy in the window system for now and consider
the possibility of changing the window system in the future to support
more advanced policy mechanisms. I don't think that shuts any doors
permanently.

Mario Kleiner May 6, 2017, 11:34 a.m. UTC | #31

On 05/05/2017 04:25 PM, Keith Packard wrote:
> Pekka Paalanen <[e-mail]> writes:
>
>> I disagree on the details, more below.
>
>> Such a RandR request is something I would not like to have to replicate
>> on Wayland. The display server contains the policy, it should not just
>> expose everything up for grabs. This is a fundamental difference
>> between X11 and Wayland architectures, and I think the output
>> information database should support both ways.
>
> Sounds like a good idea; if you want to work on how this should appear
> in Wayland, that would be great as it would provide two implementations,
> rather than just one.
>
>> I disagree. Wayland will likely have a special protocol extension
>> exclusively for advertising HMDs, so the display server will need to
>> know which ones are HMDs and which one are not, regardless whether the
>> desktop is extended there or not. This extension could also offer the
>> DRM leasing capabilities.
>
> What I was thinking that a display which the window system cannot drive
> natively should probably just be ignored entirely. An HMD which can
> natively handle a desktop (such as the PSVR system) might well be
> advertised as 'desktop capable', even though it is an HMD.
>
> However, I also think you're right -- while the window system can't deal
> with it *today*, that doesn't mean the window system won't be able to do
> that in the future.
>
> Hrm. I think the important distinction here is that the user installed
> an application which is designed to talk directly to this display. From
> that, we should probably infer that the user would like to use that
> application with the display.
>
>> Having the window manager know what is a HMD and which client is active
>> on it will let it make better management decisions and even allow
>> switching between VR apps, or temporarily switching from VR app to the
>> desktop when supported, and back.
>
> The window system will know when outputs are leased to another client,
> so it can tell when to bring them back to the desktop. In the PSVR
> instance, you'd list the device as 'desktop', but still allow
> leasing. When no lease was active, the desktop could extend into the
> PSVR environment. When the custom application started, it would request
> a lease at which time the desktop would move off of the PSVR.
>
>>> Thanks again; I'm sensing that a simple 'ignore this monitor for the
>>> desktop' might suffice for now, but that we'll end up wanting something
>>> more complicated in the future. I think the key is to try and avoid
>>> making that harder by what we do now, but I don't think we should be
>>> trying to implement a larger solution too soon.
>>
>> Yes, that I agree with.
>
> I guess that's the question -- is a simple command to ignore a monitor
> for purposes of the desktop sufficient for now? Or do we need to worry
> about a possible future in which the window system implements native HMD
> support?
>
>> Ultimately I would envision an output device database describing
>> what kind of a device it is (a normal monitor, a video projector, a
>> HMD, a TV, ...) and then the software that implements the desktop or UI
>> policy will decide how that will be used. Some policy examples:
>>
>> - A projector: do not extend desktop UI, but have the output normally
>>   available, so one can put regular windows there (presentation
>>   software). Turned on by default.
>>
>> - A HMD: Do not extend desktop, do not expose to normal apps, keep it
>>   off until special request.
>>
>> - A HMD with cinematic mode support: Extend desktop, turn on by
>>   default, allow special HMD requests.
>>
>> - A TV: Try to disable overscan or try to compensate for it by
>> default.
>
> Those all sound useful. I wonder how much we might be able to guess from
> EDID info and whether we want to programmatically do some of this
> (especially the TV thing; that's really annoying :-). In any case, a
> problem for the future.
>
>> I would suggest to have a "output device type" attribute in the
>> database, and support only one value for now: "HMD". Then all display
>> servers can encode the policy to hide all HMDs by default.
>> Implementationwise it is no different from your idea, but separating
>> device description from usage policy would be a good thing in my
>> opinion. You can still document suggested policies in the spec if you
>> wish, only the wording will be more of a recommendation than a
>> requirement.
>
> The only issue here is that now we're encoding policy in code, which is
> hard to change for the average user, rather than in a configuration
> file, which is easy to change. However, as we've defined it, these files
> are installed by the system and it would be nice if they weren't
> expected to be overridden by the user, so encoding policy there is
> almost worse than in the code.
>
> Hrm. How about we adopt your design and encode the device type in the
> file, provide a fixed policy in the window system for now and consider
> the possibility of changing the window system in the future to support
> more advanced policy mechanisms. I don't think that shuts any doors
> permanently.
>

Just please make sure that one (user configurable/opt-in if necessary) 
policy from the beginning is to allow leasing out any output to 
applications, not just HMDs. My type of scientific/medical applications 
would benefit as soon as it has the option to get a drm lease for a 
given output on both X and Wayland based systems. That's not a 
theoretical future use case, but one i'd try to offer to my users as 
soon as a stable protocol/implementation is available in a regular Linux 
distribution. It wouldn't be fun for inexperienced users if they had to 
hack the database for every model of display they want to use, or if 
every untrusted user would have to have a root password to do so.

thanks,
-mario



>
>
> _______________________________________________
> dri-devel mailing list
> [e-mail]
> https://lists.freedesktop.org/mailman/listinfo/dri-devel
>

Keith Packard May 6, 2017, 3:19 p.m. UTC | #32

Mario Kleiner <[e-mail]> writes:

> Just please make sure that one (user configurable/opt-in if necessary) 
> policy from the beginning is to allow leasing out any output to 
> applications, not just HMDs.

That's entirely a given -- the leasing API is under the control of the
application which can lease any set of resources it likes.

The only question here is how to avoid flashing the desktop contents
onto the HMD when you plug it in, and that means not listing some
displays as 'connected' in the default RandR requests. There will be a
separate request which lists them all.

Pekka Paalanen May 8, 2017, 7:33 a.m. UTC | #33

On Sat, 6 May 2017 13:34:44 +0200
Mario Kleiner <[e-mail]> wrote:

> Just please make sure that one (user configurable/opt-in if necessary) 
> policy from the beginning is to allow leasing out any output to 
> applications, not just HMDs. My type of scientific/medical applications 
> would benefit as soon as it has the option to get a drm lease for a 
> given output on both X and Wayland based systems. That's not a 
> theoretical future use case, but one i'd try to offer to my users as 
> soon as a stable protocol/implementation is available in a regular Linux 
> distribution. It wouldn't be fun for inexperienced users if they had to 
> hack the database for every model of display they want to use, or if 
> every untrusted user would have to have a root password to do so.

Hi Mario,

as Keith said, the DRM leasing API is ok for that. Such policy must not
be part of the database, IMO, so it's not an issue for the database.
This is another reason why I think the database should only describe
the hardware, not set usage patterns nor be an extension of display
server configuration.

For Wayland, we would need to experiment. I would not start with the
assumption that the Wayland extension used for grabbing HMD outputs
could also be used to grab an arbitrary output that the compositor is
already using for the desktop. Since an output is a part of the
desktop, it is possible it needs more strict or slightly different
grabbing semantics. I believe one should design both cases separately
and then see how much they have overlap.


Thanks,
pq

PS. Keith, I started writing a reply to you, but didn't finish yet.

Pekka Paalanen May 8, 2017, 10:47 a.m. UTC | #34

On Fri, 05 May 2017 07:25:14 -0700
Keith Packard <[e-mail]> wrote:

> Pekka Paalanen <[e-mail]> writes:
> 
> > I disagree on the details, more below.  
> 
> > Such a RandR request is something I would not like to have to replicate
> > on Wayland. The display server contains the policy, it should not just
> > expose everything up for grabs. This is a fundamental difference
> > between X11 and Wayland architectures, and I think the output
> > information database should support both ways.  
> 
> Sounds like a good idea; if you want to work on how this should appear
> in Wayland, that would be great as it would provide two implementations,
> rather than just one.

Hi Keith,

I would want to, but unfortunately I have barely enough time for
this fly-by commenting currently. However, I do have a fairly clear
idea of how all this talked about so far would be implemented in a
Wayland server, except for potential window management issues in case a
VR app connects both directly for a normal window and via a VR
compositor.

I forget if I mentioned this to you personally yet:
https://gist.github.com/ppaalanen/e0d2744ff71188e9a294726a37ca83c2

I wrote that before I knew about your DRM leasing work, which means
it is well outdated. The biggest difference is that instead of
flipping through the Wayland compositor, one would use DRM leases.

> > I disagree. Wayland will likely have a special protocol extension
> > exclusively for advertising HMDs, so the display server will need to
> > know which ones are HMDs and which one are not, regardless whether the
> > desktop is extended there or not. This extension could also offer the
> > DRM leasing capabilities.  
> 
> What I was thinking that a display which the window system cannot drive
> natively should probably just be ignored entirely. An HMD which can
> natively handle a desktop (such as the PSVR system) might well be
> advertised as 'desktop capable', even though it is an HMD.
> 
> However, I also think you're right -- while the window system can't deal
> with it *today*, that doesn't mean the window system won't be able to do
> that in the future.
> 
> Hrm. I think the important distinction here is that the user installed
> an application which is designed to talk directly to this display. From
> that, we should probably infer that the user would like to use that
> application with the display.

Right.


> >> Thanks again; I'm sensing that a simple 'ignore this monitor for the
> >> desktop' might suffice for now, but that we'll end up wanting something
> >> more complicated in the future. I think the key is to try and avoid
> >> making that harder by what we do now, but I don't think we should be
> >> trying to implement a larger solution too soon.  
> >
> > Yes, that I agree with.  
> 
> I guess that's the question -- is a simple command to ignore a monitor
> for purposes of the desktop sufficient for now? Or do we need to worry
> about a possible future in which the window system implements native HMD
> support?

I believe my proposal at the end of the email solves this question
both ways.

> > Ultimately I would envision an output device database describing
> > what kind of a device it is (a normal monitor, a video projector, a
> > HMD, a TV, ...) and then the software that implements the desktop or UI
> > policy will decide how that will be used. Some policy examples:
> >
> > - A projector: do not extend desktop UI, but have the output normally
> >   available, so one can put regular windows there (presentation
> >   software). Turned on by default.
> >
> > - A HMD: Do not extend desktop, do not expose to normal apps, keep it
> >   off until special request.
> >
> > - A HMD with cinematic mode support: Extend desktop, turn on by
> >   default, allow special HMD requests.
> >
> > - A TV: Try to disable overscan or try to compensate for it by
> > default.  
> 
> Those all sound useful. I wonder how much we might be able to guess from
> EDID info and whether we want to programmatically do some of this
> (especially the TV thing; that's really annoying :-). In any case, a
> problem for the future.

Indeed for the future. In the worst case, users will need to match
their devices by serial number or some other individual way one by
one.

Thinking again, I believe we have to have a way to override
database entries somehow. If we ship catch-all entries for, say,
all Sony TVs, we are bound to get some assignment wrong for
someone who then wants to make a correction without breaking
everyone else.

We are moving away from configuring outputs in a display server by the
connector name and towards configuring outputs based the actual output
device identity and type.

> > I would suggest to have a "output device type" attribute in the
> > database, and support only one value for now: "HMD". Then all display
> > servers can encode the policy to hide all HMDs by default.
> > Implementationwise it is no different from your idea, but separating
> > device description from usage policy would be a good thing in my
> > opinion. You can still document suggested policies in the spec if you
> > wish, only the wording will be more of a recommendation than a
> > requirement.  
> 
> The only issue here is that now we're encoding policy in code, which is
> hard to change for the average user, rather than in a configuration
> file, which is easy to change. However, as we've defined it, these files
> are installed by the system and it would be nice if they weren't
> expected to be overridden by the user, so encoding policy there is
> almost worse than in the code.
> 
> Hrm. How about we adopt your design and encode the device type in the
> file, provide a fixed policy in the window system for now and consider
> the possibility of changing the window system in the future to support
> more advanced policy mechanisms. I don't think that shuts any doors
> permanently.

That sounds perfect and future-proof to me.

I would like the output database to strictly describe the hardware, not
set up any policy or usage patterns. It would be something like udev
rules adding tags that you can then use in the display server
configuration for matching, except for outputs which cannot be handled
by udev.

I believe we should not have the user poke around in the output device
database unless he has a piece of equipment that is either not
recognized at all or is recognized or tagged wrong. Seems we are in
agreement here.

The policy to deal with output categories and individual outputs would
be in the display server configuration. Something like:

Section "OutputClass"
	...
	
	MatchType "HMD"
	Option "HideFromRandR" "true"
	Option "DPMS" "off"
EndSection

Heh, I wasn't even aware there actually was OutputClass already, with
only MatchDriver. Wouldn't that be a good fit for static policy
configuration in Xorg?


Thanks,
pq

Keith Packard May 8, 2017, 3:29 p.m. UTC | #35

Pekka Paalanen <[e-mail]> writes:

> I forget if I mentioned this to you personally yet:
> https://gist.github.com/ppaalanen/e0d2744ff71188e9a294726a37ca83c2

Thanks, that's a helpful bit of thinking. It looks like most of that is
still relevant, the only piece we'd swap out is how the bits actually
get to the screen, which is a technical detail.

> Thinking again, I believe we have to have a way to override
> database entries somehow. If we ship catch-all entries for, say,
> all Sony TVs, we are bound to get some assignment wrong for
> someone who then wants to make a correction without breaking
> everyone else.

I agree this would be useful, but it raises a pile of complications:

 1) Obviously these overrides will conflict with the existing entries,
    which goes against the 'don't allow conflicting entries' clause we
    discussed earlier.

 2) Where are these new entries stored? If in files, then the window
    system is going to need to read them. They are user-provided, and so
    the window system is going need to be pointed at them
    somehow. Handing data files to the window system (which may run as a
    different user) seems likely to introduce security concerns.

 3) Would we provide a protocol mechanism to update them? Or would the
    window system be expected to scan for new entries automatically?

Can we safely put this off to the future?

> We are moving away from configuring outputs in a display server by the
> connector name and towards configuring outputs based the actual output
> device identity and type.

That works where EDID is available; it's more complicated when the EDID
data is missing or corrupted (as often happens in presentation
environments). Fortunately, I think this is a problem entirely outside
the scope of the current discussion :-)

> The policy to deal with output categories and individual outputs would
> be in the display server configuration. Something like:

Oh, that's a cool idea. 

> Heh, I wasn't even aware there actually was OutputClass already, with
> only MatchDriver. Wouldn't that be a good fit for static policy
> configuration in Xorg?

I don't think so; we want it to be independent of the driver. I think,
for now, we can have a static policy implemented in the X server code
and plan on making that configurable when we figure out where we want to
configure it.

I'll go implement what we've agreed on so far and make the rest just
hard-coded for now. That way we'll have a starting point at least.

Anyone care if it's JSON or XML? I'm afraid I'm leaning to XML right now
because the X server already loads an XML parser (for DRM config files),
and I feel like introducing yet another configuration language is
probably worse than extending the use of XML? I could be convinced
either way as XML is pretty awful.

Pekka Paalanen May 9, 2017, 7:08 a.m. UTC | #36

On Mon, 08 May 2017 08:29:30 -0700
Keith Packard <[e-mail]> wrote:

> Pekka Paalanen <[e-mail]> writes:
> 
> > Thinking again, I believe we have to have a way to override
> > database entries somehow. If we ship catch-all entries for, say,
> > all Sony TVs, we are bound to get some assignment wrong for
> > someone who then wants to make a correction without breaking
> > everyone else.  
> 
> I agree this would be useful, but it raises a pile of complications:
> 
>  1) Obviously these overrides will conflict with the existing entries,
>     which goes against the 'don't allow conflicting entries' clause we
>     discussed earlier.
> 
>  2) Where are these new entries stored? If in files, then the window
>     system is going to need to read them. They are user-provided, and so
>     the window system is going need to be pointed at them
>     somehow. Handing data files to the window system (which may run as a
>     different user) seems likely to introduce security concerns.

Hi Keith,

I suppose a sysadmin could add files to the system location... in /etc,
to supplement the distributed files somewhere in /usr/share or
something?

For user-friendly settings, maybe those would need to be specific to
the display server or the DE, and we should just not spec any per-user
things. Oh, but you need Xorg use them before the DE is up... umm...
erk. :-/

>  3) Would we provide a protocol mechanism to update them? Or would the
>     window system be expected to scan for new entries automatically?

*shrug*

> Can we safely put this off to the future?

Unsure about "safely", but I think it would be the best. We have
acknowledged the issue and that's fine for now.

> > The policy to deal with output categories and individual outputs would
> > be in the display server configuration. Something like:  
> 
> Oh, that's a cool idea. 
> 
> > Heh, I wasn't even aware there actually was OutputClass already, with
> > only MatchDriver. Wouldn't that be a good fit for static policy
> > configuration in Xorg?  
> 
> I don't think so; we want it to be independent of the driver. I think,
> for now, we can have a static policy implemented in the X server code
> and plan on making that configurable when we figure out where we want to
> configure it.

I later thought about another mismatch with the current usage of
OutputClass. Since the only thing it can be used for (according to the
manual) is matching against the kernel driver and assigning the Xorg
driver, it is matching to card devices, not individual (RandR) outputs.
Right? So it's matching on the wrong level. That means we probably
cannot use the otherwise perfect name OutputClass for, well, output
classes.

> I'll go implement what we've agreed on so far and make the rest just
> hard-coded for now. That way we'll have a starting point at least.

Awesome.


Thanks,
pq

Dave Airlie May 10, 2017, 12:29 a.m. UTC | #37

>
> Just please make sure that one (user configurable/opt-in if necessary)
> policy from the beginning is to allow leasing out any output to
> applications, not just HMDs. My type of scientific/medical applications
> would benefit as soon as it has the option to get a drm lease for a given
> output on both X and Wayland based systems. That's not a theoretical future
> use case, but one i'd try to offer to my users as soon as a stable
> protocol/implementation is available in a regular Linux distribution. It
> wouldn't be fun for inexperienced users if they had to hack the database for
> every model of display they want to use, or if every untrusted user would
> have to have a root password to do so.
>

I think we need to restrict what outputs can be leased, but I might
just be paranoid,
maybe the is no security problem with having an app accessing an exclusive mode,
just makes me think of screensaver bypasses or something.

Do your apps need input? because if we lease out you won't be getting any input
from X and won't be able to open the input devices X has open.

VR is special since the input devices are all their own thing.

Dave.

Patch

diff --git a/randrproto.txt b/randrproto.txt
index 74b7c36..8dded63 100644
--- a/randrproto.txt
+++ b/randrproto.txt
@@ -1,6 +1,6 @@ 
 	       The X Resize, Rotate and Reflect Extension
-			     Version 1.5.0
-			       2015-03-14
+			     Version 1.6.0
+			       2017-04-01
 
 			      Jim Gettys
 			   [e-mail]
@@ -9,9 +9,7 @@ 
 			Hewlett Packard Company
 
 			     Keith Packard
-			[e-mail]
-		     Open Source Technology Center
-			   Intel Corporation
+			  [e-mail]
 
 1. Introduction
 
@@ -186,6 +184,24 @@  consider as single viewable areas.
 Xinerama's information now comes from the Monitors instead of directly
 from the CRTCs. The Monitor marked as Primary will be listed first.
 
+1.6. Introduction to version 1.6 of the extension
+
+Version 1.6 adds resource leasing.
+
+ • A 'Lease' is a collection of crtcs and outputs which are made
+   available to a client for direct access via kernel KMS and DRM
+   APIs. This is done by passing a suitable file descriptor back to
+   the client which has access to those resources. While leased, those
+   resources aren't used by the X server.
+
+Version 1.6 adds EDID-based output 'grabbing'.
+
+ • An 'Output Grab' matches a set of EDID Manufacturer ID and product
+   codes. For outputs with matching EDID values, the connected status
+   of the output is only visible to the grabbing client. Other clients
+   will see the output as disconnected. Attempts to configure the
+   grabbed output by other clients will fail.
+
 1.99 Acknowledgments
 
 Our thanks to the contributors to the design found on the xpert mailing
@@ -273,6 +289,10 @@  Mode
 Provider
 	A value for a PROVIDER argument does not name a defined PROVIDER.
 
+OutputGrab
+	A value for an OUTPUTGRAB argument does not name a defined
+	OUTPUTGRAB
+
 			      ❧❧❧❧❧❧❧❧❧❧❧
 
 5. Protocol Types
@@ -419,6 +439,23 @@  MONITORINFO { name: ATOM
 
 			      ❧❧❧❧❧❧❧❧❧❧❧
 
+5.7. Protocol Types added in version 1.6 of the extension
+
+OUTPUTGRAB { XID }
+
+EDIDMATCH { id: CARD16
+	     code-min: CARD16
+	     code-max: CARD16 }
+
+	These values come from the EDID specification. 'id' is the
+	Manufacturer ID value which is bytes 8 and 9 in the EDID
+	packet, stored in big endian order (MSB first). 'code-min' and
+	'code-max' define a closed-interval of Manufacturer product
+	codes, which is byte 10 and 11 of the EDID packet, stored in
+	little endian order (LSB first).
+
+			      ❧❧❧❧❧❧❧❧❧❧❧
+
 6. Extension Initialization
 
 The name of this extension is "RANDR".
@@ -1666,6 +1703,67 @@  dynamic changes in the display environment.
 	window of the screen.
 
 			      ❧❧❧❧❧❧❧❧❧❧❧
+
+7.6. Extension Requests added in version 1.6 of the extension.
+
+┌───
+    RRCreateLease
+	window : WINDOW
+	crtcs: LISTofCRTC
+	outputs: LISTofOUTPUT
+     ▶
+	nfd: CARD8
+	lease: FD
+└───
+	Errors: Window, Access, Value, CRTC, Output
+
+	Creates a new Lease for the specified crtcs and outputs from
+	the screen defined by 'window'. Returns a KMS/DRM file
+	descriptor which can control the leased objects directly
+	through the kernel. While leased, all resources will appear to
+	be 'useless' to clients other than the leasing client as
+	follows:
+
+	• Crtcs are reported as having no 'possible-outputs' and all
+	  other values reported as if the crtc were disabled.
+
+	• Outputs are reported as having no crtcs they can be
+	  connected to, no clones they can share a crtc with, will
+	  report a connection status of Disconnected, and will show
+	  the current crtc as Disabled.
+
+	The lease remains in effect until the file descriptor is
+	closed, even if the client holding the lease disconnects from
+	the X server.
+
+	Returns an Access error if any of the named resources are in
+	use or already leased to another client.
+
+	Returns a Match error if any of the named resources are in use
+	by the X server.
+
+┌───
+    RRCreateOutputGrab
+	window : WINDOW
+	outputgrab: OUTPUTGRAB
+	matches: LISTofEDIDMATCH
+└───
+	Errors: Window, Access, Value, CRTC, Output
+
+	Creates an Output Grab with the specified ID on the screen
+	associated with 'window'. Any output containing an EDID
+	property matching the grab will appear to be disconnected to
+	all clients other than the grabbing client.
+
+┌───
+    RRDestroyOutputGrab
+	outputgrab: OUTPUTGRAB
+└───
+	Errors: OutputGrab
+
+	Destroys the named OUTPUTGRAB.
+
+			      ❧❧❧❧❧❧❧❧❧❧❧
 8. Extension Events
 
 Clients MAY select for ConfigureNotify on the root window to be