| Message ID | 20230110091707.2003226-1-sakari.ailus@linux.intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [1/1] media: Documentation: Update documentation for streams | expand |
On 10/01/2023 11:17, Sakari Ailus wrote: > Document how streams interacts with formats and selections. > > Update documentation in respect to what is allowed, in particular streams > are only supported via full routes, source-only routes are not supported > right now. > > The centerpiece of the API additions are streams. Albeit routes are > configured via S_ROUTING IOCTL that also declares streams, it is streams > that are accessed through other APIs. Thus refer to streams instead of > routes in documentation. > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com> > --- > Hi folks, > > This replaces my earlier documetation patch. The commit message and the > subject have changed and the content reflects this. Largely this means > removing a few features of the interface --- for now. > > The intent is to be able to merge this very soon, thus those portions that > are still debated have been dropped (no more dependencies between streams, > for instance). > > .../userspace-api/media/v4l/dev-subdev.rst | 121 +++++++++--------- > 1 file changed, 58 insertions(+), 63 deletions(-) > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst > index 072e82b2b2786..d2badf21a62cd 100644 > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst > @@ -406,6 +406,8 @@ pixel array is not rectangular but cross-shaped or round. The maximum > size may also be smaller than the BOUNDS rectangle. > > > +.. _format-propagation: > + > Order of configuration and format propagation > --------------------------------------------- > > @@ -507,12 +509,12 @@ source pads. > Streams, multiplexed media pads and internal routing > ---------------------------------------------------- > > -Commonly V4L2 subdevices support only separate video streams, that is, only a > -single stream can pass through a media link and a media pad. Thus each pad > -contains a format configuration for that single stream. In some cases a subdev > -can do stream processing and split a stream into two or compose two streams > -into one, but the inputs and outputs for the subdev are still a single stream > -per pad. > +Simple V4L2 subdevices do not support multiple, unrelated video streams, > +and only a single stream can pass through a media link and a media pad. > +Thus each pad contains a format and selection configuration for that > +single stream. A subdev can do stream processing and split a stream into > +two or compose two streams into one, but the inputs and outputs for the > +subdev are still a single stream per pad. > > Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple > data streams are transmitted on the same bus, which is represented by a media > @@ -539,14 +541,35 @@ streams from one end of the link to the other, and subdevices have routing > tables which describe how the incoming streams from sink pads are routed to the > source pads. > > -A stream ID (often just "stream") is a media link-local identifier for a stream. > -In other words, a particular stream ID must exist on both sides of a media > +A stream ID is a media pad-local identifier for a stream. Streams IDs of > +the same stream must be equal on both ends of a link. In other words, > +a particular stream ID must exist on both sides of a media > link, but another stream ID can be used for the same stream at the other side > -of the subdevice. > +of the subdevice. The same stream ID is used to refer to the stream on > +both pads of the link on all ioctls operating on pads. This sentence feels a bit confusing. What's it trying to say? That the same Stream ID has to be on the pads on both sides of the link? Wasn't that already covered earlier? > +A stream at a specific point in the media pipeline is identified by the > +sub-devdev and a (pad, stream) pair. For subdevices that do not support Typo there in "sub-devdev". Also, there seems to be a lot of "sub-device" and "subdevice" words in the text. Which one is it? > +multiplexed streams the 'stream' field is always 0. > + > +Interaction between routes, streams, formats and selections > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > + > +The addition of streams to the V4L2 sub-device interface moves the sub-device > +formats and selections from pads to (pad, stream) pairs. Besides the > +usual pad, also the stream ID needs to be provided for setting formats and Here, and already earlier, you first say "stream" and then "stream ID". Are they the same thing? You removed my "stream ID (often just "stream")" text, which I had included to clarify that those two terms are interchangeable in certain contexts where it's obvious. > +selections. The order of configuring formats and selections along a stream is > +the same as without streams (see :ref:`format-propagation`). > + > +Instead of the sub-device wide merging of streams from all source pads > +towards all sink pads, data flows for each route are separate from each Shouldn't the above be from sink pads towards source pads? > +other. Any number of routes from streams on sink pads towards streams on > +source pads is allowed, to the extent supported by drivers. For every > +stream on a sink pad, however, only a single route is allowed. I don't follow here. Don't you first say that any number of routes is ok, then you say only a single route is ok? > -A stream at a specific point in the media pipeline is identified with the > -subdev and a (pad, stream) pair. For subdevices that do not support > -multiplexed streams the 'stream' is always 0. > +Any configurations of a stream within a pad, such as format or selections, > +are independent of similar configurations on other streams. This is > +subject to change in the future. Hmm, what does this mean? Isn't this device specific? The format for a video stream will affect the format for a metadata stream from the same source. > Configuring streams > ^^^^^^^^^^^^^^^^^^^ > @@ -560,34 +583,37 @@ There are three steps in configuring the streams: > 1) Set up links. Connect the pads between subdevices using the :ref:`Media > Controller API <media_controller>` > > -2) Routing. The routing table for the subdevice must be set with > +2) Streams. Streams are declared and their routing is configured by > +setting the routing table for the subdevice must be set with Hmm, maybe s/must be set with/using/ > :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that > -setting the routing table will reset all the stream configurations in a media > -entity. > +setting the routing table will reset formats and selections in the > +sub-device to default values. > > -3) Configure streams. Each route endpoint must be configured > -with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`. > +3) Configure formats and selections. Formats and selections of each stream > +are configured separately as documented for plain subdevices in > +:ref:`<format-propagation>`. The stream ID is set to the same stream ID > +associated with either sink or source pads of routes configured using the > +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Is this trying to say that the stream ID numbers used in the VIDIOC_SUBDEV_G_ROUTING call are the same used in the format and selection configuration? Isn't that obvious already? I at least find the sentence a bit confusing. > Multiplexed streams setup example > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > A simple example of a multiplexed stream setup might be as follows: > > -- Two identical sensors (Sensor A and Sensor B). Each sensor has a single source > - pad (pad 0) which carries two streams, pixel data stream and metadata > - stream. > +- Two identical sensors (Sensor A and Sensor B). The sensors produce image > + data only and hence do not need specific support for streams. Hmm, I have already changed all this in the v16, and I think I had other changes in the docs too. What's this patch based on? > - Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the > - sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4 > - streams. > + sensors (pads 0, 1), and one source pad (pad 2), which outputs both of > + the streams. > > - Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0), > connected to the bridge, and four source pads (pads 1-4), going to the DMA > - engine. The receiver demultiplexes the incoming streams to the four source > + engine. The receiver demultiplexes the incoming streams to two the source > pads. > > - Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a > - single source pad in the receiver. > + single source pad in the receive via a link, two of which are active. > > The sensors, the bridge and the receiver are modeled as V4L2 subdevices, > exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are > @@ -599,23 +625,7 @@ To configure this pipeline, the userspace must take the following steps: > bridge to the receiver, and the receiver to the DMA engines. This step does > not differ from normal non-multiplexed media controller setup. > > -2) Configure routing. > - > -.. flat-table:: Sensor routing table (identical on both sensors) > - :header-rows: 1 > - > - * - Sink Pad/Stream > - - Source Pad/Stream > - - Routing Flags > - - Comments > - * - 0/0 (unused) > - - 0/0 > - - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE_ONLY > - - Pixel data stream. Source route, i.e. the sink fields are unused. > - * - 0/0 (unused) > - - 0/1 > - - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE_ONLY > - - Metadata stream. Source route, i.e. the sink fields are unused. > +2) Configure routing > > .. flat-table:: Bridge routing table > :header-rows: 1 > @@ -628,18 +638,10 @@ not differ from normal non-multiplexed media controller setup. > - 2/0 > - V4L2_SUBDEV_ROUTE_FL_ACTIVE > - Pixel data stream from Sensor A > - * - 0/1 > - - 2/1 > - - V4L2_SUBDEV_ROUTE_FL_ACTIVE > - - Metadata stream from Sensor A > * - 1/0 > - - 2/2 > + - 2/1 > - V4L2_SUBDEV_ROUTE_FL_ACTIVE > - Pixel data stream from Sensor B > - * - 1/1 > - - 2/3 > - - V4L2_SUBDEV_ROUTE_FL_ACTIVE > - - Metadata stream from Sensor B > > .. flat-table:: Receiver routing table > :header-rows: 1 > @@ -655,22 +657,15 @@ not differ from normal non-multiplexed media controller setup. > * - 0/1 > - 2/0 > - V4L2_SUBDEV_ROUTE_FL_ACTIVE > - - Metadata stream from Sensor A > - * - 0/2 > - - 3/0 > - - V4L2_SUBDEV_ROUTE_FL_ACTIVE > - Pixel data stream from Sensor B > - * - 0/3 > - - 4/0 > - - V4L2_SUBDEV_ROUTE_FL_ACTIVE > - - Metadata stream from Sensor B > > -3) Configure streams > +3) Configure formats and selections > > -After configuring the routing table, the next step is configuring the streams. > -This step is similar to configuring the pads in a non-multiplexed streams > -setup, with the difference that we need to configure each (pad, stream) pair > -(i.e. route endpoint) instead of just a pad. > +After configuring the routing table, the next step is configuring the > +formats and selections for the streams. This step is similar to > +configuring the pads in a non-multiplexed streams setup, with the As you say "configuring the formats and selections for the streams", maybe similarly you could say "is similar to configuring the formats and selections for the pads". > +difference that we need to configure each (pad, stream) pair instead of > +just a pad. Do we "configure the streams/pads", or do we "configure the formats and selections for the streams/pads"? Tomi
Moi, On Tue, Jan 10, 2023 at 01:26:06PM +0200, Tomi Valkeinen wrote: > On 10/01/2023 11:17, Sakari Ailus wrote: > > Document how streams interacts with formats and selections. > > > > Update documentation in respect to what is allowed, in particular streams > > are only supported via full routes, source-only routes are not supported > > right now. > > > > The centerpiece of the API additions are streams. Albeit routes are > > configured via S_ROUTING IOCTL that also declares streams, it is streams > > that are accessed through other APIs. Thus refer to streams instead of > > routes in documentation. > > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com> > > --- > > Hi folks, > > > > This replaces my earlier documetation patch. The commit message and the > > subject have changed and the content reflects this. Largely this means > > removing a few features of the interface --- for now. > > > > The intent is to be able to merge this very soon, thus those portions that > > are still debated have been dropped (no more dependencies between streams, > > for instance). > > > > .../userspace-api/media/v4l/dev-subdev.rst | 121 +++++++++--------- > > 1 file changed, 58 insertions(+), 63 deletions(-) > > > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst > > index 072e82b2b2786..d2badf21a62cd 100644 > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst > > @@ -406,6 +406,8 @@ pixel array is not rectangular but cross-shaped or round. The maximum > > size may also be smaller than the BOUNDS rectangle. > > +.. _format-propagation: > > + > > Order of configuration and format propagation > > --------------------------------------------- > > @@ -507,12 +509,12 @@ source pads. > > Streams, multiplexed media pads and internal routing > > ---------------------------------------------------- > > -Commonly V4L2 subdevices support only separate video streams, that is, only a > > -single stream can pass through a media link and a media pad. Thus each pad > > -contains a format configuration for that single stream. In some cases a subdev > > -can do stream processing and split a stream into two or compose two streams > > -into one, but the inputs and outputs for the subdev are still a single stream > > -per pad. > > +Simple V4L2 subdevices do not support multiple, unrelated video streams, > > +and only a single stream can pass through a media link and a media pad. > > +Thus each pad contains a format and selection configuration for that > > +single stream. A subdev can do stream processing and split a stream into > > +two or compose two streams into one, but the inputs and outputs for the > > +subdev are still a single stream per pad. > > Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple > > data streams are transmitted on the same bus, which is represented by a media > > @@ -539,14 +541,35 @@ streams from one end of the link to the other, and subdevices have routing > > tables which describe how the incoming streams from sink pads are routed to the > > source pads. > > -A stream ID (often just "stream") is a media link-local identifier for a stream. > > -In other words, a particular stream ID must exist on both sides of a media > > +A stream ID is a media pad-local identifier for a stream. Streams IDs of > > +the same stream must be equal on both ends of a link. In other words, > > +a particular stream ID must exist on both sides of a media > > link, but another stream ID can be used for the same stream at the other side > > -of the subdevice. > > +of the subdevice. The same stream ID is used to refer to the stream on > > +both pads of the link on all ioctls operating on pads. > > This sentence feels a bit confusing. What's it trying to say? That the same > Stream ID has to be on the pads on both sides of the link? Wasn't that > already covered earlier? After reading the paragraph again, I think the last sentence can be dropped indeed. > > > +A stream at a specific point in the media pipeline is identified by the > > +sub-devdev and a (pad, stream) pair. For subdevices that do not support > > Typo there in "sub-devdev". Also, there seems to be a lot of "sub-device" > and "subdevice" words in the text. Which one is it? It's intended to be "sub-device". Sometimes it is shortedned to "subdev", but this is documentation so I'd avoid it here. "Subdevice" seems to be only present on routing documentation. I'll replace all instances with "sub-device". > > > +multiplexed streams the 'stream' field is always 0. > > + > > +Interaction between routes, streams, formats and selections > > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > + > > +The addition of streams to the V4L2 sub-device interface moves the sub-device > > +formats and selections from pads to (pad, stream) pairs. Besides the > > +usual pad, also the stream ID needs to be provided for setting formats and > > Here, and already earlier, you first say "stream" and then "stream ID". Are > they the same thing? You removed my "stream ID (often just "stream")" text, > which I had included to clarify that those two terms are interchangeable in > certain contexts where it's obvious. It says above that "A stream ID is a media pad-local identifier for a stream". I think (pad, stream) pair is fine --- we don't talk about pad IDs either. > > > +selections. The order of configuring formats and selections along a stream is > > +the same as without streams (see :ref:`format-propagation`). > > + > > +Instead of the sub-device wide merging of streams from all source pads > > +towards all sink pads, data flows for each route are separate from each > > Shouldn't the above be from sink pads towards source pads? Uh, yes. I'll fix for v2. > > > +other. Any number of routes from streams on sink pads towards streams on > > +source pads is allowed, to the extent supported by drivers. For every > > +stream on a sink pad, however, only a single route is allowed. > > I don't follow here. Don't you first say that any number of routes is ok, > then you say only a single route is ok? The latter "sink" should have been "source". I'll fix it. > > > -A stream at a specific point in the media pipeline is identified with the > > -subdev and a (pad, stream) pair. For subdevices that do not support > > -multiplexed streams the 'stream' is always 0. > > +Any configurations of a stream within a pad, such as format or selections, > > +are independent of similar configurations on other streams. This is > > +subject to change in the future. > > Hmm, what does this mean? Isn't this device specific? The format for a video > stream will affect the format for a metadata stream from the same source. Correct, but we don't support these yet. They should of course follow soon after the initial merge. > > > Configuring streams > > ^^^^^^^^^^^^^^^^^^^ > > @@ -560,34 +583,37 @@ There are three steps in configuring the streams: > > 1) Set up links. Connect the pads between subdevices using the :ref:`Media > > Controller API <media_controller>` > > -2) Routing. The routing table for the subdevice must be set with > > +2) Streams. Streams are declared and their routing is configured by > > +setting the routing table for the subdevice must be set with > > Hmm, maybe s/must be set with/using/ Yes. > > > :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that > > -setting the routing table will reset all the stream configurations in a media > > -entity. > > +setting the routing table will reset formats and selections in the > > +sub-device to default values. > > -3) Configure streams. Each route endpoint must be configured > > -with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`. > > +3) Configure formats and selections. Formats and selections of each stream > > +are configured separately as documented for plain subdevices in > > +:ref:`<format-propagation>`. The stream ID is set to the same stream ID > > +associated with either sink or source pads of routes configured using the > > +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. > > Is this trying to say that the stream ID numbers used in the > VIDIOC_SUBDEV_G_ROUTING call are the same used in the format and selection > configuration? Isn't that obvious already? I at least find the sentence a > bit confusing. > > > Multiplexed streams setup example > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > A simple example of a multiplexed stream setup might be as follows: > > -- Two identical sensors (Sensor A and Sensor B). Each sensor has a single source > > - pad (pad 0) which carries two streams, pixel data stream and metadata > > - stream. > > +- Two identical sensors (Sensor A and Sensor B). The sensors produce image > > + data only and hence do not need specific support for streams. > > Hmm, I have already changed all this in the v16, and I think I had other > changes in the docs too. What's this patch based on? As discussed, this patch was based on your streams/work-v16 branch. I'll rebase this on the last patch of the set, which also means removing this and related changes. > > > - Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the > > - sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4 > > - streams. > > + sensors (pads 0, 1), and one source pad (pad 2), which outputs both of > > + the streams. > > - Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0), > > connected to the bridge, and four source pads (pads 1-4), going to the DMA > > - engine. The receiver demultiplexes the incoming streams to the four source > > + engine. The receiver demultiplexes the incoming streams to two the source > > pads. > > - Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a > > - single source pad in the receiver. > > + single source pad in the receive via a link, two of which are active. > > The sensors, the bridge and the receiver are modeled as V4L2 subdevices, > > exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are > > @@ -599,23 +625,7 @@ To configure this pipeline, the userspace must take the following steps: > > bridge to the receiver, and the receiver to the DMA engines. This step does > > not differ from normal non-multiplexed media controller setup. > > -2) Configure routing. > > - > > -.. flat-table:: Sensor routing table (identical on both sensors) > > - :header-rows: 1 > > - > > - * - Sink Pad/Stream > > - - Source Pad/Stream > > - - Routing Flags > > - - Comments > > - * - 0/0 (unused) > > - - 0/0 > > - - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE_ONLY > > - - Pixel data stream. Source route, i.e. the sink fields are unused. > > - * - 0/0 (unused) > > - - 0/1 > > - - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE_ONLY > > - - Metadata stream. Source route, i.e. the sink fields are unused. > > +2) Configure routing > > .. flat-table:: Bridge routing table > > :header-rows: 1 > > @@ -628,18 +638,10 @@ not differ from normal non-multiplexed media controller setup. > > - 2/0 > > - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > - Pixel data stream from Sensor A > > - * - 0/1 > > - - 2/1 > > - - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > - - Metadata stream from Sensor A > > * - 1/0 > > - - 2/2 > > + - 2/1 > > - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > - Pixel data stream from Sensor B > > - * - 1/1 > > - - 2/3 > > - - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > - - Metadata stream from Sensor B > > .. flat-table:: Receiver routing table > > :header-rows: 1 > > @@ -655,22 +657,15 @@ not differ from normal non-multiplexed media controller setup. > > * - 0/1 > > - 2/0 > > - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > - - Metadata stream from Sensor A > > - * - 0/2 > > - - 3/0 > > - - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > - Pixel data stream from Sensor B > > - * - 0/3 > > - - 4/0 > > - - V4L2_SUBDEV_ROUTE_FL_ACTIVE > > - - Metadata stream from Sensor B > > -3) Configure streams > > +3) Configure formats and selections > > -After configuring the routing table, the next step is configuring the streams. > > -This step is similar to configuring the pads in a non-multiplexed streams > > -setup, with the difference that we need to configure each (pad, stream) pair > > -(i.e. route endpoint) instead of just a pad. > > +After configuring the routing table, the next step is configuring the > > +formats and selections for the streams. This step is similar to > > +configuring the pads in a non-multiplexed streams setup, with the > > As you say "configuring the formats and selections for the streams", maybe > similarly you could say "is similar to configuring the formats and > selections for the pads". We haven't referred the non-stream aware configuration as being specifically performed on pads. I think this should be rewritten, not just slightly adjusted. How about: After configuring routing, the next step is configuring the formats and selections for the streams. This is similar to performing this step without streams, with just one exception: the ``stream`` field needs to be assigned to the value of the stream ID. > > > +difference that we need to configure each (pad, stream) pair instead of > > +just a pad. > > Do we "configure the streams/pads", or do we "configure the formats and > selections for the streams/pads"? > > Tomi >
diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst index 072e82b2b2786..d2badf21a62cd 100644 --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst @@ -406,6 +406,8 @@ pixel array is not rectangular but cross-shaped or round. The maximum size may also be smaller than the BOUNDS rectangle. +.. _format-propagation: + Order of configuration and format propagation --------------------------------------------- @@ -507,12 +509,12 @@ source pads. Streams, multiplexed media pads and internal routing ---------------------------------------------------- -Commonly V4L2 subdevices support only separate video streams, that is, only a -single stream can pass through a media link and a media pad. Thus each pad -contains a format configuration for that single stream. In some cases a subdev -can do stream processing and split a stream into two or compose two streams -into one, but the inputs and outputs for the subdev are still a single stream -per pad. +Simple V4L2 subdevices do not support multiple, unrelated video streams, +and only a single stream can pass through a media link and a media pad. +Thus each pad contains a format and selection configuration for that +single stream. A subdev can do stream processing and split a stream into +two or compose two streams into one, but the inputs and outputs for the +subdev are still a single stream per pad. Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple data streams are transmitted on the same bus, which is represented by a media @@ -539,14 +541,35 @@ streams from one end of the link to the other, and subdevices have routing tables which describe how the incoming streams from sink pads are routed to the source pads. -A stream ID (often just "stream") is a media link-local identifier for a stream. -In other words, a particular stream ID must exist on both sides of a media +A stream ID is a media pad-local identifier for a stream. Streams IDs of +the same stream must be equal on both ends of a link. In other words, +a particular stream ID must exist on both sides of a media link, but another stream ID can be used for the same stream at the other side -of the subdevice. +of the subdevice. The same stream ID is used to refer to the stream on +both pads of the link on all ioctls operating on pads. + +A stream at a specific point in the media pipeline is identified by the +sub-devdev and a (pad, stream) pair. For subdevices that do not support +multiplexed streams the 'stream' field is always 0. + +Interaction between routes, streams, formats and selections +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The addition of streams to the V4L2 sub-device interface moves the sub-device +formats and selections from pads to (pad, stream) pairs. Besides the +usual pad, also the stream ID needs to be provided for setting formats and +selections. The order of configuring formats and selections along a stream is +the same as without streams (see :ref:`format-propagation`). + +Instead of the sub-device wide merging of streams from all source pads +towards all sink pads, data flows for each route are separate from each +other. Any number of routes from streams on sink pads towards streams on +source pads is allowed, to the extent supported by drivers. For every +stream on a sink pad, however, only a single route is allowed. -A stream at a specific point in the media pipeline is identified with the -subdev and a (pad, stream) pair. For subdevices that do not support -multiplexed streams the 'stream' is always 0. +Any configurations of a stream within a pad, such as format or selections, +are independent of similar configurations on other streams. This is +subject to change in the future. Configuring streams ^^^^^^^^^^^^^^^^^^^ @@ -560,34 +583,37 @@ There are three steps in configuring the streams: 1) Set up links. Connect the pads between subdevices using the :ref:`Media Controller API <media_controller>` -2) Routing. The routing table for the subdevice must be set with +2) Streams. Streams are declared and their routing is configured by +setting the routing table for the subdevice must be set with :ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that -setting the routing table will reset all the stream configurations in a media -entity. +setting the routing table will reset formats and selections in the +sub-device to default values. -3) Configure streams. Each route endpoint must be configured -with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`. +3) Configure formats and selections. Formats and selections of each stream +are configured separately as documented for plain subdevices in +:ref:`<format-propagation>`. The stream ID is set to the same stream ID +associated with either sink or source pads of routes configured using the +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Multiplexed streams setup example ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A simple example of a multiplexed stream setup might be as follows: -- Two identical sensors (Sensor A and Sensor B). Each sensor has a single source - pad (pad 0) which carries two streams, pixel data stream and metadata - stream. +- Two identical sensors (Sensor A and Sensor B). The sensors produce image + data only and hence do not need specific support for streams. - Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the - sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4 - streams. + sensors (pads 0, 1), and one source pad (pad 2), which outputs both of + the streams. - Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0), connected to the bridge, and four source pads (pads 1-4), going to the DMA - engine. The receiver demultiplexes the incoming streams to the four source + engine. The receiver demultiplexes the incoming streams to two the source pads. - Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a - single source pad in the receiver. + single source pad in the receive via a link, two of which are active. The sensors, the bridge and the receiver are modeled as V4L2 subdevices, exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are @@ -599,23 +625,7 @@ To configure this pipeline, the userspace must take the following steps: bridge to the receiver, and the receiver to the DMA engines. This step does not differ from normal non-multiplexed media controller setup. -2) Configure routing. - -.. flat-table:: Sensor routing table (identical on both sensors) - :header-rows: 1 - - * - Sink Pad/Stream - - Source Pad/Stream - - Routing Flags - - Comments - * - 0/0 (unused) - - 0/0 - - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE_ONLY - - Pixel data stream. Source route, i.e. the sink fields are unused. - * - 0/0 (unused) - - 0/1 - - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE_ONLY - - Metadata stream. Source route, i.e. the sink fields are unused. +2) Configure routing .. flat-table:: Bridge routing table :header-rows: 1 @@ -628,18 +638,10 @@ not differ from normal non-multiplexed media controller setup. - 2/0 - V4L2_SUBDEV_ROUTE_FL_ACTIVE - Pixel data stream from Sensor A - * - 0/1 - - 2/1 - - V4L2_SUBDEV_ROUTE_FL_ACTIVE - - Metadata stream from Sensor A * - 1/0 - - 2/2 + - 2/1 - V4L2_SUBDEV_ROUTE_FL_ACTIVE - Pixel data stream from Sensor B - * - 1/1 - - 2/3 - - V4L2_SUBDEV_ROUTE_FL_ACTIVE - - Metadata stream from Sensor B .. flat-table:: Receiver routing table :header-rows: 1 @@ -655,22 +657,15 @@ not differ from normal non-multiplexed media controller setup. * - 0/1 - 2/0 - V4L2_SUBDEV_ROUTE_FL_ACTIVE - - Metadata stream from Sensor A - * - 0/2 - - 3/0 - - V4L2_SUBDEV_ROUTE_FL_ACTIVE - Pixel data stream from Sensor B - * - 0/3 - - 4/0 - - V4L2_SUBDEV_ROUTE_FL_ACTIVE - - Metadata stream from Sensor B -3) Configure streams +3) Configure formats and selections -After configuring the routing table, the next step is configuring the streams. -This step is similar to configuring the pads in a non-multiplexed streams -setup, with the difference that we need to configure each (pad, stream) pair -(i.e. route endpoint) instead of just a pad. +After configuring the routing table, the next step is configuring the +formats and selections for the streams. This step is similar to +configuring the pads in a non-multiplexed streams setup, with the +difference that we need to configure each (pad, stream) pair instead of +just a pad. A common way to accomplish this is to start from the sensors and propagate the configurations along the stream towards the receiver,
Document how streams interacts with formats and selections. Update documentation in respect to what is allowed, in particular streams are only supported via full routes, source-only routes are not supported right now. The centerpiece of the API additions are streams. Albeit routes are configured via S_ROUTING IOCTL that also declares streams, it is streams that are accessed through other APIs. Thus refer to streams instead of routes in documentation. Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com> --- Hi folks, This replaces my earlier documetation patch. The commit message and the subject have changed and the content reflects this. Largely this means removing a few features of the interface --- for now. The intent is to be able to merge this very soon, thus those portions that are still debated have been dropped (no more dependencies between streams, for instance). .../userspace-api/media/v4l/dev-subdev.rst | 121 +++++++++--------- 1 file changed, 58 insertions(+), 63 deletions(-)