[v2,1/1] virtio-video: Add virtio video device specification
diff mbox series

Message ID 20191218130214.170703-2-keiichiw@chromium.org
State New
Headers show
Series
  • VirtIO video device specification
Related show

Commit Message

Keiichi Watanabe Dec. 18, 2019, 1:02 p.m. UTC
From: Dmitry Sepp <dmitry.sepp@opensynergy.com>

The virtio video encoder device and decoder device provide functionalities to
encode and decode video stream respectively.
Though video encoder and decoder are provided as different devices, they use a
same protocol.

Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
---
 content.tex      |   1 +
 virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 580 insertions(+)
 create mode 100644 virtio-video.tex

Comments

Gerd Hoffmann Dec. 18, 2019, 1:40 p.m. UTC | #1
Hi,

> +The device MUST mark the last buffer with the
> +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> +sequence.

No, that would build a race condition into the protocol.  The device
could complete the last buffer after the driver has sent the drain
command but before the device saw it.  So the flag would not be
reliable.

I also can't see why the flag is needed in the first place.  The driver
should know which buffers are queued still and be able to figure
whenever the drain is complete or not without depending on that flag.
So I'd suggest to simply drop it.

That is the only issue I've spotted in the protocol on a first quick
look.  There are a few places where the spec text could be improved.
I'll try to set aside some time to go over this in detail, but I can't
promise I'll find the time to do that before xmas and new year.

cheers,
  Gerd
Tomasz Figa Dec. 18, 2019, 2:08 p.m. UTC | #2
On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>   Hi,
>
> > +The device MUST mark the last buffer with the
> > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > +sequence.
>
> No, that would build a race condition into the protocol.  The device
> could complete the last buffer after the driver has sent the drain
> command but before the device saw it.  So the flag would not be
> reliable.
>
> I also can't see why the flag is needed in the first place.  The driver
> should know which buffers are queued still and be able to figure
> whenever the drain is complete or not without depending on that flag.
> So I'd suggest to simply drop it.

Unfortunately video decoders are not that simple. There are always
going to be some buffers on the decoder side used as reference frames.
Only the decoder knows when to release them, as it continues decoding
the stream. Moreover, with certain features of certain codecs (e.g.
framebuffer reordering in H.264), there could be decoded frames that
need to be held by decoder, because later bitstream may contain
earlier frames.

How we defined this in the V4L2 stateful decoder interface is that if
the decoder happened to return the last framebuffer before the drain
request arrived, it would return one more, empty. From our experience
that was the easiest thing to deal with from both the driver and the
application, so I believe the same should apply to the host device
implementation and the guest driver.

>
> That is the only issue I've spotted in the protocol on a first quick
> look.  There are a few places where the spec text could be improved.
> I'll try to set aside some time to go over this in detail, but I can't
> promise I'll find the time to do that before xmas and new year.

Thanks a lot for review.

Best regards,
Tomasz
Frediano Ziglio Dec. 18, 2019, 5:29 p.m. UTC | #3
> 
> From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> 
> The virtio video encoder device and decoder device provide functionalities to
> encode and decode video stream respectively.
> Though video encoder and decoder are provided as different devices, they use
> a
> same protocol.
> 
> Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> ---
>  content.tex      |   1 +
>  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 580 insertions(+)
>  create mode 100644 virtio-video.tex
> 
> diff --git a/content.tex b/content.tex
> index 556b373..9e56839 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> Requirements}\label{sec:Device
>  \input{virtio-vsock.tex}
>  \input{virtio-fs.tex}
>  \input{virtio-rpmb.tex}
> +\input{virtio-video.tex}
>  
>  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>  
> diff --git a/virtio-video.tex b/virtio-video.tex
> new file mode 100644
> index 0000000..30e728d
> --- /dev/null
> +++ b/virtio-video.tex
> @@ -0,0 +1,579 @@
> +\section{Video Device}\label{sec:Device Types / Video Device}
> +
> +The virtio video encoder device and decoder device are virtual devices that
> +supports encoding and decoding respectively. Though the encoder and the
> decoder
> +are different devices, they use the same protocol.
> +
> +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> +
> +\begin{description}
> +\item[30] encoder device
> +\item[31] decoder device
> +\end{description}
> +
> +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> +
> +\begin{description}
> +\item[0] controlq - queue for sending control commands.
> +\item[1] eventq - queue for sending events happened in the device.
> +\end{description}
> +
> +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> bits}
> +
> +\begin{description}
> +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> video
> +  buffers.
> +\end{description}
> +
> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device /
> Feature bits}
> +
> +The device MUST offer at least one of feature bits.
> +
> +\subsection{Device configuration layout}\label{sec:Device Types / Video
> Device / Device configuration layout}
> +
> +Video device configuration uses the following layout structure:
> +
> +\begin{lstlisting}
> +struct virtio_video_config {
> +        le32 max_cap_len;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> +  MUST set this value.
> +\end{description}
> +
> +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> Device Initialization}
> +
> +\devicenormative{\subsubsection}{Device Initialization}{Device Types / Video
> Device / Device Initialization}
> +
> +The driver SHOULD query device capability by using the
> +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> +setup.
> +
> +\subsection{Device Operation}\label{sec:Device Types / Video Device / Device
> Operation}
> +
> +The driver allocates input and output buffers and queues the buffers
> +to the device. The device performs operations on the buffers according
> +to the function in question.
> +
> +\subsubsection{Device Operation: Create stream}
> +
> +To process buffers, the device needs to associate them with a certain
> +video stream (essentially, a context). Streams are created by
> +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> +determined by the device.
> +
> +\subsubsection{Device Operation: Create buffers}
> +
> +Buffers are used to store the actual data as well as the relevant
> +metadata. Scatter lists are supported, so the buffer doesn't need to
> +be contiguous in guest physical memory.
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> +  resource that is backed by a buffer allocated from the driver's
> +  memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> +  is no longer needed.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Stream parameter control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters for
> +  input and output streams from the device.
> +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> +  device.
> +\item After setting stream parameters, the driver may issue
> +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> be
> +  changed implicitly by the device during the set operation.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Process buffers}
> +
> +\begin{itemize*}
> +\item If the function and the buffer type require so, write data to
> +the buffer memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> +processing in the device.
> +\item The request completes asynchronously when the device has
> +finished with the buffer.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Buffer processing control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> +  return all of the already queued buffers.
> +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> +  already queued buffers from the input or the output queue. This also
> +  includes input or output buffers that can be currently owned by the
> +  device's processing pipeline.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Asynchronous events}
> +
> +While processing buffers, the device can send asynchronous event
> +notifications to the driver. The behaviour depends on the exact
> +stream. For example, the decoder device sends a resolution change
> +event when it encounters new resolution metadata in the stream.
> +
> +\subsubsection{Device Operation: Request header}
> +
> +All requests and responses on the control virt queue have a fixed
> +header using the following layout structure and definitions:
> +
> +\begin{lstlisting}
> +enum virtio_video_ctrl_type {
> +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> +
> +        /* request */
> +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> +        VIRTIO_VIDEO_T_STREAM_CREATE,
> +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> +        VIRTIO_VIDEO_T_SET_PARAMS,
> +        VIRTIO_VIDEO_T_GET_PARAMS,
> +
> +        /* response */
> +        VIRTIO_VIDEO_S_OK = 0x0200,
> +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> +
> +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> +};
> +
> +struct virtio_video_ctrl_hdr {
> +        le32 type;
> +        le32 stream_id;
> +        le32 len; /* Length of the structure in bytes. */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{type}] is the type of the driver request or the device
> +response.
> +\item[\field{stream_id}] specifies a target stream.
> +\item[\field{len}] is the length of data in bytes, which includes
> +length of the header.

I suppose is implicit that it's the size of the container structure.

> +\end{description}
> +
> +\subsubsection{Device Operation: controlq}
> +
> +\begin{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> +supported formats.
> +
> +The driver uses \field{struct virtio_video_get_capability} to send a
> +query request.
> +
> +\begin{lstlisting}
> +enum virtio_video_buf_type {
> +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> +};
> +
> +struct virtio_video_get_capability {
> +        struct virtio_video_ctrl_hdr hdr;
> +        enum virtio_video_buf_type buf_type;

Here you used an enumeration but later you are using just le32 type
with a comment. Why this difference?

> +};
> +\end{lstlisting}
> +\begin{description}
> +\item[\field{buf_type}] is the buffer type that the driver asks
> +information about. The driver MUST set either
> +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}.
> +\end{description}
> +
> +The device responds a capability by using \field{struct
> +virtio_video_get_capability_resp}.
> +\begin{lstlisting}
> +enum virtio_video_format {
> +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,

Really minor: maybe you want to insert an empty line to be coherent
with other enumerations?

> +        /* Raw formats */
> +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> +        VIRTIO_VIDEO_FORMAT_YUV420,
> +        VIRTIO_VIDEO_FORMAT_YVU420,
> +
> +        /* Compressed formats */
> +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> +};
> +
> +enum virtio_video_profile {
> +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> +
> +        /* H.264 */
> +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> VIRTIO_VIDEO_PROFILE_H264_BASELINE,

Maybe you want

VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_MIN,

like others?

> +        VIRTIO_VIDEO_PROFILE_H264_MAIN,
> +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +
> +        /* VP8 */
> +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> +
> +        /* VP9 */
> +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +};
> +
> +struct virtio_video_format_range {
> +        le32 min;
> +        le32 max;
> +        le32 step;
> +        u8 paddings[4];

Here you padded at 8-byte like many other structures but not for
virtio_video_ctrl_hdr. Is this expected?

> +};
> +
> +struct virtio_video_format_desc {
> +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> +        le64 mask;

This will limit the formats to 64 while "num_descs" below is a
32 bit.

> +        struct virtio_video_format_range width;
> +        struct virtio_video_format_range height;
> +        le32 num_rates;
> +        u8 padding[4];
> +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> +};
> +
> +struct virtio_video_get_capability_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 num_descs;
> +        /* Followed by struct virtio_video_format_desc desc[] */
> +};
> +\end{lstlisting}
> +
> +The format description \field{struct virtio_video_format_desc}
> +includes the following fields:
> +\begin{description}
> +\item[\field{format}] specifies an image format. The device MUST set one
> +  of \field{enum virtio_video_format}.
> +\item[\field{profile}] specifies a profile of the compressed image format
> +  specified in \field{format}. The driver SHOULD ignore this value if
> +  \field{format} is a raw format.

Why not documenting that MUST be VIRTIO_VIDEO_PROFILE_UNDEFINED for
raw formats so the field could be extended in the future?

> +\item[\field{mask}] is a bitset that represents the supported
> +  combination of input and output format. If \textit{i}-th bit is set
> +  in \field{mask} of \textit{j}-th \field{struct
> +  virtio_video_format_desc} for input, the device supports encoding or
> +  decoding from the \textit{j}-th input format to \textit{i}-th output
> +  format.
> +\item[\field{width, height}] represents a range of resolutions
> +  supported by the device. If its \field{step} is not applicable, its
> +  \field{min} is equal to its \field{max}.
> +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> case of decoder, the driver SHOULD ignore this value.
> +\item[\field{frame_rates}] is an array of supported frame rates.

I suppose here we are talking about bitrates, right? Could be confused by
FPS.

> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +enum virtio_video_mem_type {
> +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> +};
> +
> +struct virtio_video_stream_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        char debug_name[64];
> +};

This structure has a size not multiple of 8-bytes. Not an issue, but
the same apply to other structures that instead are padded to 8-byte size.

> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{in_mem_type}] is a type of buffer management for input
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{out_mem_type}] is a type of buffer management for output
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{debug_name}] is a text string for a debug purpose.

Must be NUL-terminated ? UTF-8 ?

> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> +  queued buffers through the pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_drain {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_mem_entry {
> +        le64 addr;
> +        le32 length;
> +        u8 padding[4];
> +};
> +
> +struct virtio_video_resource_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        le32 nr_entries;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.

Is it input or output?

> +\item[\field{nr_entries}] number of \field{struct
> +  virtio_video_mem_entry} memory entries.

I suppose that the structure is followed by an array of
virtio_video_mem_entry after a padding of 4-bytes for
alignment.

> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_resource_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        u8 padding[4];

Is this padding for future extensions?

> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> +queue.
> +
> +\begin{lstlisting}
> +#define VIRTIO_VIDEO_MAX_PLANES 8
> +
> +struct virtio_video_resource_queue {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        le32 resource_id;
> +        le64 timestamp;

I suppose you would like this field aligned to avoid
hidden paddings.

> +        le32 nr_data_size;
> +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buf_type of the .
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{nr_data_size}] number of \field{data_size} entries.
> +\item[\field{data_size}] number of data bytes within a plane.
> +\end{description}
> +
... omissis ...
> --
> 2.24.1.735.g03f4e72817-goog
> 
> 

Frediano
Gerd Hoffmann Dec. 19, 2019, 7:46 a.m. UTC | #4
On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote:
> On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> >
> >   Hi,
> >
> > > +The device MUST mark the last buffer with the
> > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > +sequence.
> >
> > No, that would build a race condition into the protocol.  The device
> > could complete the last buffer after the driver has sent the drain
> > command but before the device saw it.  So the flag would not be
> > reliable.
> >
> > I also can't see why the flag is needed in the first place.  The driver
> > should know which buffers are queued still and be able to figure
> > whenever the drain is complete or not without depending on that flag.
> > So I'd suggest to simply drop it.
> 
> Unfortunately video decoders are not that simple. There are always
> going to be some buffers on the decoder side used as reference frames.
> Only the decoder knows when to release them, as it continues decoding
> the stream.

Not clearly defined in the spec:  When is the decoder supposed to send
the response for a queue request?  When it finished decoding (i.e. frame
is ready for playback), or when it doesn't need the buffer any more for
decoding (i.e. buffer can be re-queued or pages can be released)?

> How we defined this in the V4L2 stateful decoder interface is that if
> the decoder happened to return the last framebuffer before the drain
> request arrived, it would return one more, empty.

Ok.  That is not clear from the spec.  I've read the drain request as as
"please stop decoding and complete all queue requests which are in the
virtqueue, even when you didn't process them yet".  In which case
completing the last pending queue request would clearly indicate the
draining request is complete.  Also completing the drain request should
only happen when the operation is finished.

I think the various states a buffer can have and how queuing & deciding
& draining changes these states should be clarified in the
specification.

cheers,
  Gerd
Dmitry Sepp Dec. 19, 2019, 9:26 a.m. UTC | #5
Hi Gerd,

On Mittwoch, 18. Dezember 2019 14:40:37 CET Gerd Hoffmann wrote:
>   Hi,
> 
> > +The device MUST mark the last buffer with the
> > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > +sequence.
> 
> No, that would build a race condition into the protocol.  The device
> could complete the last buffer after the driver has sent the drain
> command but before the device saw it.  So the flag would not be
> reliable.
No, then it means the device was not in drain, but, for example, hit a 
resolution change in the stream and tells us that this is the last buffer with 
the old resolution.

> 
> I also can't see why the flag is needed in the first place.  The driver
> should know which buffers are queued still and be able to figure
> whenever the drain is complete or not without depending on that flag.
> So I'd suggest to simply drop it.
This flag is used not for drain only. In marks the completion of whatever 
specific buffer sequence, like a full end-of-stream, resolution change, drain 
etc. We also need this to handle nested sequences. For instance, a resolution 
change event might happen while in drain.

Regards,
Dmitry.

> 
> That is the only issue I've spotted in the protocol on a first quick
> look.  There are a few places where the spec text could be improved.
> I'll try to set aside some time to go over this in detail, but I can't
> promise I'll find the time to do that before xmas and new year.
> 
> cheers,
>   Gerd
Dmitry Sepp Dec. 19, 2019, 9:48 a.m. UTC | #6
Hi,

On Donnerstag, 19. Dezember 2019 08:46:39 CET Gerd Hoffmann wrote:
> On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote:
> > On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> > >   Hi,
> > >   
> > > > +The device MUST mark the last buffer with the
> > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > +sequence.
> > > 
> > > No, that would build a race condition into the protocol.  The device
> > > could complete the last buffer after the driver has sent the drain
> > > command but before the device saw it.  So the flag would not be
> > > reliable.
> > > 
> > > I also can't see why the flag is needed in the first place.  The driver
> > > should know which buffers are queued still and be able to figure
> > > whenever the drain is complete or not without depending on that flag.
> > > So I'd suggest to simply drop it.
> > 
> > Unfortunately video decoders are not that simple. There are always
> > going to be some buffers on the decoder side used as reference frames.
> > Only the decoder knows when to release them, as it continues decoding
> > the stream.
> 
> Not clearly defined in the spec:  When is the decoder supposed to send
> the response for a queue request?  When it finished decoding (i.e. frame
> is ready for playback), or when it doesn't need the buffer any more for
> decoding (i.e. buffer can be re-queued or pages can be released)?
In my eyes the both statements mean almost the same and both are valid. I 
think whatever underlying libraries are used for decoding on the device side, 
they simply won't return us the buffer as long as the HW device needs them to 
continue its normal operation. So your first sentence applies to output buffers, 
the second - to input buffers.

My understanding is as follows: we send the response for a queue request as 
soon as the HW device on the host side passes the buffer ownership back to the 
client (like when VIDIOC_DQBUF has returned a buffer).

Thanks for reviewing!

Regards,
Dmitry.

> 
> > How we defined this in the V4L2 stateful decoder interface is that if
> > the decoder happened to return the last framebuffer before the drain
> > request arrived, it would return one more, empty.
> 
> Ok.  That is not clear from the spec.  I've read the drain request as as
> "please stop decoding and complete all queue requests which are in the
> virtqueue, even when you didn't process them yet".  In which case
> completing the last pending queue request would clearly indicate the
> draining request is complete.  Also completing the drain request should
> only happen when the operation is finished.
> 
> I think the various states a buffer can have and how queuing & deciding
> & draining changes these states should be clarified in the
> specification.
> 
> cheers,
>   Gerd
Tomasz Figa Dec. 19, 2019, 9:59 a.m. UTC | #7
On Thu, Dec 19, 2019 at 6:48 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi,
>
> On Donnerstag, 19. Dezember 2019 08:46:39 CET Gerd Hoffmann wrote:
> > On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote:
> > > On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> > > >   Hi,
> > > >
> > > > > +The device MUST mark the last buffer with the
> > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > +sequence.
> > > >
> > > > No, that would build a race condition into the protocol.  The device
> > > > could complete the last buffer after the driver has sent the drain
> > > > command but before the device saw it.  So the flag would not be
> > > > reliable.
> > > >
> > > > I also can't see why the flag is needed in the first place.  The driver
> > > > should know which buffers are queued still and be able to figure
> > > > whenever the drain is complete or not without depending on that flag.
> > > > So I'd suggest to simply drop it.
> > >
> > > Unfortunately video decoders are not that simple. There are always
> > > going to be some buffers on the decoder side used as reference frames.
> > > Only the decoder knows when to release them, as it continues decoding
> > > the stream.
> >
> > Not clearly defined in the spec:  When is the decoder supposed to send
> > the response for a queue request?  When it finished decoding (i.e. frame
> > is ready for playback), or when it doesn't need the buffer any more for
> > decoding (i.e. buffer can be re-queued or pages can be released)?
> In my eyes the both statements mean almost the same and both are valid. I
> think whatever underlying libraries are used for decoding on the device side,
> they simply won't return us the buffer as long as the HW device needs them to
> continue its normal operation. So your first sentence applies to output buffers,
> the second - to input buffers.
>
> My understanding is as follows: we send the response for a queue request as
> soon as the HW device on the host side passes the buffer ownership back to the
> client (like when VIDIOC_DQBUF has returned a buffer).

That's how it's defined in V4L2 and what makes the most sense from the
video decoding point of view, as one wants to display frames as soon
as they are available.

However that still doesn't let the driver know which buffers will be
dequeued when. A simple example of this scenario is when the guest is
done displaying a frame and requeues the buffer back to the decoder.
Then the decoder will not choose it for decoding next frames into as
long as the frame in that buffer is still used as a reference frame,
even if one sends the drain request.

>
> Thanks for reviewing!
>
> Regards,
> Dmitry.
>
> >
> > > How we defined this in the V4L2 stateful decoder interface is that if
> > > the decoder happened to return the last framebuffer before the drain
> > > request arrived, it would return one more, empty.
> >
> > Ok.  That is not clear from the spec.  I've read the drain request as as
> > "please stop decoding and complete all queue requests which are in the
> > virtqueue, even when you didn't process them yet".  In which case
> > completing the last pending queue request would clearly indicate the
> > draining request is complete.  Also completing the drain request should
> > only happen when the operation is finished.
> >
> > I think the various states a buffer can have and how queuing & deciding
> > & draining changes these states should be clarified in the
> > specification.
> >
> > cheers,
> >   Gerd
>
>
Tomasz Figa Dec. 19, 2019, 9:59 a.m. UTC | #8
On Thu, Dec 19, 2019 at 6:26 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi Gerd,
>
> On Mittwoch, 18. Dezember 2019 14:40:37 CET Gerd Hoffmann wrote:
> >   Hi,
> >
> > > +The device MUST mark the last buffer with the
> > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > +sequence.
> >
> > No, that would build a race condition into the protocol.  The device
> > could complete the last buffer after the driver has sent the drain
> > command but before the device saw it.  So the flag would not be
> > reliable.
> No, then it means the device was not in drain, but, for example, hit a
> resolution change in the stream and tells us that this is the last buffer with
> the old resolution.
>
> >
> > I also can't see why the flag is needed in the first place.  The driver
> > should know which buffers are queued still and be able to figure
> > whenever the drain is complete or not without depending on that flag.
> > So I'd suggest to simply drop it.
> This flag is used not for drain only. In marks the completion of whatever
> specific buffer sequence, like a full end-of-stream, resolution change, drain
> etc. We also need this to handle nested sequences. For instance, a resolution
> change event might happen while in drain.

Good point. The resolution change event is in a different virtqueue,
so it's not serialized with the request completions.
Dmitry Sepp Dec. 19, 2019, 10:54 a.m. UTC | #9
Hi Tomasz,

On Donnerstag, 19. Dezember 2019 10:59:02 CET Tomasz Figa wrote:
> On Thu, Dec 19, 2019 at 6:48 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> 
wrote:
> > Hi,
> > 
> > On Donnerstag, 19. Dezember 2019 08:46:39 CET Gerd Hoffmann wrote:
> > > On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote:
> > > > On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> 
wrote:
> > > > >   Hi,
> > > > >   
> > > > > > +The device MUST mark the last buffer with the
> > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > > +sequence.
> > > > > 
> > > > > No, that would build a race condition into the protocol.  The device
> > > > > could complete the last buffer after the driver has sent the drain
> > > > > command but before the device saw it.  So the flag would not be
> > > > > reliable.
> > > > > 
> > > > > I also can't see why the flag is needed in the first place.  The
> > > > > driver
> > > > > should know which buffers are queued still and be able to figure
> > > > > whenever the drain is complete or not without depending on that
> > > > > flag.
> > > > > So I'd suggest to simply drop it.
> > > > 
> > > > Unfortunately video decoders are not that simple. There are always
> > > > going to be some buffers on the decoder side used as reference frames.
> > > > Only the decoder knows when to release them, as it continues decoding
> > > > the stream.
> > > 
> > > Not clearly defined in the spec:  When is the decoder supposed to send
> > > the response for a queue request?  When it finished decoding (i.e. frame
> > > is ready for playback), or when it doesn't need the buffer any more for
> > > decoding (i.e. buffer can be re-queued or pages can be released)?
> > 
> > In my eyes the both statements mean almost the same and both are valid. I
> > think whatever underlying libraries are used for decoding on the device
> > side, they simply won't return us the buffer as long as the HW device
> > needs them to continue its normal operation. So your first sentence
> > applies to output buffers, the second - to input buffers.
> > 
> > My understanding is as follows: we send the response for a queue request
> > as
> > soon as the HW device on the host side passes the buffer ownership back to
> > the client (like when VIDIOC_DQBUF has returned a buffer).
> 
> That's how it's defined in V4L2 and what makes the most sense from the
> video decoding point of view, as one wants to display frames as soon
> as they are available.
> 
> However that still doesn't let the driver know which buffers will be
> dequeued when. A simple example of this scenario is when the guest is
> done displaying a frame and requeues the buffer back to the decoder.
> Then the decoder will not choose it for decoding next frames into as
> long as the frame in that buffer is still used as a reference frame,
> even if one sends the drain request.
It might be that I'm getting your point wrong, but do you mean some hardware 
can mark a buffer as ready to be displayed yet still using the underlying 
memory to decode other frames? This means, if you occasionally/intentionally 
write to the buffer you mess up the whole decoding pipeline. That would be 
strange at least...

Regds,
Dmitry.

> 
> > Thanks for reviewing!
> > 
> > Regards,
> > Dmitry.
> > 
> > > > How we defined this in the V4L2 stateful decoder interface is that if
> > > > the decoder happened to return the last framebuffer before the drain
> > > > request arrived, it would return one more, empty.
> > > 
> > > Ok.  That is not clear from the spec.  I've read the drain request as as
> > > "please stop decoding and complete all queue requests which are in the
> > > virtqueue, even when you didn't process them yet".  In which case
> > > completing the last pending queue request would clearly indicate the
> > > draining request is complete.  Also completing the drain request should
> > > only happen when the operation is finished.
> > > 
> > > I think the various states a buffer can have and how queuing & deciding
> > > & draining changes these states should be clarified in the
> > > specification.
> > > 
> > > cheers,
> > > 
> > >   Gerd
Tomasz Figa Dec. 19, 2019, 12:05 p.m. UTC | #10
On Thu, Dec 19, 2019 at 7:55 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi Tomasz,
>
> On Donnerstag, 19. Dezember 2019 10:59:02 CET Tomasz Figa wrote:
> > On Thu, Dec 19, 2019 at 6:48 PM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> wrote:
> > > Hi,
> > >
> > > On Donnerstag, 19. Dezember 2019 08:46:39 CET Gerd Hoffmann wrote:
> > > > On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote:
> > > > > On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com>
> wrote:
> > > > > >   Hi,
> > > > > >
> > > > > > > +The device MUST mark the last buffer with the
> > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > > > +sequence.
> > > > > >
> > > > > > No, that would build a race condition into the protocol.  The device
> > > > > > could complete the last buffer after the driver has sent the drain
> > > > > > command but before the device saw it.  So the flag would not be
> > > > > > reliable.
> > > > > >
> > > > > > I also can't see why the flag is needed in the first place.  The
> > > > > > driver
> > > > > > should know which buffers are queued still and be able to figure
> > > > > > whenever the drain is complete or not without depending on that
> > > > > > flag.
> > > > > > So I'd suggest to simply drop it.
> > > > >
> > > > > Unfortunately video decoders are not that simple. There are always
> > > > > going to be some buffers on the decoder side used as reference frames.
> > > > > Only the decoder knows when to release them, as it continues decoding
> > > > > the stream.
> > > >
> > > > Not clearly defined in the spec:  When is the decoder supposed to send
> > > > the response for a queue request?  When it finished decoding (i.e. frame
> > > > is ready for playback), or when it doesn't need the buffer any more for
> > > > decoding (i.e. buffer can be re-queued or pages can be released)?
> > >
> > > In my eyes the both statements mean almost the same and both are valid. I
> > > think whatever underlying libraries are used for decoding on the device
> > > side, they simply won't return us the buffer as long as the HW device
> > > needs them to continue its normal operation. So your first sentence
> > > applies to output buffers, the second - to input buffers.
> > >
> > > My understanding is as follows: we send the response for a queue request
> > > as
> > > soon as the HW device on the host side passes the buffer ownership back to
> > > the client (like when VIDIOC_DQBUF has returned a buffer).
> >
> > That's how it's defined in V4L2 and what makes the most sense from the
> > video decoding point of view, as one wants to display frames as soon
> > as they are available.
> >
> > However that still doesn't let the driver know which buffers will be
> > dequeued when. A simple example of this scenario is when the guest is
> > done displaying a frame and requeues the buffer back to the decoder.
> > Then the decoder will not choose it for decoding next frames into as
> > long as the frame in that buffer is still used as a reference frame,
> > even if one sends the drain request.
> It might be that I'm getting your point wrong, but do you mean some hardware
> can mark a buffer as ready to be displayed yet still using the underlying
> memory to decode other frames? This means, if you occasionally/intentionally
> write to the buffer you mess up the whole decoding pipeline. That would be
> strange at least...

That's correct. It depends on the hardware, but in principle we don't
want to copy the frames decoded to temporary buffers for using them as
reference frames, as that would waste bandwidth and increase latency.
The contract between the kernel and the application is that it must
not write to the frame buffers if it wants to get correct decoding
results. But after all, I don't see a reason why the application would
write to those buffers.

Best regards,
Tomasz
Gerd Hoffmann Dec. 19, 2019, 12:54 p.m. UTC | #11
Hi,

> > I also can't see why the flag is needed in the first place.  The driver
> > should know which buffers are queued still and be able to figure
> > whenever the drain is complete or not without depending on that flag.
> > So I'd suggest to simply drop it.
> This flag is used not for drain only. In marks the completion of whatever 
> specific buffer sequence, like a full end-of-stream, resolution change, drain 
> etc. We also need this to handle nested sequences. For instance, a resolution 
> change event might happen while in drain.

Ah, ok.  That makes sense (please clarify this in the spec).

cheers,
  Gerd
Gerd Hoffmann Dec. 19, 2019, 1:01 p.m. UTC | #12
Hi,

> > Not clearly defined in the spec:  When is the decoder supposed to send
> > the response for a queue request?  When it finished decoding (i.e. frame
> > is ready for playback), or when it doesn't need the buffer any more for
> > decoding (i.e. buffer can be re-queued or pages can be released)?
> In my eyes the both statements mean almost the same and both are valid.

Well, no.  When the device decoded a P-Frame it can notify the device,
saying "here is your decoded frame".  But the device might still need
the buffer with the decoded frame to properly decode the following B/I
Frames which reference the P-Frame.

cheers,
  Gerd
Gerd Hoffmann Dec. 19, 2019, 1:12 p.m. UTC | #13
Hi,

> > However that still doesn't let the driver know which buffers will be
> > dequeued when. A simple example of this scenario is when the guest is
> > done displaying a frame and requeues the buffer back to the decoder.
> > Then the decoder will not choose it for decoding next frames into as
> > long as the frame in that buffer is still used as a reference frame,
> > even if one sends the drain request.
> It might be that I'm getting your point wrong, but do you mean some hardware 
> can mark a buffer as ready to be displayed yet still using the underlying 
> memory to decode other frames?

Yes, this is how I understand Tomasz Figa.

> This means, if you occasionally/intentionally 
> write to the buffer you mess up the whole decoding pipeline.

And to avoid this the buffer handling aspect must be clarified in the
specification.  Is the device allowed to continue using the buffer after
finishing decoding and completing the queue request?  If so, how do we
hand over buffer ownership back to the driver so it can free the pages?
drain request?  How do we handle re-using buffers?  Can the driver
simply re-queue them and expect the device figures by itself whenever it
can use the buffer or whenever it is still needed as reference frame?

cheers,
  Gerd
Dmitry Sepp Dec. 19, 2019, 1:28 p.m. UTC | #14
Hi Keiichi,

Thank you for the update. Please see some comments below.

Also, we need to bring the virtio_video_control back as it is in fact used by 
the driver to enumerate supported encoder controls. But yes, it still needs to 
be documemnted, it's true.

On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> 
> The virtio video encoder device and decoder device provide functionalities
> to encode and decode video stream respectively.
> Though video encoder and decoder are provided as different devices, they use
> a same protocol.
> 
> Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> ---
>  content.tex      |   1 +
>  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 580 insertions(+)
>  create mode 100644 virtio-video.tex
> 
> diff --git a/content.tex b/content.tex
> index 556b373..9e56839 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> Requirements}\label{sec:Device \input{virtio-vsock.tex}
>  \input{virtio-fs.tex}
>  \input{virtio-rpmb.tex}
> +\input{virtio-video.tex}
> 
>  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> 
> diff --git a/virtio-video.tex b/virtio-video.tex
> new file mode 100644
> index 0000000..30e728d
> --- /dev/null
> +++ b/virtio-video.tex
> @@ -0,0 +1,579 @@
> +\section{Video Device}\label{sec:Device Types / Video Device}
> +
> +The virtio video encoder device and decoder device are virtual devices that
> +supports encoding and decoding respectively. Though the encoder and the
> decoder +are different devices, they use the same protocol.
> +
> +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> +
> +\begin{description}
> +\item[30] encoder device
> +\item[31] decoder device
> +\end{description}
> +
> +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> +
> +\begin{description}
> +\item[0] controlq - queue for sending control commands.
> +\item[1] eventq - queue for sending events happened in the device.
> +\end{description}
> +
> +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> bits} +
> +\begin{description}
> +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> video +  buffers.
> +\end{description}
> +
> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device
> / Feature bits} +
> +The device MUST offer at least one of feature bits.
> +
> +\subsection{Device configuration layout}\label{sec:Device Types / Video
> Device / Device configuration layout} +
> +Video device configuration uses the following layout structure:
> +
> +\begin{lstlisting}
> +struct virtio_video_config {
> +        le32 max_cap_len;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> +  MUST set this value.
> +\end{description}
> +
> +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> Device Initialization} +
> +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> Video Device / Device Initialization} +
> +The driver SHOULD query device capability by using the
> +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> +setup.
> +
> +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> Device Operation} +
> +The driver allocates input and output buffers and queues the buffers
> +to the device. The device performs operations on the buffers according
> +to the function in question.
> +
> +\subsubsection{Device Operation: Create stream}
> +
> +To process buffers, the device needs to associate them with a certain
> +video stream (essentially, a context). Streams are created by
> +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> +determined by the device.
> +
> +\subsubsection{Device Operation: Create buffers}
> +
> +Buffers are used to store the actual data as well as the relevant
> +metadata. Scatter lists are supported, so the buffer doesn't need to
> +be contiguous in guest physical memory.
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> +  resource that is backed by a buffer allocated from the driver's
> +  memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> +  is no longer needed.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Stream parameter control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> for +  input and output streams from the device.
> +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> +  device.
> +\item After setting stream parameters, the driver may issue
> +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> be +  changed implicitly by the device during the set operation.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Process buffers}
> +
> +\begin{itemize*}
> +\item If the function and the buffer type require so, write data to
> +the buffer memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> +processing in the device.
> +\item The request completes asynchronously when the device has
> +finished with the buffer.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Buffer processing control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> +  return all of the already queued buffers.
> +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> +  already queued buffers from the input or the output queue. This also
> +  includes input or output buffers that can be currently owned by the
> +  device's processing pipeline.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Asynchronous events}
> +
> +While processing buffers, the device can send asynchronous event
> +notifications to the driver. The behaviour depends on the exact
> +stream. For example, the decoder device sends a resolution change
> +event when it encounters new resolution metadata in the stream.
> +
> +\subsubsection{Device Operation: Request header}
> +
> +All requests and responses on the control virt queue have a fixed
> +header using the following layout structure and definitions:
> +
> +\begin{lstlisting}
> +enum virtio_video_ctrl_type {
> +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> +
> +        /* request */
> +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> +        VIRTIO_VIDEO_T_STREAM_CREATE,
> +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> +        VIRTIO_VIDEO_T_SET_PARAMS,
> +        VIRTIO_VIDEO_T_GET_PARAMS,
> +
> +        /* response */
> +        VIRTIO_VIDEO_S_OK = 0x0200,
> +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> +
> +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> +};
> +
> +struct virtio_video_ctrl_hdr {
> +        le32 type;
> +        le32 stream_id;
> +        le32 len; /* Length of the structure in bytes. */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{type}] is the type of the driver request or the device
> +response.
> +\item[\field{stream_id}] specifies a target stream.
> +\item[\field{len}] is the length of data in bytes, which includes
> +length of the header.
> +\end{description}
> +
> +\subsubsection{Device Operation: controlq}
> +
> +\begin{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> +supported formats.
> +
> +The driver uses \field{struct virtio_video_get_capability} to send a
> +query request.
> +
> +\begin{lstlisting}
> +enum virtio_video_buf_type {
> +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> +};
I personally didn't like the previous term: pin_type. But, to be honest, I 
don't like the buf_type neither. Consider the GET/SET_PARAMS request: buf_type 
there looks a bit unnatural. We are trying to get stream parameters there, not 
some parameters of whatever buffer. Also I don't see any strict reason to mimic 
v4l2 naming scheme.

I'd better rename it to PORT_TYPE or QUEUE_TYPE.

> +
> +struct virtio_video_get_capability {
> +        struct virtio_video_ctrl_hdr hdr;
> +        enum virtio_video_buf_type buf_type;
> +};
> +\end{lstlisting}
> +\begin{description}
> +\item[\field{buf_type}] is the buffer type that the driver asks
> +information about. The driver MUST set either
> +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> +
> +The device responds a capability by using \field{struct
> +virtio_video_get_capability_resp}.
> +\begin{lstlisting}
> +enum virtio_video_format {
> +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> +        /* Raw formats */
> +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> +        VIRTIO_VIDEO_FORMAT_YUV420,
> +        VIRTIO_VIDEO_FORMAT_YVU420,
Let's add some variants of RGB, like RGBA, ARGB. We need it for the encoder in 
particular .

> +
> +        /* Compressed formats */
> +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
Let's add H265, MPEG4, MPEG2. We already support and use them.

Regards,
Dmitry.

> +};
> +
> +enum virtio_video_profile {
> +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> +
> +        /* H.264 */
> +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> VIRTIO_VIDEO_PROFILE_H264_BASELINE, +       
> VIRTIO_VIDEO_PROFILE_H264_MAIN,
> +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> +        /* VP8 */
> +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> +
> +        /* VP9 */
> +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +};
> +
> +struct virtio_video_format_range {
> +        le32 min;
> +        le32 max;
> +        le32 step;
> +        u8 paddings[4];
> +};
> +
> +struct virtio_video_format_desc {
> +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> +        le64 mask;
> +        struct virtio_video_format_range width;
> +        struct virtio_video_format_range height;
> +        le32 num_rates;
> +        u8 padding[4];
> +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> +};
> +
> +struct virtio_video_get_capability_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 num_descs;
> +        /* Followed by struct virtio_video_format_desc desc[] */
> +};
> +\end{lstlisting}
> +
> +The format description \field{struct virtio_video_format_desc}
> +includes the following fields:
> +\begin{description}
> +\item[\field{format}] specifies an image format. The device MUST set one
> +  of \field{enum virtio_video_format}.
> +\item[\field{profile}] specifies a profile of the compressed image format
> +  specified in \field{format}. The driver SHOULD ignore this value if
> +  \field{format} is a raw format.
> +\item[\field{mask}] is a bitset that represents the supported
> +  combination of input and output format. If \textit{i}-th bit is set
> +  in \field{mask} of \textit{j}-th \field{struct
> +  virtio_video_format_desc} for input, the device supports encoding or
> +  decoding from the \textit{j}-th input format to \textit{i}-th output
> +  format.
> +\item[\field{width, height}] represents a range of resolutions
> +  supported by the device. If its \field{step} is not applicable, its
> +  \field{min} is equal to its \field{max}.
> +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> case of decoder, the driver SHOULD ignore this value.
> +\item[\field{frame_rates}] is an array of supported frame rates.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +enum virtio_video_mem_type {
> +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> +};
> +
> +struct virtio_video_stream_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        char debug_name[64];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{in_mem_type}] is a type of buffer management for input
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{out_mem_type}] is a type of buffer management for output
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{debug_name}] is a text string for a debug purpose.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> +  queued buffers through the pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_drain {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_mem_entry {
> +        le64 addr;
> +        le32 length;
> +        u8 padding[4];
> +};
> +
> +struct virtio_video_resource_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        le32 nr_entries;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{nr_entries}] number of \field{struct
> +  virtio_video_mem_entry} memory entries.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_resource_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> +queue.
> +
> +\begin{lstlisting}
> +#define VIRTIO_VIDEO_MAX_PLANES 8
> +
> +struct virtio_video_resource_queue {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        le32 resource_id;
> +        le64 timestamp;
> +        le32 nr_data_size;
> +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buf_type of the .
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{nr_data_size}] number of \field{data_size} entries.
> +\item[\field{data_size}] number of data bytes within a plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +enum virtio_video_buffer_flag {
> +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> +        /* Encoder only */
> +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> +};
> +
> +struct virtio_video_resource_queue_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le64 timestamp;
> +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> +        le32 size;  /* Encoded size */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{flags}] mark specific buffers in the sequence.
> +\item[\field{size}] data size in the buffer (encoder only).
> +\end{description}
> +
> +The device sends a response to the queue request asynchronously when
> +it has finished processing the buffer.
> +
> +The device SHOULD mark a buffer that triggered a processing error with
> +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> +
> +The device MUST mark the last buffer with the
> +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> +sequence.
> +
> +In case of encoder, to denote a particular frame type the devie MUST
> +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> +  buffers back from the input or the output queue of the device. The
> +  device SHOULD return all of the buffers from the respective queue as
> +  soon as possible without pushing the buffers through the processing
> +  pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_queue_clear {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> +  output of a stream.
> +
> +\begin{lstlisting}
> +struct virtio_video_plane_format {
> +        le32 plane_size;
> +        le32 stride;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{plane_size}] size of the plane in bytes.
> +\item[\field{stride}] stride used for the plane in bytes.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_params {
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> +        le32 frame_width;
> +        le32 frame_height;
> +        le32 min_buffers;
> +        le32 max_buffers;
> +        le32 frame_rate;
> +        struct virtio_video_crop {
> +                le32 left;
> +                le32 top;
> +                le32 width;
> +                le32 height;
> +        } crop;
> +        le32 num_planes;
> +        struct virtio_video_plane_format
> plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{frame_width}] the value to get/set.
> +\item[\field{frame_height}] the value to get/set.
> +\item[\field{pixel_format}] the value to get/set.
> +\item[\field{min_buffers}] minimum buffers required to handle the
> +  format (r/o).
> +\item[\field{max_buffers}] maximum buffers required to handle the
> +  format (r/o).
> +\item[\field{frame_rate}] the value to get/set.
> +\item[\field{crop}] cropping (composing) rectangle.
> +\item[\field{num_planes}] number of planes used to store pixel data
> +(r/o).
> +\item[\field{plane_formats}] description of each plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_get_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +};
> +
> +struct virtio_video_get_params_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\item[\field{params}] parameter values.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> +
> +
> +\begin{lstlisting}
> +struct virtio_video_set_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{params}] parameters to set.
> +\end{description}
> +
> +Setting stream parameters might have side effects within the device.
> +For example, the device MAY perform alignment of width and height,
> +change the number of planes it uses for the format, or do whatever
> +changes that are required to continue normal operation using the
> +updated parameters. It is up to the driver to check the parameter set
> +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> +
> +\end{description}
> +
> +\subsubsection{Device Operation: eventq}
> +
> +The device can report events on the event queue. The driver initially
> +populates the queue with device-writeable buffers. When the device
> +needs to report an event, it fills a buffer and notifies the driver.
> +The driver consumes the report and adds a new buffer to the virtqueue.
> +
> +\begin{lstlisting}
> +enum virtio_video_event_type {
> +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> +        /* For all functions */
> +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> +        /* For decoder only */
> +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> +};
> +
> +struct virtio_video_event {
> +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> +        le32 stream_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{event_type}] type of the triggered event .
> +\item[\field{stream_id}] id of the source stream.
> +\end{description}
> +
> +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> +whenever it encounters new resolution data in the stream. This
> +includes the case of the initial device configuration after metadata
> +has been parsed and the case of dynamic resolution change.
Keiichi Watanabe Dec. 20, 2019, 2:05 p.m. UTC | #15
Hi Frediano,
Thanks for reviewing!

On Thu, Dec 19, 2019 at 2:29 AM Frediano Ziglio <fziglio@redhat.com> wrote:
>
> >
> > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> >
> > The virtio video encoder device and decoder device provide functionalities to
> > encode and decode video stream respectively.
> > Though video encoder and decoder are provided as different devices, they use
> > a
> > same protocol.
> >
> > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > ---
> >  content.tex      |   1 +
> >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 580 insertions(+)
> >  create mode 100644 virtio-video.tex
> >
> > diff --git a/content.tex b/content.tex
> > index 556b373..9e56839 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > Requirements}\label{sec:Device
> >  \input{virtio-vsock.tex}
> >  \input{virtio-fs.tex}
> >  \input{virtio-rpmb.tex}
> > +\input{virtio-video.tex}
> >
> >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >
> > diff --git a/virtio-video.tex b/virtio-video.tex
> > new file mode 100644
> > index 0000000..30e728d
> > --- /dev/null
> > +++ b/virtio-video.tex
> > @@ -0,0 +1,579 @@
> > +\section{Video Device}\label{sec:Device Types / Video Device}
> > +
> > +The virtio video encoder device and decoder device are virtual devices that
> > +supports encoding and decoding respectively. Though the encoder and the
> > decoder
> > +are different devices, they use the same protocol.
> > +
> > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> > +
> > +\begin{description}
> > +\item[30] encoder device
> > +\item[31] decoder device
> > +\end{description}
> > +
> > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> > +
> > +\begin{description}
> > +\item[0] controlq - queue for sending control commands.
> > +\item[1] eventq - queue for sending events happened in the device.
> > +\end{description}
> > +
> > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> > bits}
> > +
> > +\begin{description}
> > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> > video
> > +  buffers.
> > +\end{description}
> > +
> > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device /
> > Feature bits}
> > +
> > +The device MUST offer at least one of feature bits.
> > +
> > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > Device / Device configuration layout}
> > +
> > +Video device configuration uses the following layout structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_config {
> > +        le32 max_cap_len;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > +  MUST set this value.
> > +\end{description}
> > +
> > +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> > Device Initialization}
> > +
> > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / Video
> > Device / Device Initialization}
> > +
> > +The driver SHOULD query device capability by using the
> > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > +setup.
> > +
> > +\subsection{Device Operation}\label{sec:Device Types / Video Device / Device
> > Operation}
> > +
> > +The driver allocates input and output buffers and queues the buffers
> > +to the device. The device performs operations on the buffers according
> > +to the function in question.
> > +
> > +\subsubsection{Device Operation: Create stream}
> > +
> > +To process buffers, the device needs to associate them with a certain
> > +video stream (essentially, a context). Streams are created by
> > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > +determined by the device.
> > +
> > +\subsubsection{Device Operation: Create buffers}
> > +
> > +Buffers are used to store the actual data as well as the relevant
> > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > +be contiguous in guest physical memory.
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > +  resource that is backed by a buffer allocated from the driver's
> > +  memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > +  is no longer needed.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Stream parameter control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters for
> > +  input and output streams from the device.
> > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> > +  device.
> > +\item After setting stream parameters, the driver may issue
> > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> > be
> > +  changed implicitly by the device during the set operation.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Process buffers}
> > +
> > +\begin{itemize*}
> > +\item If the function and the buffer type require so, write data to
> > +the buffer memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > +processing in the device.
> > +\item The request completes asynchronously when the device has
> > +finished with the buffer.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Buffer processing control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > +  return all of the already queued buffers.
> > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > +  already queued buffers from the input or the output queue. This also
> > +  includes input or output buffers that can be currently owned by the
> > +  device's processing pipeline.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Asynchronous events}
> > +
> > +While processing buffers, the device can send asynchronous event
> > +notifications to the driver. The behaviour depends on the exact
> > +stream. For example, the decoder device sends a resolution change
> > +event when it encounters new resolution metadata in the stream.
> > +
> > +\subsubsection{Device Operation: Request header}
> > +
> > +All requests and responses on the control virt queue have a fixed
> > +header using the following layout structure and definitions:
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_ctrl_type {
> > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > +
> > +        /* request */
> > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > +
> > +        /* response */
> > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > +
> > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > +};
> > +
> > +struct virtio_video_ctrl_hdr {
> > +        le32 type;
> > +        le32 stream_id;
> > +        le32 len; /* Length of the structure in bytes. */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{type}] is the type of the driver request or the device
> > +response.
> > +\item[\field{stream_id}] specifies a target stream.
> > +\item[\field{len}] is the length of data in bytes, which includes
> > +length of the header.
>
> I suppose is implicit that it's the size of the container structure.
>

You're right. All struct except virtio_video_get_capability_resp have
fixed size and virtio_video_get_capability_resp has num_descs. So, no
need to have len.
Let me remove this in the next iteration.


> > +\end{description}
> > +
> > +\subsubsection{Device Operation: controlq}
> > +
> > +\begin{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > +supported formats.
> > +
> > +The driver uses \field{struct virtio_video_get_capability} to send a
> > +query request.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buf_type {
> > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > +};
> > +
> > +struct virtio_video_get_capability {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        enum virtio_video_buf_type buf_type;
>
> Here you used an enumeration but later you are using just le32 type
> with a comment. Why this difference?

No difference. I will change this to use le32 since virtio-gpu's spec does so.

>
> > +};
> > +\end{lstlisting}
> > +\begin{description}
> > +\item[\field{buf_type}] is the buffer type that the driver asks
> > +information about. The driver MUST set either
> > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}.
> > +\end{description}
> > +
> > +The device responds a capability by using \field{struct
> > +virtio_video_get_capability_resp}.
> > +\begin{lstlisting}
> > +enum virtio_video_format {
> > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
>
> Really minor: maybe you want to insert an empty line to be coherent
> with other enumerations?

Yes. Thanks.

>
> > +        /* Raw formats */
> > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > +
> > +        /* Compressed formats */
> > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > +};
> > +
> > +enum virtio_video_profile {
> > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > +
> > +        /* H.264 */
> > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > VIRTIO_VIDEO_PROFILE_H264_BASELINE,
>
> Maybe you want
>
> VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_MIN,
>
> like others?

Thanks for catching this. I think we can find this kind of errors once
we update the driver implementation.

>
> > +        VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > +
> > +        /* VP8 */
> > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > +
> > +        /* VP9 */
> > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +};
> > +
> > +struct virtio_video_format_range {
> > +        le32 min;
> > +        le32 max;
> > +        le32 step;
> > +        u8 paddings[4];
>
> Here you padded at 8-byte like many other structures but not for
> virtio_video_ctrl_hdr. Is this expected?

I padded here for 64-bits alignment. So, I should have padded for
ctrl_hdr as you said. But, no need anymore as we will remove |len|
from the struct as you suggested above:)

>
> > +};
> > +
> > +struct virtio_video_format_desc {
> > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > +        le64 mask;
>
> This will limit the formats to 64 while "num_descs" below is a
> 32 bit.

It's my intention. I will explicitly write that "num_desc" must not exceed 64.
I guess 64 is big enough as a number of supported input/output format
and couldn't find a better idea to represent this bipartite graph. If
you have another idea, please let me hear.

>
> > +        struct virtio_video_format_range width;
> > +        struct virtio_video_format_range height;
> > +        le32 num_rates;
> > +        u8 padding[4];
> > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > +};
> > +
> > +struct virtio_video_get_capability_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 num_descs;
> > +        /* Followed by struct virtio_video_format_desc desc[] */
> > +};
> > +\end{lstlisting}
> > +
> > +The format description \field{struct virtio_video_format_desc}
> > +includes the following fields:
> > +\begin{description}
> > +\item[\field{format}] specifies an image format. The device MUST set one
> > +  of \field{enum virtio_video_format}.
> > +\item[\field{profile}] specifies a profile of the compressed image format
> > +  specified in \field{format}. The driver SHOULD ignore this value if
> > +  \field{format} is a raw format.
>
> Why not documenting that MUST be VIRTIO_VIDEO_PROFILE_UNDEFINED for
> raw formats so the field could be extended in the future?

Sounds good. Will do.

>
> > +\item[\field{mask}] is a bitset that represents the supported
> > +  combination of input and output format. If \textit{i}-th bit is set
> > +  in \field{mask} of \textit{j}-th \field{struct
> > +  virtio_video_format_desc} for input, the device supports encoding or
> > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > +  format.
> > +\item[\field{width, height}] represents a range of resolutions
> > +  supported by the device. If its \field{step} is not applicable, its
> > +  \field{min} is equal to its \field{max}.
> > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> > case of decoder, the driver SHOULD ignore this value.
> > +\item[\field{frame_rates}] is an array of supported frame rates.
>
> I suppose here we are talking about bitrates, right? Could be confused by
> FPS.

Nope. This is for frame rates or frame intervals. In V4L2 driver, the
values will be used for VIDIOC_ENUM_FRAMEINTERVALS ioctl.
(Dmitry, please correct me if I am wrong)

>
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_mem_type {
> > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > +};
> > +
> > +struct virtio_video_stream_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        char debug_name[64];
> > +};
>
> This structure has a size not multiple of 8-bytes. Not an issue, but
> the same apply to other structures that instead are padded to 8-byte size.

It's my fault. I tried to add paddings to make all structs' size
multiple of 8-bytes, I overlooked some.

>
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{in_mem_type}] is a type of buffer management for input
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{out_mem_type}] is a type of buffer management for output
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{debug_name}] is a text string for a debug purpose.
>
> Must be NUL-terminated ? UTF-8 ?

Yeah, I will add a description similar to one for "tag" in virtio_fs_config.

>
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > +  queued buffers through the pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_drain {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_mem_entry {
> > +        le64 addr;
> > +        le32 length;
> > +        u8 padding[4];
> > +};
> > +
> > +struct virtio_video_resource_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        le32 nr_entries;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
>
> Is it input or output?

Will add a field for it.

>
> > +\item[\field{nr_entries}] number of \field{struct
> > +  virtio_video_mem_entry} memory entries.
>
> I suppose that the structure is followed by an array of
> virtio_video_mem_entry after a padding of 4-bytes for
> alignment.
>
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_resource_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        u8 padding[4];
>
> Is this padding for future extensions?

Nope, it's mistake. When I was editing structs, I forgot to add/remove
padding. I hope we will not have such mistakes after we have an
updated driver implementation.

>
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > +queue.
> > +
> > +\begin{lstlisting}
> > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > +
> > +struct virtio_video_resource_queue {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        le32 resource_id;
> > +        le64 timestamp;
>
> I suppose you would like this field aligned to avoid
> hidden paddings.

Thanks.

Best regards,
Keiichi

>
> > +        le32 nr_data_size;
> > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buf_type of the .
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > +\item[\field{data_size}] number of data bytes within a plane.
> > +\end{description}
> > +
> ... omissis ...
> > --
> > 2.24.1.735.g03f4e72817-goog
> >
> >
>
> Frediano
>
Keiichi Watanabe Dec. 20, 2019, 3:26 p.m. UTC | #16
Hi Dmitry,
Thanks for the feedback.

On Thu, Dec 19, 2019 at 10:28 PM Dmitry Sepp
<dmitry.sepp@opensynergy.com> wrote:
>
> Hi Keiichi,
>
> Thank you for the update. Please see some comments below.
>
> Also, we need to bring the virtio_video_control back as it is in fact used by
> the driver to enumerate supported encoder controls. But yes, it still needs to
> be documemnted, it's true.

Yeah, we need it eventually, but I'm wondering what is the difference
between values set by SET_PARAMS and ones by controls.
I saw your driver implementation has BITRATE, PROFILE and LEVEL as
controls. What's the reason why you didn't add them in
virtio_video_params?

>
> On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> >
> > The virtio video encoder device and decoder device provide functionalities
> > to encode and decode video stream respectively.
> > Though video encoder and decoder are provided as different devices, they use
> > a same protocol.
> >
> > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > ---
> >  content.tex      |   1 +
> >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 580 insertions(+)
> >  create mode 100644 virtio-video.tex
> >
> > diff --git a/content.tex b/content.tex
> > index 556b373..9e56839 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> >  \input{virtio-fs.tex}
> >  \input{virtio-rpmb.tex}
> > +\input{virtio-video.tex}
> >
> >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >
> > diff --git a/virtio-video.tex b/virtio-video.tex
> > new file mode 100644
> > index 0000000..30e728d
> > --- /dev/null
> > +++ b/virtio-video.tex
> > @@ -0,0 +1,579 @@
> > +\section{Video Device}\label{sec:Device Types / Video Device}
> > +
> > +The virtio video encoder device and decoder device are virtual devices that
> > +supports encoding and decoding respectively. Though the encoder and the
> > decoder +are different devices, they use the same protocol.
> > +
> > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> > +
> > +\begin{description}
> > +\item[30] encoder device
> > +\item[31] decoder device
> > +\end{description}
> > +
> > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> > +
> > +\begin{description}
> > +\item[0] controlq - queue for sending control commands.
> > +\item[1] eventq - queue for sending events happened in the device.
> > +\end{description}
> > +
> > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> > bits} +
> > +\begin{description}
> > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> > video +  buffers.
> > +\end{description}
> > +
> > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device
> > / Feature bits} +
> > +The device MUST offer at least one of feature bits.
> > +
> > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > Device / Device configuration layout} +
> > +Video device configuration uses the following layout structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_config {
> > +        le32 max_cap_len;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > +  MUST set this value.
> > +\end{description}
> > +
> > +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> > Device Initialization} +
> > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > Video Device / Device Initialization} +
> > +The driver SHOULD query device capability by using the
> > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > +setup.
> > +
> > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > Device Operation} +
> > +The driver allocates input and output buffers and queues the buffers
> > +to the device. The device performs operations on the buffers according
> > +to the function in question.
> > +
> > +\subsubsection{Device Operation: Create stream}
> > +
> > +To process buffers, the device needs to associate them with a certain
> > +video stream (essentially, a context). Streams are created by
> > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > +determined by the device.
> > +
> > +\subsubsection{Device Operation: Create buffers}
> > +
> > +Buffers are used to store the actual data as well as the relevant
> > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > +be contiguous in guest physical memory.
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > +  resource that is backed by a buffer allocated from the driver's
> > +  memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > +  is no longer needed.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Stream parameter control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> > for +  input and output streams from the device.
> > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> > +  device.
> > +\item After setting stream parameters, the driver may issue
> > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> > be +  changed implicitly by the device during the set operation.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Process buffers}
> > +
> > +\begin{itemize*}
> > +\item If the function and the buffer type require so, write data to
> > +the buffer memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > +processing in the device.
> > +\item The request completes asynchronously when the device has
> > +finished with the buffer.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Buffer processing control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > +  return all of the already queued buffers.
> > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > +  already queued buffers from the input or the output queue. This also
> > +  includes input or output buffers that can be currently owned by the
> > +  device's processing pipeline.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Asynchronous events}
> > +
> > +While processing buffers, the device can send asynchronous event
> > +notifications to the driver. The behaviour depends on the exact
> > +stream. For example, the decoder device sends a resolution change
> > +event when it encounters new resolution metadata in the stream.
> > +
> > +\subsubsection{Device Operation: Request header}
> > +
> > +All requests and responses on the control virt queue have a fixed
> > +header using the following layout structure and definitions:
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_ctrl_type {
> > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > +
> > +        /* request */
> > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > +
> > +        /* response */
> > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > +
> > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > +};
> > +
> > +struct virtio_video_ctrl_hdr {
> > +        le32 type;
> > +        le32 stream_id;
> > +        le32 len; /* Length of the structure in bytes. */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{type}] is the type of the driver request or the device
> > +response.
> > +\item[\field{stream_id}] specifies a target stream.
> > +\item[\field{len}] is the length of data in bytes, which includes
> > +length of the header.
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: controlq}
> > +
> > +\begin{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > +supported formats.
> > +
> > +The driver uses \field{struct virtio_video_get_capability} to send a
> > +query request.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buf_type {
> > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > +};
> I personally didn't like the previous term: pin_type. But, to be honest, I
> don't like the buf_type neither. Consider the GET/SET_PARAMS request: buf_type
> there looks a bit unnatural. We are trying to get stream parameters there, not
> some parameters of whatever buffer. Also I don't see any strict reason to mimic
> v4l2 naming scheme.
>
> I'd better rename it to PORT_TYPE or QUEUE_TYPE.

To be honest, I also wasn't satisfied with the name "buf_type".
QUEUE_TYPE sounds better.
Will rename it.

>
> > +
> > +struct virtio_video_get_capability {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        enum virtio_video_buf_type buf_type;
> > +};
> > +\end{lstlisting}
> > +\begin{description}
> > +\item[\field{buf_type}] is the buffer type that the driver asks
> > +information about. The driver MUST set either
> > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > +
> > +The device responds a capability by using \field{struct
> > +virtio_video_get_capability_resp}.
> > +\begin{lstlisting}
> > +enum virtio_video_format {
> > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > +        /* Raw formats */
> > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > +        VIRTIO_VIDEO_FORMAT_YVU420,
> Let's add some variants of RGB, like RGBA, ARGB. We need it for the encoder in
> particular .
>

Sounds good.
BTW, which "ARGB8888" or "A8R8G8B8" is preferred? While the first one
comes from DRM's FourCC, the second one comes from virtio_gpu_formats.
I personally prefer the first one, then we can have a naming convention like:
VIRTIO_VIDEO_FORMAT_<name from drm_fourcc.h>

> > +
> > +        /* Compressed formats */
> > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> Let's add H265, MPEG4, MPEG2. We already support and use them.

Sure.

Best regards,
Keiichi

>
> Regards,
> Dmitry.
>
> > +};
> > +
> > +enum virtio_video_profile {
> > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > +
> > +        /* H.264 */
> > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > +        /* VP8 */
> > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > +
> > +        /* VP9 */
> > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +};
> > +
> > +struct virtio_video_format_range {
> > +        le32 min;
> > +        le32 max;
> > +        le32 step;
> > +        u8 paddings[4];
> > +};
> > +
> > +struct virtio_video_format_desc {
> > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > +        le64 mask;
> > +        struct virtio_video_format_range width;
> > +        struct virtio_video_format_range height;
> > +        le32 num_rates;
> > +        u8 padding[4];
> > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > +};
> > +
> > +struct virtio_video_get_capability_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 num_descs;
> > +        /* Followed by struct virtio_video_format_desc desc[] */
> > +};
> > +\end{lstlisting}
> > +
> > +The format description \field{struct virtio_video_format_desc}
> > +includes the following fields:
> > +\begin{description}
> > +\item[\field{format}] specifies an image format. The device MUST set one
> > +  of \field{enum virtio_video_format}.
> > +\item[\field{profile}] specifies a profile of the compressed image format
> > +  specified in \field{format}. The driver SHOULD ignore this value if
> > +  \field{format} is a raw format.
> > +\item[\field{mask}] is a bitset that represents the supported
> > +  combination of input and output format. If \textit{i}-th bit is set
> > +  in \field{mask} of \textit{j}-th \field{struct
> > +  virtio_video_format_desc} for input, the device supports encoding or
> > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > +  format.
> > +\item[\field{width, height}] represents a range of resolutions
> > +  supported by the device. If its \field{step} is not applicable, its
> > +  \field{min} is equal to its \field{max}.
> > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> > case of decoder, the driver SHOULD ignore this value.
> > +\item[\field{frame_rates}] is an array of supported frame rates.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_mem_type {
> > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > +};
> > +
> > +struct virtio_video_stream_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        char debug_name[64];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{in_mem_type}] is a type of buffer management for input
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{out_mem_type}] is a type of buffer management for output
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{debug_name}] is a text string for a debug purpose.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > +  queued buffers through the pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_drain {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_mem_entry {
> > +        le64 addr;
> > +        le32 length;
> > +        u8 padding[4];
> > +};
> > +
> > +struct virtio_video_resource_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        le32 nr_entries;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{nr_entries}] number of \field{struct
> > +  virtio_video_mem_entry} memory entries.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_resource_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > +queue.
> > +
> > +\begin{lstlisting}
> > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > +
> > +struct virtio_video_resource_queue {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        le32 resource_id;
> > +        le64 timestamp;
> > +        le32 nr_data_size;
> > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buf_type of the .
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > +\item[\field{data_size}] number of data bytes within a plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buffer_flag {
> > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > +        /* Encoder only */
> > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > +};
> > +
> > +struct virtio_video_resource_queue_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le64 timestamp;
> > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > +        le32 size;  /* Encoded size */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{flags}] mark specific buffers in the sequence.
> > +\item[\field{size}] data size in the buffer (encoder only).
> > +\end{description}
> > +
> > +The device sends a response to the queue request asynchronously when
> > +it has finished processing the buffer.
> > +
> > +The device SHOULD mark a buffer that triggered a processing error with
> > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > +
> > +The device MUST mark the last buffer with the
> > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > +sequence.
> > +
> > +In case of encoder, to denote a particular frame type the devie MUST
> > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > +  buffers back from the input or the output queue of the device. The
> > +  device SHOULD return all of the buffers from the respective queue as
> > +  soon as possible without pushing the buffers through the processing
> > +  pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_queue_clear {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > +  output of a stream.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_plane_format {
> > +        le32 plane_size;
> > +        le32 stride;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{plane_size}] size of the plane in bytes.
> > +\item[\field{stride}] stride used for the plane in bytes.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_params {
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > +        le32 frame_width;
> > +        le32 frame_height;
> > +        le32 min_buffers;
> > +        le32 max_buffers;
> > +        le32 frame_rate;
> > +        struct virtio_video_crop {
> > +                le32 left;
> > +                le32 top;
> > +                le32 width;
> > +                le32 height;
> > +        } crop;
> > +        le32 num_planes;
> > +        struct virtio_video_plane_format
> > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{frame_width}] the value to get/set.
> > +\item[\field{frame_height}] the value to get/set.
> > +\item[\field{pixel_format}] the value to get/set.
> > +\item[\field{min_buffers}] minimum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{max_buffers}] maximum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{frame_rate}] the value to get/set.
> > +\item[\field{crop}] cropping (composing) rectangle.
> > +\item[\field{num_planes}] number of planes used to store pixel data
> > +(r/o).
> > +\item[\field{plane_formats}] description of each plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_get_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +};
> > +
> > +struct virtio_video_get_params_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\item[\field{params}] parameter values.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > +
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_set_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{params}] parameters to set.
> > +\end{description}
> > +
> > +Setting stream parameters might have side effects within the device.
> > +For example, the device MAY perform alignment of width and height,
> > +change the number of planes it uses for the format, or do whatever
> > +changes that are required to continue normal operation using the
> > +updated parameters. It is up to the driver to check the parameter set
> > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > +
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: eventq}
> > +
> > +The device can report events on the event queue. The driver initially
> > +populates the queue with device-writeable buffers. When the device
> > +needs to report an event, it fills a buffer and notifies the driver.
> > +The driver consumes the report and adds a new buffer to the virtqueue.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_event_type {
> > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > +        /* For all functions */
> > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > +        /* For decoder only */
> > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > +};
> > +
> > +struct virtio_video_event {
> > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > +        le32 stream_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{event_type}] type of the triggered event .
> > +\item[\field{stream_id}] id of the source stream.
> > +\end{description}
> > +
> > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > +whenever it encounters new resolution data in the stream. This
> > +includes the case of the initial device configuration after metadata
> > +has been parsed and the case of dynamic resolution change.
>
>
Dmitry Sepp Dec. 20, 2019, 3:33 p.m. UTC | #17
Hi Keiichi,

On Freitag, 20. Dezember 2019 15:05:02 CET Keiichi Watanabe wrote:
> Hi Frediano,
> Thanks for reviewing!
> 
> On Thu, Dec 19, 2019 at 2:29 AM Frediano Ziglio <fziglio@redhat.com> wrote:
> > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > 
> > > The virtio video encoder device and decoder device provide
> > > functionalities to encode and decode video stream respectively.
> > > Though video encoder and decoder are provided as different devices, they
> > > use a
> > > same protocol.
> > > 
> > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > ---
> > > 
> > >  content.tex      |   1 +
> > >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 580 insertions(+)
> > >  create mode 100644 virtio-video.tex
> > > 
> > > diff --git a/content.tex b/content.tex
> > > index 556b373..9e56839 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > Requirements}\label{sec:Device
> > > 
> > >  \input{virtio-vsock.tex}
> > >  \input{virtio-fs.tex}
> > >  \input{virtio-rpmb.tex}
> > > 
> > > +\input{virtio-video.tex}
> > > 
> > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > 
> > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > new file mode 100644
> > > index 0000000..30e728d
> > > --- /dev/null
> > > +++ b/virtio-video.tex
> > > @@ -0,0 +1,579 @@
> > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > +
> > > +The virtio video encoder device and decoder device are virtual devices
> > > that +supports encoding and decoding respectively. Though the encoder
> > > and the decoder
> > > +are different devices, they use the same protocol.
> > > +
> > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device
> > > ID}
> > > +
> > > +\begin{description}
> > > +\item[30] encoder device
> > > +\item[31] decoder device
> > > +\end{description}
> > > +
> > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > Virtqueues} +
> > > +\begin{description}
> > > +\item[0] controlq - queue for sending control commands.
> > > +\item[1] eventq - queue for sending events happened in the device.
> > > +\end{description}
> > > +
> > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > Feature
> > > bits}
> > > +
> > > +\begin{description}
> > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used
> > > for
> > > video
> > > +  buffers.
> > > +\end{description}
> > > +
> > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > Device / Feature bits}
> > > +
> > > +The device MUST offer at least one of feature bits.
> > > +
> > > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > > Device / Device configuration layout}
> > > +
> > > +Video device configuration uses the following layout structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_config {
> > > +        le32 max_cap_len;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > +  MUST set this value.
> > > +\end{description}
> > > +
> > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > Device /
> > > Device Initialization}
> > > +
> > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > > Video Device / Device Initialization}
> > > +
> > > +The driver SHOULD query device capability by using the
> > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > > +setup.
> > > +
> > > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > > Device Operation}
> > > +
> > > +The driver allocates input and output buffers and queues the buffers
> > > +to the device. The device performs operations on the buffers according
> > > +to the function in question.
> > > +
> > > +\subsubsection{Device Operation: Create stream}
> > > +
> > > +To process buffers, the device needs to associate them with a certain
> > > +video stream (essentially, a context). Streams are created by
> > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > +determined by the device.
> > > +
> > > +\subsubsection{Device Operation: Create buffers}
> > > +
> > > +Buffers are used to store the actual data as well as the relevant
> > > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > > +be contiguous in guest physical memory.
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > +  resource that is backed by a buffer allocated from the driver's
> > > +  memory.
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > > +  is no longer needed.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Stream parameter control}
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > parameters for +  input and output streams from the device.
> > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to
> > > the +  device.
> > > +\item After setting stream parameters, the driver may issue
> > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output
> > > can be
> > > +  changed implicitly by the device during the set operation.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Process buffers}
> > > +
> > > +\begin{itemize*}
> > > +\item If the function and the buffer type require so, write data to
> > > +the buffer memory.
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > +processing in the device.
> > > +\item The request completes asynchronously when the device has
> > > +finished with the buffer.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Buffer processing control}
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > > +  return all of the already queued buffers.
> > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > > +  already queued buffers from the input or the output queue. This also
> > > +  includes input or output buffers that can be currently owned by the
> > > +  device's processing pipeline.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Asynchronous events}
> > > +
> > > +While processing buffers, the device can send asynchronous event
> > > +notifications to the driver. The behaviour depends on the exact
> > > +stream. For example, the decoder device sends a resolution change
> > > +event when it encounters new resolution metadata in the stream.
> > > +
> > > +\subsubsection{Device Operation: Request header}
> > > +
> > > +All requests and responses on the control virt queue have a fixed
> > > +header using the following layout structure and definitions:
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_ctrl_type {
> > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > +
> > > +        /* request */
> > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > +
> > > +        /* response */
> > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > +
> > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > +};
> > > +
> > > +struct virtio_video_ctrl_hdr {
> > > +        le32 type;
> > > +        le32 stream_id;
> > > +        le32 len; /* Length of the structure in bytes. */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{type}] is the type of the driver request or the device
> > > +response.
> > > +\item[\field{stream_id}] specifies a target stream.
> > > +\item[\field{len}] is the length of data in bytes, which includes
> > > +length of the header.
> > 
> > I suppose is implicit that it's the size of the container structure.
> 
> You're right. All struct except virtio_video_get_capability_resp have
> fixed size and virtio_video_get_capability_resp has num_descs. So, no
> need to have len.
> Let me remove this in the next iteration.
> 
> > > +\end{description}
> > > +
> > > +\subsubsection{Device Operation: controlq}
> > > +
> > > +\begin{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > +supported formats.
> > > +
> > > +The driver uses \field{struct virtio_video_get_capability} to send a
> > > +query request.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_buf_type {
> > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > +};
> > > +
> > > +struct virtio_video_get_capability {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        enum virtio_video_buf_type buf_type;
> > 
> > Here you used an enumeration but later you are using just le32 type
> > with a comment. Why this difference?
> 
> No difference. I will change this to use le32 since virtio-gpu's spec does
> so.
> > > +};
> > > +\end{lstlisting}
> > > +\begin{description}
> > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > +information about. The driver MUST set either
> > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > +
> > > +The device responds a capability by using \field{struct
> > > +virtio_video_get_capability_resp}.
> > > +\begin{lstlisting}
> > > +enum virtio_video_format {
> > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > 
> > Really minor: maybe you want to insert an empty line to be coherent
> > with other enumerations?
> 
> Yes. Thanks.
> 
> > > +        /* Raw formats */
> > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > +
> > > +        /* Compressed formats */
> > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > +};
> > > +
> > > +enum virtio_video_profile {
> > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > +
> > > +        /* H.264 */
> > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > VIRTIO_VIDEO_PROFILE_H264_BASELINE,
> > 
> > Maybe you want
> > 
> > VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_MIN,
> > 
> > like others?
> 
> Thanks for catching this. I think we can find this kind of errors once
> we update the driver implementation.
> 
> > > +        VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > +
> > > +        /* VP8 */
> > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > +
> > > +        /* VP9 */
> > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > +};
> > > +
> > > +struct virtio_video_format_range {
> > > +        le32 min;
> > > +        le32 max;
> > > +        le32 step;
> > > +        u8 paddings[4];
> > 
> > Here you padded at 8-byte like many other structures but not for
> > virtio_video_ctrl_hdr. Is this expected?
> 
> I padded here for 64-bits alignment. So, I should have padded for
> ctrl_hdr as you said. But, no need anymore as we will remove |len|
> from the struct as you suggested above:)
> 
> > > +};
> > > +
> > > +struct virtio_video_format_desc {
> > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > +        le64 mask;
> > 
> > This will limit the formats to 64 while "num_descs" below is a
> > 32 bit.
> 
> It's my intention. I will explicitly write that "num_desc" must not exceed
> 64. I guess 64 is big enough as a number of supported input/output format
> and couldn't find a better idea to represent this bipartite graph. If you
> have another idea, please let me hear.
> 
> > > +        struct virtio_video_format_range width;
> > > +        struct virtio_video_format_range height;
> > > +        le32 num_rates;
> > > +        u8 padding[4];
> > > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > > +};
> > > +
> > > +struct virtio_video_get_capability_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 num_descs;
> > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +The format description \field{struct virtio_video_format_desc}
> > > +includes the following fields:
> > > +\begin{description}
> > > +\item[\field{format}] specifies an image format. The device MUST set
> > > one
> > > +  of \field{enum virtio_video_format}.
> > > +\item[\field{profile}] specifies a profile of the compressed image
> > > format
> > > +  specified in \field{format}. The driver SHOULD ignore this value if
> > > +  \field{format} is a raw format.
> > 
> > Why not documenting that MUST be VIRTIO_VIDEO_PROFILE_UNDEFINED for
> > raw formats so the field could be extended in the future?
> 
> Sounds good. Will do.
> 
> > > +\item[\field{mask}] is a bitset that represents the supported
> > > +  combination of input and output format. If \textit{i}-th bit is set
> > > +  in \field{mask} of \textit{j}-th \field{struct
> > > +  virtio_video_format_desc} for input, the device supports encoding or
> > > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > > +  format.
> > > +\item[\field{width, height}] represents a range of resolutions
> > > +  supported by the device. If its \field{step} is not applicable, its
> > > +  \field{min} is equal to its \field{max}.
> > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}.
> > > In
> > > case of decoder, the driver SHOULD ignore this value.
> > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > 
> > I suppose here we are talking about bitrates, right? Could be confused by
> > FPS.
> 
> Nope. This is for frame rates or frame intervals. In V4L2 driver, the
> values will be used for VIDIOC_ENUM_FRAMEINTERVALS ioctl.
> (Dmitry, please correct me if I am wrong)
Right, this is correct.

Regards,
Dmitry.

> 
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_mem_type {
> > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > +};
> > > +
> > > +struct virtio_video_stream_create {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > +        char debug_name[64];
> > > +};
> > 
> > This structure has a size not multiple of 8-bytes. Not an issue, but
> > the same apply to other structures that instead are padded to 8-byte size.
> 
> It's my fault. I tried to add paddings to make all structs' size
> multiple of 8-bytes, I overlooked some.
> 
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > +buffers. The driver MUST set a value in \field{enum
> > > +virtio_video_mem_type}.
> > > +\item[\field{out_mem_type}] is a type of buffer management for output
> > > +buffers. The driver MUST set a value in \field{enum
> > > +virtio_video_mem_type}.
> > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > 
> > Must be NUL-terminated ? UTF-8 ?
> 
> Yeah, I will add a description similar to one for "tag" in virtio_fs_config.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_stream_destroy {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > +  queued buffers through the pipeline.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_stream_drain {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_mem_entry {
> > > +        le64 addr;
> > > +        le32 length;
> > > +        u8 padding[4];
> > > +};
> > > +
> > > +struct virtio_video_resource_create {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 resource_id;
> > > +        le32 nr_entries;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{resource_id}] internal id of the resource.
> > 
> > Is it input or output?
> 
> Will add a field for it.
> 
> > > +\item[\field{nr_entries}] number of \field{struct
> > > +  virtio_video_mem_entry} memory entries.
> > 
> > I suppose that the structure is followed by an array of
> > virtio_video_mem_entry after a padding of 4-bytes for
> > alignment.
> > 
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_resource_destroy {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 resource_id;
> > > +        u8 padding[4];
> > 
> > Is this padding for future extensions?
> 
> Nope, it's mistake. When I was editing structs, I forgot to add/remove
> padding. I hope we will not have such mistakes after we have an
> updated driver implementation.
> 
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > +queue.
> > > +
> > > +\begin{lstlisting}
> > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > +
> > > +struct virtio_video_resource_queue {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type;
> > > +        le32 resource_id;
> > > +        le64 timestamp;
> > 
> > I suppose you would like this field aligned to avoid
> > hidden paddings.
> 
> Thanks.
> 
> Best regards,
> Keiichi
> 
> > > +        le32 nr_data_size;
> > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buf_type of the .
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > +  for synchronisation.
> > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > +\item[\field{data_size}] number of data bytes within a plane.
> > > +\end{description}
> > > +
> > 
> > ... omissis ...
> > 
> > > --
> > > 2.24.1.735.g03f4e72817-goog
> > 
> > Frediano
Tomasz Figa Dec. 21, 2019, 6:46 a.m. UTC | #18
On Sat, Dec 21, 2019 at 12:46 AM Dmitry Sepp
<dmitry.sepp@opensynergy.com> wrote:
> On Freitag, 20. Dezember 2019 16:26:50 CET Keiichi Watanabe wrote:
> > On Thu, Dec 19, 2019 at 10:28 PM Dmitry Sepp
> > <dmitry.sepp@opensynergy.com> wrote:
> > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
[snip]
> > > > +enum virtio_video_format {
> > > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > + /* Raw formats */
> > > > + VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > + VIRTIO_VIDEO_FORMAT_YUV420,
> > > > + VIRTIO_VIDEO_FORMAT_YVU420,
> > >
> > > Let's add some variants of RGB, like RGBA, ARGB. We need it for the
> > > encoder in particular .
> >
> > Sounds good.
> > BTW, which "ARGB8888" or "A8R8G8B8" is preferred? While the first one
> > comes from DRM's FourCC, the second one comes from virtio_gpu_formats.
> > I personally prefer the first one, then we can have a naming convention
> > like: VIRTIO_VIDEO_FORMAT_<name from drm_fourcc.h>
> >
>
> I'd go with ARGB8888 and BGRA8888 (might be with X variants).

Just to make sure we're talking about the same formats. DRM naming
convention is based on the little endian convention, which for 32-bit
formats means that you interpret the whole pixel as a packed 32-bit
word on a little endian system. For ARGB8888 that would mean (bit 31)
ARGB (bit 0) in the 32-bit word and (byte 0) B, G, R, A (byte 3) when
looking at separate bytes in memory. Does that correspond to your
expected format?

We also have to be specific about the A and X formats, as the A format
should be supported only if the hardware (host) doesn't ignore the
alpha channel. I haven't seen any hardware capable of encoding alpha
channel yet, but apparently for WebM the standard is to just encode
the alpha channel into another stream as Y, together with dummy U and
V values. [1] That sounds like something that would be handled by two
separate encoding streams and not just one that accepts RGBA on the
input.

[1] http://wiki.webmproject.org/alpha-channel

Best regards,
Tomasz
Dmitry Sepp Dec. 30, 2019, 12:16 p.m. UTC | #19
On Donnerstag, 19. Dezember 2019 14:28:23 CET Dmitry Sepp wrote:
> Hi Keiichi,
> 
> Thank you for the update. Please see some comments below.
> 
> Also, we need to bring the virtio_video_control back as it is in fact used
> by the driver to enumerate supported encoder controls. But yes, it still
> needs to be documemnted, it's true.
> 
> On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > 
> > The virtio video encoder device and decoder device provide functionalities
> > to encode and decode video stream respectively.
> > Though video encoder and decoder are provided as different devices, they
> > use a same protocol.
> > 
> > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > ---
> > 
> >  content.tex      |   1 +
> >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 580 insertions(+)
> >  create mode 100644 virtio-video.tex
> > 
> > diff --git a/content.tex b/content.tex
> > index 556b373..9e56839 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > 
> >  \input{virtio-fs.tex}
> >  \input{virtio-rpmb.tex}
> > 
> > +\input{virtio-video.tex}
> > 
> >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > 
> > diff --git a/virtio-video.tex b/virtio-video.tex
> > new file mode 100644
> > index 0000000..30e728d
> > --- /dev/null
> > +++ b/virtio-video.tex
> > @@ -0,0 +1,579 @@
> > +\section{Video Device}\label{sec:Device Types / Video Device}
> > +
> > +The virtio video encoder device and decoder device are virtual devices
> > that +supports encoding and decoding respectively. Though the encoder and
> > the decoder +are different devices, they use the same protocol.
> > +
> > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> > +
> > +\begin{description}
> > +\item[30] encoder device
> > +\item[31] decoder device
> > +\end{description}
> > +
> > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > Virtqueues} +
> > +\begin{description}
> > +\item[0] controlq - queue for sending control commands.
> > +\item[1] eventq - queue for sending events happened in the device.
> > +\end{description}
> > +
> > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> > bits} +
> > +\begin{description}
> > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used
> > for
> > video +  buffers.
> > +\end{description}
> > +
> > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > Device
> > / Feature bits} +
> > +The device MUST offer at least one of feature bits.
> > +
> > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > Device / Device configuration layout} +
> > +Video device configuration uses the following layout structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_config {
> > +        le32 max_cap_len;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > +  MUST set this value.
> > +\end{description}
> > +
> > +\subsection{Device Initialization}\label{sec:Device Types / Video Device
> > /
> > Device Initialization} +
> > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > Video Device / Device Initialization} +
> > +The driver SHOULD query device capability by using the
> > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > +setup.
> > +
> > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > Device Operation} +
> > +The driver allocates input and output buffers and queues the buffers
> > +to the device. The device performs operations on the buffers according
> > +to the function in question.
> > +
> > +\subsubsection{Device Operation: Create stream}
> > +
> > +To process buffers, the device needs to associate them with a certain
> > +video stream (essentially, a context). Streams are created by
> > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > +determined by the device.
> > +
> > +\subsubsection{Device Operation: Create buffers}
> > +
> > +Buffers are used to store the actual data as well as the relevant
> > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > +be contiguous in guest physical memory.
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > +  resource that is backed by a buffer allocated from the driver's
> > +  memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > +  is no longer needed.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Stream parameter control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> > for +  input and output streams from the device.
> > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to
> > the +  device.
> > +\item After setting stream parameters, the driver may issue
> > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output
> > can be +  changed implicitly by the device during the set operation.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Process buffers}
> > +
> > +\begin{itemize*}
> > +\item If the function and the buffer type require so, write data to
> > +the buffer memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > +processing in the device.
> > +\item The request completes asynchronously when the device has
> > +finished with the buffer.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Buffer processing control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > +  return all of the already queued buffers.
> > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > +  already queued buffers from the input or the output queue. This also
> > +  includes input or output buffers that can be currently owned by the
> > +  device's processing pipeline.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Asynchronous events}
> > +
> > +While processing buffers, the device can send asynchronous event
> > +notifications to the driver. The behaviour depends on the exact
> > +stream. For example, the decoder device sends a resolution change
> > +event when it encounters new resolution metadata in the stream.
> > +
> > +\subsubsection{Device Operation: Request header}
> > +
> > +All requests and responses on the control virt queue have a fixed
> > +header using the following layout structure and definitions:
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_ctrl_type {
> > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > +
> > +        /* request */
> > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > +
> > +        /* response */
> > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > +
> > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > +};
> > +
> > +struct virtio_video_ctrl_hdr {
> > +        le32 type;
> > +        le32 stream_id;
> > +        le32 len; /* Length of the structure in bytes. */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{type}] is the type of the driver request or the device
> > +response.
> > +\item[\field{stream_id}] specifies a target stream.
> > +\item[\field{len}] is the length of data in bytes, which includes
> > +length of the header.
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: controlq}
> > +
> > +\begin{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > +supported formats.
> > +
> > +The driver uses \field{struct virtio_video_get_capability} to send a
> > +query request.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buf_type {
> > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > +};
> 
> I personally didn't like the previous term: pin_type. But, to be honest, I
> don't like the buf_type neither. Consider the GET/SET_PARAMS request:
> buf_type there looks a bit unnatural. We are trying to get stream
> parameters there, not some parameters of whatever buffer. Also I don't see
> any strict reason to mimic v4l2 naming scheme.
> 
> I'd better rename it to PORT_TYPE or QUEUE_TYPE.
> 
> > +
> > +struct virtio_video_get_capability {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        enum virtio_video_buf_type buf_type;
> > +};
> > +\end{lstlisting}
> > +\begin{description}
> > +\item[\field{buf_type}] is the buffer type that the driver asks
> > +information about. The driver MUST set either
> > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > +
> > +The device responds a capability by using \field{struct
> > +virtio_video_get_capability_resp}.
> > +\begin{lstlisting}
> > +enum virtio_video_format {
> > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > +        /* Raw formats */
> > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > +        VIRTIO_VIDEO_FORMAT_YVU420,
> 
> Let's add some variants of RGB, like RGBA, ARGB. We need it for the encoder
> in particular .
> 
> > +
> > +        /* Compressed formats */
> > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> 
> Let's add H265, MPEG4, MPEG2. We already support and use them.
> 
> Regards,
> Dmitry.
> 
> > +};
> > +
> > +enum virtio_video_profile {
> > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > +
> > +        /* H.264 */
> > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > +        /* VP8 */
> > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > +
> > +        /* VP9 */
> > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +};
> > +
> > +struct virtio_video_format_range {
> > +        le32 min;
> > +        le32 max;
> > +        le32 step;
> > +        u8 paddings[4];
> > +};
> > +
> > +struct virtio_video_format_desc {
> > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > +        le64 mask;
> > +        struct virtio_video_format_range width;
> > +        struct virtio_video_format_range height;
> > +        le32 num_rates;
> > +        u8 padding[4];
> > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > +};
> > +
> > +struct virtio_video_get_capability_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 num_descs;
> > +        /* Followed by struct virtio_video_format_desc desc[] */
> > +};
> > +\end{lstlisting}
> > +
We also see the need to add a max_streams value to this structure so as to 
explicitly provide a limit on the number of streams the guest can create.

Regards,
Dmitry.

> > +The format description \field{struct virtio_video_format_desc}
> > +includes the following fields:
> > +\begin{description}
> > +\item[\field{format}] specifies an image format. The device MUST set one
> > +  of \field{enum virtio_video_format}.
> > +\item[\field{profile}] specifies a profile of the compressed image format
> > +  specified in \field{format}. The driver SHOULD ignore this value if
> > +  \field{format} is a raw format.
> > +\item[\field{mask}] is a bitset that represents the supported
> > +  combination of input and output format. If \textit{i}-th bit is set
> > +  in \field{mask} of \textit{j}-th \field{struct
> > +  virtio_video_format_desc} for input, the device supports encoding or
> > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > +  format.
> > +\item[\field{width, height}] represents a range of resolutions
> > +  supported by the device. If its \field{step} is not applicable, its
> > +  \field{min} is equal to its \field{max}.
> > +\item[\field{num_rates}] is the length of an array \field{frame_rates}.
> > In
> > case of decoder, the driver SHOULD ignore this value.
> > +\item[\field{frame_rates}] is an array of supported frame rates.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_mem_type {
> > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > +};
> > +
> > +struct virtio_video_stream_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        char debug_name[64];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{in_mem_type}] is a type of buffer management for input
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{out_mem_type}] is a type of buffer management for output
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{debug_name}] is a text string for a debug purpose.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > +  queued buffers through the pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_drain {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_mem_entry {
> > +        le64 addr;
> > +        le32 length;
> > +        u8 padding[4];
> > +};
> > +
> > +struct virtio_video_resource_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        le32 nr_entries;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{nr_entries}] number of \field{struct
> > +  virtio_video_mem_entry} memory entries.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_resource_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > +queue.
> > +
> > +\begin{lstlisting}
> > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > +
> > +struct virtio_video_resource_queue {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        le32 resource_id;
> > +        le64 timestamp;
> > +        le32 nr_data_size;
> > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buf_type of the .
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > +\item[\field{data_size}] number of data bytes within a plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buffer_flag {
> > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > +        /* Encoder only */
> > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > +};
> > +
> > +struct virtio_video_resource_queue_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le64 timestamp;
> > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > +        le32 size;  /* Encoded size */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{flags}] mark specific buffers in the sequence.
> > +\item[\field{size}] data size in the buffer (encoder only).
> > +\end{description}
> > +
> > +The device sends a response to the queue request asynchronously when
> > +it has finished processing the buffer.
> > +
> > +The device SHOULD mark a buffer that triggered a processing error with
> > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > +
> > +The device MUST mark the last buffer with the
> > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > +sequence.
> > +
> > +In case of encoder, to denote a particular frame type the devie MUST
> > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > +  buffers back from the input or the output queue of the device. The
> > +  device SHOULD return all of the buffers from the respective queue as
> > +  soon as possible without pushing the buffers through the processing
> > +  pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_queue_clear {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > +  output of a stream.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_plane_format {
> > +        le32 plane_size;
> > +        le32 stride;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{plane_size}] size of the plane in bytes.
> > +\item[\field{stride}] stride used for the plane in bytes.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_params {
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > +        le32 frame_width;
> > +        le32 frame_height;
> > +        le32 min_buffers;
> > +        le32 max_buffers;
> > +        le32 frame_rate;
> > +        struct virtio_video_crop {
> > +                le32 left;
> > +                le32 top;
> > +                le32 width;
> > +                le32 height;
> > +        } crop;
> > +        le32 num_planes;
> > +        struct virtio_video_plane_format
> > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{frame_width}] the value to get/set.
> > +\item[\field{frame_height}] the value to get/set.
> > +\item[\field{pixel_format}] the value to get/set.
> > +\item[\field{min_buffers}] minimum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{max_buffers}] maximum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{frame_rate}] the value to get/set.
> > +\item[\field{crop}] cropping (composing) rectangle.
> > +\item[\field{num_planes}] number of planes used to store pixel data
> > +(r/o).
> > +\item[\field{plane_formats}] description of each plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_get_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +};
> > +
> > +struct virtio_video_get_params_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\item[\field{params}] parameter values.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > +
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_set_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{params}] parameters to set.
> > +\end{description}
> > +
> > +Setting stream parameters might have side effects within the device.
> > +For example, the device MAY perform alignment of width and height,
> > +change the number of planes it uses for the format, or do whatever
> > +changes that are required to continue normal operation using the
> > +updated parameters. It is up to the driver to check the parameter set
> > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > +
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: eventq}
> > +
> > +The device can report events on the event queue. The driver initially
> > +populates the queue with device-writeable buffers. When the device
> > +needs to report an event, it fills a buffer and notifies the driver.
> > +The driver consumes the report and adds a new buffer to the virtqueue.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_event_type {
> > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > +        /* For all functions */
> > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > +        /* For decoder only */
> > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > +};
> > +
> > +struct virtio_video_event {
> > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > +        le32 stream_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{event_type}] type of the triggered event .
> > +\item[\field{stream_id}] id of the source stream.
> > +\end{description}
> > +
> > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > +whenever it encounters new resolution data in the stream. This
> > +includes the case of the initial device configuration after metadata
> > +has been parsed and the case of dynamic resolution change.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
Dmitry Sepp Jan. 3, 2020, 3:47 p.m. UTC | #20
Hi Keiichi,

On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> 
> The virtio video encoder device and decoder device provide functionalities
> to encode and decode video stream respectively.
> Though video encoder and decoder are provided as different devices, they use
> a same protocol.
> 
> Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> ---
>  content.tex      |   1 +
>  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 580 insertions(+)
>  create mode 100644 virtio-video.tex
> 
> diff --git a/content.tex b/content.tex
> index 556b373..9e56839 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> Requirements}\label{sec:Device \input{virtio-vsock.tex}
>  \input{virtio-fs.tex}
>  \input{virtio-rpmb.tex}
> +\input{virtio-video.tex}
> 
>  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> 
> diff --git a/virtio-video.tex b/virtio-video.tex
> new file mode 100644
> index 0000000..30e728d
> --- /dev/null
> +++ b/virtio-video.tex
> @@ -0,0 +1,579 @@
> +\section{Video Device}\label{sec:Device Types / Video Device}
> +
> +The virtio video encoder device and decoder device are virtual devices that
> +supports encoding and decoding respectively. Though the encoder and the
> decoder +are different devices, they use the same protocol.
> +
> +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> +
> +\begin{description}
> +\item[30] encoder device
> +\item[31] decoder device
> +\end{description}
> +
> +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> +
> +\begin{description}
> +\item[0] controlq - queue for sending control commands.
> +\item[1] eventq - queue for sending events happened in the device.
> +\end{description}
> +
> +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> bits} +
> +\begin{description}
> +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> video +  buffers.
> +\end{description}
> +
> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device
> / Feature bits} +
> +The device MUST offer at least one of feature bits.
> +
> +\subsection{Device configuration layout}\label{sec:Device Types / Video
> Device / Device configuration layout} +
> +Video device configuration uses the following layout structure:
> +
> +\begin{lstlisting}
> +struct virtio_video_config {
> +        le32 max_cap_len;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> +  MUST set this value.
> +\end{description}
> +
> +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> Device Initialization} +
> +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> Video Device / Device Initialization} +
> +The driver SHOULD query device capability by using the
> +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> +setup.
> +
> +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> Device Operation} +
> +The driver allocates input and output buffers and queues the buffers
> +to the device. The device performs operations on the buffers according
> +to the function in question.
> +
> +\subsubsection{Device Operation: Create stream}
> +
> +To process buffers, the device needs to associate them with a certain
> +video stream (essentially, a context). Streams are created by
> +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> +determined by the device.
> +
> +\subsubsection{Device Operation: Create buffers}
> +
> +Buffers are used to store the actual data as well as the relevant
> +metadata. Scatter lists are supported, so the buffer doesn't need to
> +be contiguous in guest physical memory.
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> +  resource that is backed by a buffer allocated from the driver's
> +  memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> +  is no longer needed.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Stream parameter control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> for +  input and output streams from the device.
> +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> +  device.
> +\item After setting stream parameters, the driver may issue
> +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> be +  changed implicitly by the device during the set operation.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Process buffers}
> +
> +\begin{itemize*}
> +\item If the function and the buffer type require so, write data to
> +the buffer memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> +processing in the device.
> +\item The request completes asynchronously when the device has
> +finished with the buffer.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Buffer processing control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> +  return all of the already queued buffers.
> +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> +  already queued buffers from the input or the output queue. This also
> +  includes input or output buffers that can be currently owned by the
> +  device's processing pipeline.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Asynchronous events}
> +
> +While processing buffers, the device can send asynchronous event
> +notifications to the driver. The behaviour depends on the exact
> +stream. For example, the decoder device sends a resolution change
> +event when it encounters new resolution metadata in the stream.
> +
> +\subsubsection{Device Operation: Request header}
> +
> +All requests and responses on the control virt queue have a fixed
> +header using the following layout structure and definitions:
> +
> +\begin{lstlisting}
> +enum virtio_video_ctrl_type {
> +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> +
> +        /* request */
> +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> +        VIRTIO_VIDEO_T_STREAM_CREATE,
> +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> +        VIRTIO_VIDEO_T_SET_PARAMS,
> +        VIRTIO_VIDEO_T_GET_PARAMS,
> +
> +        /* response */
> +        VIRTIO_VIDEO_S_OK = 0x0200,
> +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> +
> +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> +};
> +
> +struct virtio_video_ctrl_hdr {
> +        le32 type;
> +        le32 stream_id;
> +        le32 len; /* Length of the structure in bytes. */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{type}] is the type of the driver request or the device
> +response.
> +\item[\field{stream_id}] specifies a target stream.
> +\item[\field{len}] is the length of data in bytes, which includes
> +length of the header.
> +\end{description}
> +
> +\subsubsection{Device Operation: controlq}
> +
> +\begin{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> +supported formats.
> +
> +The driver uses \field{struct virtio_video_get_capability} to send a
> +query request.
> +
> +\begin{lstlisting}
> +enum virtio_video_buf_type {
> +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> +};
> +
> +struct virtio_video_get_capability {
> +        struct virtio_video_ctrl_hdr hdr;
> +        enum virtio_video_buf_type buf_type;
> +};
> +\end{lstlisting}
> +\begin{description}
> +\item[\field{buf_type}] is the buffer type that the driver asks
> +information about. The driver MUST set either
> +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> +
> +The device responds a capability by using \field{struct
> +virtio_video_get_capability_resp}.
> +\begin{lstlisting}
> +enum virtio_video_format {
> +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> +        /* Raw formats */
> +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> +        VIRTIO_VIDEO_FORMAT_YUV420,
> +        VIRTIO_VIDEO_FORMAT_YVU420,
> +
> +        /* Compressed formats */
> +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> +};
> +
> +enum virtio_video_profile {
> +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> +
> +        /* H.264 */
> +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> VIRTIO_VIDEO_PROFILE_H264_BASELINE, +       
> VIRTIO_VIDEO_PROFILE_H264_MAIN,
> +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> +        /* VP8 */
> +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> +
> +        /* VP9 */
> +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +};
> +
> +struct virtio_video_format_range {
> +        le32 min;
> +        le32 max;
> +        le32 step;
> +        u8 paddings[4];
> +};
> +
> +struct virtio_video_format_desc {
> +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> +        le64 mask;
> +        struct virtio_video_format_range width;
> +        struct virtio_video_format_range height;
> +        le32 num_rates;
> +        u8 padding[4];
> +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> +};
> +
> +struct virtio_video_get_capability_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 num_descs;
> +        /* Followed by struct virtio_video_format_desc desc[] */
> +};
> +\end{lstlisting}
> +
> +The format description \field{struct virtio_video_format_desc}
> +includes the following fields:
> +\begin{description}
> +\item[\field{format}] specifies an image format. The device MUST set one
> +  of \field{enum virtio_video_format}.
> +\item[\field{profile}] specifies a profile of the compressed image format
> +  specified in \field{format}. The driver SHOULD ignore this value if
> +  \field{format} is a raw format.
> +\item[\field{mask}] is a bitset that represents the supported
> +  combination of input and output format. If \textit{i}-th bit is set
> +  in \field{mask} of \textit{j}-th \field{struct
> +  virtio_video_format_desc} for input, the device supports encoding or
> +  decoding from the \textit{j}-th input format to \textit{i}-th output
> +  format.
> +\item[\field{width, height}] represents a range of resolutions
> +  supported by the device. If its \field{step} is not applicable, its
> +  \field{min} is equal to its \field{max}.
> +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> case of decoder, the driver SHOULD ignore this value.
> +\item[\field{frame_rates}] is an array of supported frame rates.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +enum virtio_video_mem_type {
> +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> +};
> +
> +struct virtio_video_stream_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        char debug_name[64];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{in_mem_type}] is a type of buffer management for input
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{out_mem_type}] is a type of buffer management for output
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{debug_name}] is a text string for a debug purpose.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> +  queued buffers through the pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_drain {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_mem_entry {
> +        le64 addr;
> +        le32 length;
> +        u8 padding[4];
> +};
> +
> +struct virtio_video_resource_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        le32 nr_entries;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{nr_entries}] number of \field{struct
> +  virtio_video_mem_entry} memory entries.
> +\end{description}
> +

How should one deal with multiplanar formats? Do we create one resource per 
plane? Otherwise we need a way to send mem entries for each plane in one 
request.

Currently the v1 driver attaches backing to the same resource multiple times 
when a buffer has multiple planes. Not the nicest way to be honest.

Regards,
Dmitry.

> +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_resource_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> +queue.
> +
> +\begin{lstlisting}
> +#define VIRTIO_VIDEO_MAX_PLANES 8
> +
> +struct virtio_video_resource_queue {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        le32 resource_id;
> +        le64 timestamp;
> +        le32 nr_data_size;
> +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buf_type of the .
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{nr_data_size}] number of \field{data_size} entries.
> +\item[\field{data_size}] number of data bytes within a plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +enum virtio_video_buffer_flag {
> +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> +        /* Encoder only */
> +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> +};
> +
> +struct virtio_video_resource_queue_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le64 timestamp;
> +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> +        le32 size;  /* Encoded size */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{flags}] mark specific buffers in the sequence.
> +\item[\field{size}] data size in the buffer (encoder only).
> +\end{description}
> +
> +The device sends a response to the queue request asynchronously when
> +it has finished processing the buffer.
> +
> +The device SHOULD mark a buffer that triggered a processing error with
> +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> +
> +The device MUST mark the last buffer with the
> +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> +sequence.
> +
> +In case of encoder, to denote a particular frame type the devie MUST
> +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> +  buffers back from the input or the output queue of the device. The
> +  device SHOULD return all of the buffers from the respective queue as
> +  soon as possible without pushing the buffers through the processing
> +  pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_queue_clear {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> +  output of a stream.
> +
> +\begin{lstlisting}
> +struct virtio_video_plane_format {
> +        le32 plane_size;
> +        le32 stride;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{plane_size}] size of the plane in bytes.
> +\item[\field{stride}] stride used for the plane in bytes.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_params {
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> +        le32 frame_width;
> +        le32 frame_height;
> +        le32 min_buffers;
> +        le32 max_buffers;
> +        le32 frame_rate;
> +        struct virtio_video_crop {
> +                le32 left;
> +                le32 top;
> +                le32 width;
> +                le32 height;
> +        } crop;
> +        le32 num_planes;
> +        struct virtio_video_plane_format
> plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{frame_width}] the value to get/set.
> +\item[\field{frame_height}] the value to get/set.
> +\item[\field{pixel_format}] the value to get/set.
> +\item[\field{min_buffers}] minimum buffers required to handle the
> +  format (r/o).
> +\item[\field{max_buffers}] maximum buffers required to handle the
> +  format (r/o).
> +\item[\field{frame_rate}] the value to get/set.
> +\item[\field{crop}] cropping (composing) rectangle.
> +\item[\field{num_planes}] number of planes used to store pixel data
> +(r/o).
> +\item[\field{plane_formats}] description of each plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_get_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +};
> +
> +struct virtio_video_get_params_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\item[\field{params}] parameter values.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> +
> +
> +\begin{lstlisting}
> +struct virtio_video_set_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{params}] parameters to set.
> +\end{description}
> +
> +Setting stream parameters might have side effects within the device.
> +For example, the device MAY perform alignment of width and height,
> +change the number of planes it uses for the format, or do whatever
> +changes that are required to continue normal operation using the
> +updated parameters. It is up to the driver to check the parameter set
> +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> +
> +\end{description}
> +
> +\subsubsection{Device Operation: eventq}
> +
> +The device can report events on the event queue. The driver initially
> +populates the queue with device-writeable buffers. When the device
> +needs to report an event, it fills a buffer and notifies the driver.
> +The driver consumes the report and adds a new buffer to the virtqueue.
> +
> +\begin{lstlisting}
> +enum virtio_video_event_type {
> +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> +        /* For all functions */
> +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> +        /* For decoder only */
> +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> +};
> +
> +struct virtio_video_event {
> +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> +        le32 stream_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{event_type}] type of the triggered event .
> +\item[\field{stream_id}] id of the source stream.
> +\end{description}
> +
> +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> +whenever it encounters new resolution data in the stream. This
> +includes the case of the initial device configuration after metadata
> +has been parsed and the case of dynamic resolution change.
Tomasz Figa Jan. 6, 2020, 6:31 a.m. UTC | #21
On Mon, Dec 30, 2019 at 9:16 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
>
> On Donnerstag, 19. Dezember 2019 14:28:23 CET Dmitry Sepp wrote:
> > Hi Keiichi,
> >
> > Thank you for the update. Please see some comments below.
> >
> > Also, we need to bring the virtio_video_control back as it is in fact used
> > by the driver to enumerate supported encoder controls. But yes, it still
> > needs to be documemnted, it's true.
> >
> > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > >
> > > The virtio video encoder device and decoder device provide functionalities
> > > to encode and decode video stream respectively.
> > > Though video encoder and decoder are provided as different devices, they
> > > use a same protocol.
> > >
> > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > ---
> > >
> > >  content.tex      |   1 +
> > >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 580 insertions(+)
> > >  create mode 100644 virtio-video.tex
> > >
> > > diff --git a/content.tex b/content.tex
> > > index 556b373..9e56839 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > >
> > >  \input{virtio-fs.tex}
> > >  \input{virtio-rpmb.tex}
> > >
> > > +\input{virtio-video.tex}
> > >
> > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > >
> > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > new file mode 100644
> > > index 0000000..30e728d
> > > --- /dev/null
> > > +++ b/virtio-video.tex
> > > @@ -0,0 +1,579 @@
> > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > +
> > > +The virtio video encoder device and decoder device are virtual devices
> > > that +supports encoding and decoding respectively. Though the encoder and
> > > the decoder +are different devices, they use the same protocol.
> > > +
> > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> > > +
> > > +\begin{description}
> > > +\item[30] encoder device
> > > +\item[31] decoder device
> > > +\end{description}
> > > +
> > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > Virtqueues} +
> > > +\begin{description}
> > > +\item[0] controlq - queue for sending control commands.
> > > +\item[1] eventq - queue for sending events happened in the device.
> > > +\end{description}
> > > +
> > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> > > bits} +
> > > +\begin{description}
> > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used
> > > for
> > > video +  buffers.
> > > +\end{description}
> > > +
> > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > Device
> > > / Feature bits} +
> > > +The device MUST offer at least one of feature bits.
> > > +
> > > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > > Device / Device configuration layout} +
> > > +Video device configuration uses the following layout structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_config {
> > > +        le32 max_cap_len;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > +  MUST set this value.
> > > +\end{description}
> > > +
> > > +\subsection{Device Initialization}\label{sec:Device Types / Video Device
> > > /
> > > Device Initialization} +
> > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > > Video Device / Device Initialization} +
> > > +The driver SHOULD query device capability by using the
> > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > > +setup.
> > > +
> > > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > > Device Operation} +
> > > +The driver allocates input and output buffers and queues the buffers
> > > +to the device. The device performs operations on the buffers according
> > > +to the function in question.
> > > +
> > > +\subsubsection{Device Operation: Create stream}
> > > +
> > > +To process buffers, the device needs to associate them with a certain
> > > +video stream (essentially, a context). Streams are created by
> > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > +determined by the device.
> > > +
> > > +\subsubsection{Device Operation: Create buffers}
> > > +
> > > +Buffers are used to store the actual data as well as the relevant
> > > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > > +be contiguous in guest physical memory.
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > +  resource that is backed by a buffer allocated from the driver's
> > > +  memory.
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > > +  is no longer needed.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Stream parameter control}
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> > > for +  input and output streams from the device.
> > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to
> > > the +  device.
> > > +\item After setting stream parameters, the driver may issue
> > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output
> > > can be +  changed implicitly by the device during the set operation.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Process buffers}
> > > +
> > > +\begin{itemize*}
> > > +\item If the function and the buffer type require so, write data to
> > > +the buffer memory.
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > +processing in the device.
> > > +\item The request completes asynchronously when the device has
> > > +finished with the buffer.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Buffer processing control}
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > > +  return all of the already queued buffers.
> > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > > +  already queued buffers from the input or the output queue. This also
> > > +  includes input or output buffers that can be currently owned by the
> > > +  device's processing pipeline.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Asynchronous events}
> > > +
> > > +While processing buffers, the device can send asynchronous event
> > > +notifications to the driver. The behaviour depends on the exact
> > > +stream. For example, the decoder device sends a resolution change
> > > +event when it encounters new resolution metadata in the stream.
> > > +
> > > +\subsubsection{Device Operation: Request header}
> > > +
> > > +All requests and responses on the control virt queue have a fixed
> > > +header using the following layout structure and definitions:
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_ctrl_type {
> > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > +
> > > +        /* request */
> > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > +
> > > +        /* response */
> > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > +
> > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > +};
> > > +
> > > +struct virtio_video_ctrl_hdr {
> > > +        le32 type;
> > > +        le32 stream_id;
> > > +        le32 len; /* Length of the structure in bytes. */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{type}] is the type of the driver request or the device
> > > +response.
> > > +\item[\field{stream_id}] specifies a target stream.
> > > +\item[\field{len}] is the length of data in bytes, which includes
> > > +length of the header.
> > > +\end{description}
> > > +
> > > +\subsubsection{Device Operation: controlq}
> > > +
> > > +\begin{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > +supported formats.
> > > +
> > > +The driver uses \field{struct virtio_video_get_capability} to send a
> > > +query request.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_buf_type {
> > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > +};
> >
> > I personally didn't like the previous term: pin_type. But, to be honest, I
> > don't like the buf_type neither. Consider the GET/SET_PARAMS request:
> > buf_type there looks a bit unnatural. We are trying to get stream
> > parameters there, not some parameters of whatever buffer. Also I don't see
> > any strict reason to mimic v4l2 naming scheme.
> >
> > I'd better rename it to PORT_TYPE or QUEUE_TYPE.
> >
> > > +
> > > +struct virtio_video_get_capability {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        enum virtio_video_buf_type buf_type;
> > > +};
> > > +\end{lstlisting}
> > > +\begin{description}
> > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > +information about. The driver MUST set either
> > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > +
> > > +The device responds a capability by using \field{struct
> > > +virtio_video_get_capability_resp}.
> > > +\begin{lstlisting}
> > > +enum virtio_video_format {
> > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > +        /* Raw formats */
> > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> >
> > Let's add some variants of RGB, like RGBA, ARGB. We need it for the encoder
> > in particular .
> >
> > > +
> > > +        /* Compressed formats */
> > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> >
> > Let's add H265, MPEG4, MPEG2. We already support and use them.
> >
> > Regards,
> > Dmitry.
> >
> > > +};
> > > +
> > > +enum virtio_video_profile {
> > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > +
> > > +        /* H.264 */
> > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > +        /* VP8 */
> > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > +
> > > +        /* VP9 */
> > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > +};
> > > +
> > > +struct virtio_video_format_range {
> > > +        le32 min;
> > > +        le32 max;
> > > +        le32 step;
> > > +        u8 paddings[4];
> > > +};
> > > +
> > > +struct virtio_video_format_desc {
> > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > +        le64 mask;
> > > +        struct virtio_video_format_range width;
> > > +        struct virtio_video_format_range height;
> > > +        le32 num_rates;
> > > +        u8 padding[4];
> > > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > > +};
> > > +
> > > +struct virtio_video_get_capability_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 num_descs;
> > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > +};
> > > +\end{lstlisting}
> > > +
> We also see the need to add a max_streams value to this structure so as to
> explicitly provide a limit on the number of streams the guest can create.

What would be the advantage over just trying to create one and
failing? The maximum number would be only meaningful for the special
case when the streams are always only created by one user space
process. Otherwise, if several processes do that, they are not aware
of each other and the number could be higher than they can actually
create, because other processes could have some streams created
already.

Best regards,
Tomasz

>
> Regards,
> Dmitry.
>
> > > +The format description \field{struct virtio_video_format_desc}
> > > +includes the following fields:
> > > +\begin{description}
> > > +\item[\field{format}] specifies an image format. The device MUST set one
> > > +  of \field{enum virtio_video_format}.
> > > +\item[\field{profile}] specifies a profile of the compressed image format
> > > +  specified in \field{format}. The driver SHOULD ignore this value if
> > > +  \field{format} is a raw format.
> > > +\item[\field{mask}] is a bitset that represents the supported
> > > +  combination of input and output format. If \textit{i}-th bit is set
> > > +  in \field{mask} of \textit{j}-th \field{struct
> > > +  virtio_video_format_desc} for input, the device supports encoding or
> > > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > > +  format.
> > > +\item[\field{width, height}] represents a range of resolutions
> > > +  supported by the device. If its \field{step} is not applicable, its
> > > +  \field{min} is equal to its \field{max}.
> > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}.
> > > In
> > > case of decoder, the driver SHOULD ignore this value.
> > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_mem_type {
> > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > +};
> > > +
> > > +struct virtio_video_stream_create {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > +        char debug_name[64];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > +buffers. The driver MUST set a value in \field{enum
> > > +virtio_video_mem_type}.
> > > +\item[\field{out_mem_type}] is a type of buffer management for output
> > > +buffers. The driver MUST set a value in \field{enum
> > > +virtio_video_mem_type}.
> > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_stream_destroy {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > +  queued buffers through the pipeline.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_stream_drain {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_mem_entry {
> > > +        le64 addr;
> > > +        le32 length;
> > > +        u8 padding[4];
> > > +};
> > > +
> > > +struct virtio_video_resource_create {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 resource_id;
> > > +        le32 nr_entries;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\item[\field{nr_entries}] number of \field{struct
> > > +  virtio_video_mem_entry} memory entries.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_resource_destroy {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 resource_id;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > +queue.
> > > +
> > > +\begin{lstlisting}
> > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > +
> > > +struct virtio_video_resource_queue {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type;
> > > +        le32 resource_id;
> > > +        le64 timestamp;
> > > +        le32 nr_data_size;
> > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buf_type of the .
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > +  for synchronisation.
> > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > +\item[\field{data_size}] number of data bytes within a plane.
> > > +\end{description}
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_buffer_flag {
> > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > +        /* Encoder only */
> > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > +};
> > > +
> > > +struct virtio_video_resource_queue_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le64 timestamp;
> > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > +        le32 size;  /* Encoded size */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > +  for synchronisation.
> > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > +\item[\field{size}] data size in the buffer (encoder only).
> > > +\end{description}
> > > +
> > > +The device sends a response to the queue request asynchronously when
> > > +it has finished processing the buffer.
> > > +
> > > +The device SHOULD mark a buffer that triggered a processing error with
> > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > +
> > > +The device MUST mark the last buffer with the
> > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > +sequence.
> > > +
> > > +In case of encoder, to denote a particular frame type the devie MUST
> > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > +  buffers back from the input or the output queue of the device. The
> > > +  device SHOULD return all of the buffers from the respective queue as
> > > +  soon as possible without pushing the buffers through the processing
> > > +  pipeline.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_queue_clear {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buffer type.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > +  output of a stream.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_plane_format {
> > > +        le32 plane_size;
> > > +        le32 stride;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{plane_size}] size of the plane in bytes.
> > > +\item[\field{stride}] stride used for the plane in bytes.
> > > +\end{description}
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_params {
> > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > +        le32 frame_width;
> > > +        le32 frame_height;
> > > +        le32 min_buffers;
> > > +        le32 max_buffers;
> > > +        le32 frame_rate;
> > > +        struct virtio_video_crop {
> > > +                le32 left;
> > > +                le32 top;
> > > +                le32 width;
> > > +                le32 height;
> > > +        } crop;
> > > +        le32 num_planes;
> > > +        struct virtio_video_plane_format
> > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{frame_width}] the value to get/set.
> > > +\item[\field{frame_height}] the value to get/set.
> > > +\item[\field{pixel_format}] the value to get/set.
> > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > +  format (r/o).
> > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > +  format (r/o).
> > > +\item[\field{frame_rate}] the value to get/set.
> > > +\item[\field{crop}] cropping (composing) rectangle.
> > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > +(r/o).
> > > +\item[\field{plane_formats}] description of each plane.
> > > +\end{description}
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_get_params {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > +};
> > > +
> > > +struct virtio_video_get_params_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        struct virtio_video_params params;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buffer type.
> > > +\item[\field{params}] parameter values.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > +
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_set_params {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        struct virtio_video_params params;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{params}] parameters to set.
> > > +\end{description}
> > > +
> > > +Setting stream parameters might have side effects within the device.
> > > +For example, the device MAY perform alignment of width and height,
> > > +change the number of planes it uses for the format, or do whatever
> > > +changes that are required to continue normal operation using the
> > > +updated parameters. It is up to the driver to check the parameter set
> > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > +
> > > +\end{description}
> > > +
> > > +\subsubsection{Device Operation: eventq}
> > > +
> > > +The device can report events on the event queue. The driver initially
> > > +populates the queue with device-writeable buffers. When the device
> > > +needs to report an event, it fills a buffer and notifies the driver.
> > > +The driver consumes the report and adds a new buffer to the virtqueue.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_event_type {
> > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > +        /* For all functions */
> > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > +        /* For decoder only */
> > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > +};
> > > +
> > > +struct virtio_video_event {
> > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > +        le32 stream_id;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{event_type}] type of the triggered event .
> > > +\item[\field{stream_id}] id of the source stream.
> > > +\end{description}
> > > +
> > > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > +whenever it encounters new resolution data in the stream. This
> > > +includes the case of the initial device configuration after metadata
> > > +has been parsed and the case of dynamic resolution change.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>
>
Gerd Hoffmann Jan. 6, 2020, 8:33 a.m. UTC | #22
Hi,

> > We also see the need to add a max_streams value to this structure so as to
> > explicitly provide a limit on the number of streams the guest can create.
> 
> What would be the advantage over just trying to create one and
> failing? The maximum number would be only meaningful for the special
> case when the streams are always only created by one user space
> process. Otherwise, if several processes do that, they are not aware
> of each other and the number could be higher than they can actually
> create, because other processes could have some streams created
> already.

Also the number of streams might not be fixed but depend on stream
parameters, i.e. hardware can decode one hd or two sd streams ...

cheers,
  Gerd
Gerd Hoffmann Jan. 6, 2020, 8:47 a.m. UTC | #23
Hi,

> How should one deal with multiplanar formats? Do we create one resource per 
> plane? Otherwise we need a way to send mem entries for each plane in one 
> request.

DRM uses arrays of handles and offsets (see struct drm_framebuffer).  A
handle references a gem object (roughly the same as a resource), and the
offset specifies the start of the plane within the gem object.  That
allows both a single gem object with planes stored at different offsets
and one gem object per plane.  virtio-video could do the same, or pick
one of the two approaches and support only that.

cheers,
  Gerd
Dmitry Sepp Jan. 6, 2020, 9:29 a.m. UTC | #24
Hi,

On Montag, 6. Januar 2020 09:33:35 CET Gerd Hoffmann wrote:
>   Hi,
> 
> > > We also see the need to add a max_streams value to this structure so as
> > > to
> > > explicitly provide a limit on the number of streams the guest can
> > > create.
> > 
> > What would be the advantage over just trying to create one and
> > failing? The maximum number would be only meaningful for the special
> > case when the streams are always only created by one user space
> > process. Otherwise, if several processes do that, they are not aware
> > of each other and the number could be higher than they can actually
> > create, because other processes could have some streams created
> > already.
> 
> Also the number of streams might not be fixed but depend on stream
> parameters, i.e. hardware can decode one hd or two sd streams ...
> 
Ok, you are right. We'd better return an error from the device side.

Regards,
Dmitry.

> cheers,
>   Gerd
Keiichi Watanabe Jan. 6, 2020, 10:21 a.m. UTC | #25
Hi Dmitry,

On Sat, Jan 4, 2020 at 12:47 AM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi Keiichi,
>
> On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> >
> > The virtio video encoder device and decoder device provide functionalities
> > to encode and decode video stream respectively.
> > Though video encoder and decoder are provided as different devices, they use
> > a same protocol.
> >
> > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > ---
> >  content.tex      |   1 +
> >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 580 insertions(+)
> >  create mode 100644 virtio-video.tex
> >
> > diff --git a/content.tex b/content.tex
> > index 556b373..9e56839 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> >  \input{virtio-fs.tex}
> >  \input{virtio-rpmb.tex}
> > +\input{virtio-video.tex}
> >
> >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >
> > diff --git a/virtio-video.tex b/virtio-video.tex
> > new file mode 100644
> > index 0000000..30e728d
> > --- /dev/null
> > +++ b/virtio-video.tex
> > @@ -0,0 +1,579 @@
> > +\section{Video Device}\label{sec:Device Types / Video Device}
> > +
> > +The virtio video encoder device and decoder device are virtual devices that
> > +supports encoding and decoding respectively. Though the encoder and the
> > decoder +are different devices, they use the same protocol.
> > +
> > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> > +
> > +\begin{description}
> > +\item[30] encoder device
> > +\item[31] decoder device
> > +\end{description}
> > +
> > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> > +
> > +\begin{description}
> > +\item[0] controlq - queue for sending control commands.
> > +\item[1] eventq - queue for sending events happened in the device.
> > +\end{description}
> > +
> > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> > bits} +
> > +\begin{description}
> > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> > video +  buffers.
> > +\end{description}
> > +
> > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device
> > / Feature bits} +
> > +The device MUST offer at least one of feature bits.
> > +
> > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > Device / Device configuration layout} +
> > +Video device configuration uses the following layout structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_config {
> > +        le32 max_cap_len;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > +  MUST set this value.
> > +\end{description}
> > +
> > +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> > Device Initialization} +
> > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > Video Device / Device Initialization} +
> > +The driver SHOULD query device capability by using the
> > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > +setup.
> > +
> > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > Device Operation} +
> > +The driver allocates input and output buffers and queues the buffers
> > +to the device. The device performs operations on the buffers according
> > +to the function in question.
> > +
> > +\subsubsection{Device Operation: Create stream}
> > +
> > +To process buffers, the device needs to associate them with a certain
> > +video stream (essentially, a context). Streams are created by
> > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > +determined by the device.
> > +
> > +\subsubsection{Device Operation: Create buffers}
> > +
> > +Buffers are used to store the actual data as well as the relevant
> > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > +be contiguous in guest physical memory.
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > +  resource that is backed by a buffer allocated from the driver's
> > +  memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > +  is no longer needed.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Stream parameter control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> > for +  input and output streams from the device.
> > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> > +  device.
> > +\item After setting stream parameters, the driver may issue
> > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> > be +  changed implicitly by the device during the set operation.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Process buffers}
> > +
> > +\begin{itemize*}
> > +\item If the function and the buffer type require so, write data to
> > +the buffer memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > +processing in the device.
> > +\item The request completes asynchronously when the device has
> > +finished with the buffer.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Buffer processing control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > +  return all of the already queued buffers.
> > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > +  already queued buffers from the input or the output queue. This also
> > +  includes input or output buffers that can be currently owned by the
> > +  device's processing pipeline.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Asynchronous events}
> > +
> > +While processing buffers, the device can send asynchronous event
> > +notifications to the driver. The behaviour depends on the exact
> > +stream. For example, the decoder device sends a resolution change
> > +event when it encounters new resolution metadata in the stream.
> > +
> > +\subsubsection{Device Operation: Request header}
> > +
> > +All requests and responses on the control virt queue have a fixed
> > +header using the following layout structure and definitions:
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_ctrl_type {
> > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > +
> > +        /* request */
> > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > +
> > +        /* response */
> > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > +
> > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > +};
> > +
> > +struct virtio_video_ctrl_hdr {
> > +        le32 type;
> > +        le32 stream_id;
> > +        le32 len; /* Length of the structure in bytes. */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{type}] is the type of the driver request or the device
> > +response.
> > +\item[\field{stream_id}] specifies a target stream.
> > +\item[\field{len}] is the length of data in bytes, which includes
> > +length of the header.
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: controlq}
> > +
> > +\begin{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > +supported formats.
> > +
> > +The driver uses \field{struct virtio_video_get_capability} to send a
> > +query request.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buf_type {
> > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > +};
> > +
> > +struct virtio_video_get_capability {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        enum virtio_video_buf_type buf_type;
> > +};
> > +\end{lstlisting}
> > +\begin{description}
> > +\item[\field{buf_type}] is the buffer type that the driver asks
> > +information about. The driver MUST set either
> > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > +
> > +The device responds a capability by using \field{struct
> > +virtio_video_get_capability_resp}.
> > +\begin{lstlisting}
> > +enum virtio_video_format {
> > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > +        /* Raw formats */
> > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > +
> > +        /* Compressed formats */
> > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > +};
> > +
> > +enum virtio_video_profile {
> > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > +
> > +        /* H.264 */
> > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > +        /* VP8 */
> > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > +
> > +        /* VP9 */
> > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +};
> > +
> > +struct virtio_video_format_range {
> > +        le32 min;
> > +        le32 max;
> > +        le32 step;
> > +        u8 paddings[4];
> > +};
> > +
> > +struct virtio_video_format_desc {
> > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > +        le64 mask;
> > +        struct virtio_video_format_range width;
> > +        struct virtio_video_format_range height;
> > +        le32 num_rates;
> > +        u8 padding[4];
> > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > +};
> > +
> > +struct virtio_video_get_capability_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 num_descs;
> > +        /* Followed by struct virtio_video_format_desc desc[] */
> > +};
> > +\end{lstlisting}
> > +
> > +The format description \field{struct virtio_video_format_desc}
> > +includes the following fields:
> > +\begin{description}
> > +\item[\field{format}] specifies an image format. The device MUST set one
> > +  of \field{enum virtio_video_format}.
> > +\item[\field{profile}] specifies a profile of the compressed image format
> > +  specified in \field{format}. The driver SHOULD ignore this value if
> > +  \field{format} is a raw format.
> > +\item[\field{mask}] is a bitset that represents the supported
> > +  combination of input and output format. If \textit{i}-th bit is set
> > +  in \field{mask} of \textit{j}-th \field{struct
> > +  virtio_video_format_desc} for input, the device supports encoding or
> > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > +  format.
> > +\item[\field{width, height}] represents a range of resolutions
> > +  supported by the device. If its \field{step} is not applicable, its
> > +  \field{min} is equal to its \field{max}.
> > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> > case of decoder, the driver SHOULD ignore this value.
> > +\item[\field{frame_rates}] is an array of supported frame rates.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_mem_type {
> > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > +};
> > +
> > +struct virtio_video_stream_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        char debug_name[64];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{in_mem_type}] is a type of buffer management for input
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{out_mem_type}] is a type of buffer management for output
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{debug_name}] is a text string for a debug purpose.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > +  queued buffers through the pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_drain {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_mem_entry {
> > +        le64 addr;
> > +        le32 length;
> > +        u8 padding[4];
> > +};
> > +
> > +struct virtio_video_resource_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        le32 nr_entries;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{nr_entries}] number of \field{struct
> > +  virtio_video_mem_entry} memory entries.
> > +\end{description}
> > +
>
> How should one deal with multiplanar formats? Do we create one resource per
> plane? Otherwise we need a way to send mem entries for each plane in one
> request.

That's a good point. We should be able to create a resource that
corresponds multiple planes.
How about changing the struct like this:

struct virtio_video_resource_create {
        struct virtio_video_ctrl_hdr hdr;
        le32 resource_id;
        le32 nums_entries[VIRTIO_VIDEO_MAX_PLANES];
        u8 padding[4];
        /* Followed by struct virtio_video_mem_entry entries[].
           |length of entries[]| == |sum of nums_entries[i] for each i| */
};

Best regards,
Keiichi

>
> Currently the v1 driver attaches backing to the same resource multiple times
> when a buffer has multiple planes. Not the nicest way to be honest.
>
> Regards,
> Dmitry.
>
> > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_resource_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > +queue.
> > +
> > +\begin{lstlisting}
> > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > +
> > +struct virtio_video_resource_queue {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        le32 resource_id;
> > +        le64 timestamp;
> > +        le32 nr_data_size;
> > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buf_type of the .
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > +\item[\field{data_size}] number of data bytes within a plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buffer_flag {
> > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > +        /* Encoder only */
> > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > +};
> > +
> > +struct virtio_video_resource_queue_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le64 timestamp;
> > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > +        le32 size;  /* Encoded size */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{flags}] mark specific buffers in the sequence.
> > +\item[\field{size}] data size in the buffer (encoder only).
> > +\end{description}
> > +
> > +The device sends a response to the queue request asynchronously when
> > +it has finished processing the buffer.
> > +
> > +The device SHOULD mark a buffer that triggered a processing error with
> > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > +
> > +The device MUST mark the last buffer with the
> > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > +sequence.
> > +
> > +In case of encoder, to denote a particular frame type the devie MUST
> > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > +  buffers back from the input or the output queue of the device. The
> > +  device SHOULD return all of the buffers from the respective queue as
> > +  soon as possible without pushing the buffers through the processing
> > +  pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_queue_clear {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > +  output of a stream.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_plane_format {
> > +        le32 plane_size;
> > +        le32 stride;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{plane_size}] size of the plane in bytes.
> > +\item[\field{stride}] stride used for the plane in bytes.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_params {
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > +        le32 frame_width;
> > +        le32 frame_height;
> > +        le32 min_buffers;
> > +        le32 max_buffers;
> > +        le32 frame_rate;
> > +        struct virtio_video_crop {
> > +                le32 left;
> > +                le32 top;
> > +                le32 width;
> > +                le32 height;
> > +        } crop;
> > +        le32 num_planes;
> > +        struct virtio_video_plane_format
> > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{frame_width}] the value to get/set.
> > +\item[\field{frame_height}] the value to get/set.
> > +\item[\field{pixel_format}] the value to get/set.
> > +\item[\field{min_buffers}] minimum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{max_buffers}] maximum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{frame_rate}] the value to get/set.
> > +\item[\field{crop}] cropping (composing) rectangle.
> > +\item[\field{num_planes}] number of planes used to store pixel data
> > +(r/o).
> > +\item[\field{plane_formats}] description of each plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_get_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +};
> > +
> > +struct virtio_video_get_params_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\item[\field{params}] parameter values.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > +
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_set_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{params}] parameters to set.
> > +\end{description}
> > +
> > +Setting stream parameters might have side effects within the device.
> > +For example, the device MAY perform alignment of width and height,
> > +change the number of planes it uses for the format, or do whatever
> > +changes that are required to continue normal operation using the
> > +updated parameters. It is up to the driver to check the parameter set
> > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > +
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: eventq}
> > +
> > +The device can report events on the event queue. The driver initially
> > +populates the queue with device-writeable buffers. When the device
> > +needs to report an event, it fills a buffer and notifies the driver.
> > +The driver consumes the report and adds a new buffer to the virtqueue.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_event_type {
> > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > +        /* For all functions */
> > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > +        /* For decoder only */
> > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > +};
> > +
> > +struct virtio_video_event {
> > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > +        le32 stream_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{event_type}] type of the triggered event .
> > +\item[\field{stream_id}] id of the source stream.
> > +\end{description}
> > +
> > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > +whenever it encounters new resolution data in the stream. This
> > +includes the case of the initial device configuration after metadata
> > +has been parsed and the case of dynamic resolution change.
>
>
Dmitry Sepp Jan. 6, 2020, 2:59 p.m. UTC | #26
Hi,

a couple of new comments:

On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> 
> The virtio video encoder device and decoder device provide functionalities
> to encode and decode video stream respectively.
> Though video encoder and decoder are provided as different devices, they use
> a same protocol.
> 
> Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> ---
>  content.tex      |   1 +
>  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 580 insertions(+)
>  create mode 100644 virtio-video.tex
> 
> diff --git a/content.tex b/content.tex
> index 556b373..9e56839 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> Requirements}\label{sec:Device \input{virtio-vsock.tex}
>  \input{virtio-fs.tex}
>  \input{virtio-rpmb.tex}
> +\input{virtio-video.tex}
> 
>  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> 
> diff --git a/virtio-video.tex b/virtio-video.tex
> new file mode 100644
> index 0000000..30e728d
> --- /dev/null
> +++ b/virtio-video.tex
> @@ -0,0 +1,579 @@
> +\section{Video Device}\label{sec:Device Types / Video Device}
> +
> +The virtio video encoder device and decoder device are virtual devices that
> +supports encoding and decoding respectively. Though the encoder and the
> decoder +are different devices, they use the same protocol.
> +
> +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> +
> +\begin{description}
> +\item[30] encoder device
> +\item[31] decoder device
> +\end{description}
> +
> +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> +
> +\begin{description}
> +\item[0] controlq - queue for sending control commands.
> +\item[1] eventq - queue for sending events happened in the device.
> +\end{description}
> +
> +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> bits} +
> +\begin{description}
> +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> video +  buffers.
> +\end{description}
> +
> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device
> / Feature bits} +
> +The device MUST offer at least one of feature bits.
> +
> +\subsection{Device configuration layout}\label{sec:Device Types / Video
> Device / Device configuration layout} +
> +Video device configuration uses the following layout structure:
> +
> +\begin{lstlisting}
> +struct virtio_video_config {
> +        le32 max_cap_len;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> +  MUST set this value.
> +\end{description}
> +
> +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> Device Initialization} +
> +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> Video Device / Device Initialization} +
> +The driver SHOULD query device capability by using the
> +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> +setup.
> +
> +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> Device Operation} +
> +The driver allocates input and output buffers and queues the buffers
> +to the device. The device performs operations on the buffers according
> +to the function in question.
> +
> +\subsubsection{Device Operation: Create stream}
> +
> +To process buffers, the device needs to associate them with a certain
> +video stream (essentially, a context). Streams are created by
> +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> +determined by the device.
> +
> +\subsubsection{Device Operation: Create buffers}
> +
> +Buffers are used to store the actual data as well as the relevant
> +metadata. Scatter lists are supported, so the buffer doesn't need to
> +be contiguous in guest physical memory.
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> +  resource that is backed by a buffer allocated from the driver's
> +  memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> +  is no longer needed.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Stream parameter control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> for +  input and output streams from the device.
> +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> +  device.
> +\item After setting stream parameters, the driver may issue
> +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> be +  changed implicitly by the device during the set operation.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Process buffers}
> +
> +\begin{itemize*}
> +\item If the function and the buffer type require so, write data to
> +the buffer memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> +processing in the device.
> +\item The request completes asynchronously when the device has
> +finished with the buffer.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Buffer processing control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> +  return all of the already queued buffers.
> +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> +  already queued buffers from the input or the output queue. This also
> +  includes input or output buffers that can be currently owned by the
> +  device's processing pipeline.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Asynchronous events}
> +
> +While processing buffers, the device can send asynchronous event
> +notifications to the driver. The behaviour depends on the exact
> +stream. For example, the decoder device sends a resolution change
> +event when it encounters new resolution metadata in the stream.
> +
> +\subsubsection{Device Operation: Request header}
> +
> +All requests and responses on the control virt queue have a fixed
> +header using the following layout structure and definitions:
> +
> +\begin{lstlisting}
> +enum virtio_video_ctrl_type {
> +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> +
> +        /* request */
> +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> +        VIRTIO_VIDEO_T_STREAM_CREATE,
> +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> +        VIRTIO_VIDEO_T_SET_PARAMS,
> +        VIRTIO_VIDEO_T_GET_PARAMS,
> +
> +        /* response */
> +        VIRTIO_VIDEO_S_OK = 0x0200,
> +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> +
> +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> +};
> +
> +struct virtio_video_ctrl_hdr {
> +        le32 type;
> +        le32 stream_id;
> +        le32 len; /* Length of the structure in bytes. */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{type}] is the type of the driver request or the device
> +response.
> +\item[\field{stream_id}] specifies a target stream.
> +\item[\field{len}] is the length of data in bytes, which includes
> +length of the header.
> +\end{description}
> +
> +\subsubsection{Device Operation: controlq}
> +
> +\begin{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> +supported formats.
> +
> +The driver uses \field{struct virtio_video_get_capability} to send a
> +query request.
> +
> +\begin{lstlisting}
> +enum virtio_video_buf_type {
> +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> +};
> +
> +struct virtio_video_get_capability {
> +        struct virtio_video_ctrl_hdr hdr;
> +        enum virtio_video_buf_type buf_type;
> +};
> +\end{lstlisting}
> +\begin{description}
> +\item[\field{buf_type}] is the buffer type that the driver asks
> +information about. The driver MUST set either
> +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> +
> +The device responds a capability by using \field{struct
> +virtio_video_get_capability_resp}.
> +\begin{lstlisting}
> +enum virtio_video_format {
> +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> +        /* Raw formats */
> +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> +        VIRTIO_VIDEO_FORMAT_YUV420,
> +        VIRTIO_VIDEO_FORMAT_YVU420,
> +
> +        /* Compressed formats */
> +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> +};
> +
> +enum virtio_video_profile {
> +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> +
> +        /* H.264 */
> +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> VIRTIO_VIDEO_PROFILE_H264_BASELINE, +       
> VIRTIO_VIDEO_PROFILE_H264_MAIN,
> +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> +        /* VP8 */
> +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> +
> +        /* VP9 */
> +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +};
> +
> +struct virtio_video_format_range {
> +        le32 min;
> +        le32 max;
> +        le32 step;
> +        u8 paddings[4];
> +};
> +
> +struct virtio_video_format_desc {
> +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> +        le64 mask;
> +        struct virtio_video_format_range width;
> +        struct virtio_video_format_range height;
> +        le32 num_rates;
> +        u8 padding[4];
> +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> +};
> +
> +struct virtio_video_get_capability_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 num_descs;
> +        /* Followed by struct virtio_video_format_desc desc[] */
> +};
> +\end{lstlisting}
> +
> +The format description \field{struct virtio_video_format_desc}
> +includes the following fields:
> +\begin{description}
> +\item[\field{format}] specifies an image format. The device MUST set one
> +  of \field{enum virtio_video_format}.
> +\item[\field{profile}] specifies a profile of the compressed image format
> +  specified in \field{format}. The driver SHOULD ignore this value if
> +  \field{format} is a raw format.

So how should this be used? The spec does not define any way to set profile for 
the device. It is very important for encoder.

Also, shouldn't the profile come together with level? Would make sense for 
encoders.

> +\item[\field{mask}] is a bitset that represents the supported
> +  combination of input and output format. If \textit{i}-th bit is set
> +  in \field{mask} of \textit{j}-th \field{struct
> +  virtio_video_format_desc} for input, the device supports encoding or
> +  decoding from the \textit{j}-th input format to \textit{i}-th output
> +  format.
> +\item[\field{width, height}] represents a range of resolutions
> +  supported by the device. If its \field{step} is not applicable, its
> +  \field{min} is equal to its \field{max}.
> +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> case of decoder, the driver SHOULD ignore this value.
> +\item[\field{frame_rates}] is an array of supported frame rates.
> +\end{description}
> +

I'd guess frame rates depend on the resolution as well. This dependency was 
clear in the v1 spec, but in the v2 there is no dependency anymore. I think we 
need to update this.

Best regards,
Dmitry.

> +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +enum virtio_video_mem_type {
> +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> +};
> +
> +struct virtio_video_stream_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        char debug_name[64];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{in_mem_type}] is a type of buffer management for input
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{out_mem_type}] is a type of buffer management for output
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{debug_name}] is a text string for a debug purpose.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> +  queued buffers through the pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_drain {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_mem_entry {
> +        le64 addr;
> +        le32 length;
> +        u8 padding[4];
> +};
> +
> +struct virtio_video_resource_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        le32 nr_entries;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{nr_entries}] number of \field{struct
> +  virtio_video_mem_entry} memory entries.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_resource_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> +queue.
> +
> +\begin{lstlisting}
> +#define VIRTIO_VIDEO_MAX_PLANES 8
> +
> +struct virtio_video_resource_queue {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        le32 resource_id;
> +        le64 timestamp;
> +        le32 nr_data_size;
> +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buf_type of the .
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{nr_data_size}] number of \field{data_size} entries.
> +\item[\field{data_size}] number of data bytes within a plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +enum virtio_video_buffer_flag {
> +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> +        /* Encoder only */
> +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> +};
> +
> +struct virtio_video_resource_queue_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le64 timestamp;
> +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> +        le32 size;  /* Encoded size */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{flags}] mark specific buffers in the sequence.
> +\item[\field{size}] data size in the buffer (encoder only).
> +\end{description}
> +
> +The device sends a response to the queue request asynchronously when
> +it has finished processing the buffer.
> +
> +The device SHOULD mark a buffer that triggered a processing error with
> +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> +
> +The device MUST mark the last buffer with the
> +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> +sequence.
> +
> +In case of encoder, to denote a particular frame type the devie MUST
> +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> +  buffers back from the input or the output queue of the device. The
> +  device SHOULD return all of the buffers from the respective queue as
> +  soon as possible without pushing the buffers through the processing
> +  pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_queue_clear {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> +  output of a stream.
> +
> +\begin{lstlisting}
> +struct virtio_video_plane_format {
> +        le32 plane_size;
> +        le32 stride;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{plane_size}] size of the plane in bytes.
> +\item[\field{stride}] stride used for the plane in bytes.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_params {
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> +        le32 frame_width;
> +        le32 frame_height;
> +        le32 min_buffers;
> +        le32 max_buffers;
> +        le32 frame_rate;
> +        struct virtio_video_crop {
> +                le32 left;
> +                le32 top;
> +                le32 width;
> +                le32 height;
> +        } crop;
> +        le32 num_planes;
> +        struct virtio_video_plane_format
> plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{frame_width}] the value to get/set.
> +\item[\field{frame_height}] the value to get/set.
> +\item[\field{pixel_format}] the value to get/set.
> +\item[\field{min_buffers}] minimum buffers required to handle the
> +  format (r/o).
> +\item[\field{max_buffers}] maximum buffers required to handle the
> +  format (r/o).
> +\item[\field{frame_rate}] the value to get/set.
> +\item[\field{crop}] cropping (composing) rectangle.
> +\item[\field{num_planes}] number of planes used to store pixel data
> +(r/o).
> +\item[\field{plane_formats}] description of each plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_get_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +};
> +
> +struct virtio_video_get_params_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\item[\field{params}] parameter values.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> +
> +
> +\begin{lstlisting}
> +struct virtio_video_set_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{params}] parameters to set.
> +\end{description}
> +
> +Setting stream parameters might have side effects within the device.
> +For example, the device MAY perform alignment of width and height,
> +change the number of planes it uses for the format, or do whatever
> +changes that are required to continue normal operation using the
> +updated parameters. It is up to the driver to check the parameter set
> +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> +
> +\end{description}
> +
> +\subsubsection{Device Operation: eventq}
> +
> +The device can report events on the event queue. The driver initially
> +populates the queue with device-writeable buffers. When the device
> +needs to report an event, it fills a buffer and notifies the driver.
> +The driver consumes the report and adds a new buffer to the virtqueue.
> +
> +\begin{lstlisting}
> +enum virtio_video_event_type {
> +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> +        /* For all functions */
> +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> +        /* For decoder only */
> +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> +};
> +
> +struct virtio_video_event {
> +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> +        le32 stream_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{event_type}] type of the triggered event .
> +\item[\field{stream_id}] id of the source stream.
> +\end{description}
> +
> +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> +whenever it encounters new resolution data in the stream. This
> +includes the case of the initial device configuration after metadata
> +has been parsed and the case of dynamic resolution change.
Keiichi Watanabe Jan. 7, 2020, 1:24 p.m. UTC | #27
Hi Dmitry,

On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi,
>
> a couple of new comments:
>
> On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> >
> > The virtio video encoder device and decoder device provide functionalities
> > to encode and decode video stream respectively.
> > Though video encoder and decoder are provided as different devices, they use
> > a same protocol.
> >
> > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > ---
> >  content.tex      |   1 +
> >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 580 insertions(+)
> >  create mode 100644 virtio-video.tex
> >
> > diff --git a/content.tex b/content.tex
> > index 556b373..9e56839 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> >  \input{virtio-fs.tex}
> >  \input{virtio-rpmb.tex}
> > +\input{virtio-video.tex}
> >
> >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >
> > diff --git a/virtio-video.tex b/virtio-video.tex
> > new file mode 100644
> > index 0000000..30e728d
> > --- /dev/null
> > +++ b/virtio-video.tex
> > @@ -0,0 +1,579 @@
> > +\section{Video Device}\label{sec:Device Types / Video Device}
> > +
> > +The virtio video encoder device and decoder device are virtual devices that
> > +supports encoding and decoding respectively. Though the encoder and the
> > decoder +are different devices, they use the same protocol.
> > +
> > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> > +
> > +\begin{description}
> > +\item[30] encoder device
> > +\item[31] decoder device
> > +\end{description}
> > +
> > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> > +
> > +\begin{description}
> > +\item[0] controlq - queue for sending control commands.
> > +\item[1] eventq - queue for sending events happened in the device.
> > +\end{description}
> > +
> > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature
> > bits} +
> > +\begin{description}
> > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for
> > video +  buffers.
> > +\end{description}
> > +
> > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device
> > / Feature bits} +
> > +The device MUST offer at least one of feature bits.
> > +
> > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > Device / Device configuration layout} +
> > +Video device configuration uses the following layout structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_config {
> > +        le32 max_cap_len;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > +  MUST set this value.
> > +\end{description}
> > +
> > +\subsection{Device Initialization}\label{sec:Device Types / Video Device /
> > Device Initialization} +
> > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > Video Device / Device Initialization} +
> > +The driver SHOULD query device capability by using the
> > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > +setup.
> > +
> > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > Device Operation} +
> > +The driver allocates input and output buffers and queues the buffers
> > +to the device. The device performs operations on the buffers according
> > +to the function in question.
> > +
> > +\subsubsection{Device Operation: Create stream}
> > +
> > +To process buffers, the device needs to associate them with a certain
> > +video stream (essentially, a context). Streams are created by
> > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > +determined by the device.
> > +
> > +\subsubsection{Device Operation: Create buffers}
> > +
> > +Buffers are used to store the actual data as well as the relevant
> > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > +be contiguous in guest physical memory.
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > +  resource that is backed by a buffer allocated from the driver's
> > +  memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > +  is no longer needed.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Stream parameter control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters
> > for +  input and output streams from the device.
> > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> > +  device.
> > +\item After setting stream parameters, the driver may issue
> > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can
> > be +  changed implicitly by the device during the set operation.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Process buffers}
> > +
> > +\begin{itemize*}
> > +\item If the function and the buffer type require so, write data to
> > +the buffer memory.
> > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > +processing in the device.
> > +\item The request completes asynchronously when the device has
> > +finished with the buffer.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Buffer processing control}
> > +
> > +\begin{itemize*}
> > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > +  return all of the already queued buffers.
> > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > +  already queued buffers from the input or the output queue. This also
> > +  includes input or output buffers that can be currently owned by the
> > +  device's processing pipeline.
> > +\end{itemize*}
> > +
> > +\subsubsection{Device Operation: Asynchronous events}
> > +
> > +While processing buffers, the device can send asynchronous event
> > +notifications to the driver. The behaviour depends on the exact
> > +stream. For example, the decoder device sends a resolution change
> > +event when it encounters new resolution metadata in the stream.
> > +
> > +\subsubsection{Device Operation: Request header}
> > +
> > +All requests and responses on the control virt queue have a fixed
> > +header using the following layout structure and definitions:
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_ctrl_type {
> > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > +
> > +        /* request */
> > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > +
> > +        /* response */
> > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > +
> > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > +};
> > +
> > +struct virtio_video_ctrl_hdr {
> > +        le32 type;
> > +        le32 stream_id;
> > +        le32 len; /* Length of the structure in bytes. */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{type}] is the type of the driver request or the device
> > +response.
> > +\item[\field{stream_id}] specifies a target stream.
> > +\item[\field{len}] is the length of data in bytes, which includes
> > +length of the header.
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: controlq}
> > +
> > +\begin{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > +supported formats.
> > +
> > +The driver uses \field{struct virtio_video_get_capability} to send a
> > +query request.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buf_type {
> > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > +};
> > +
> > +struct virtio_video_get_capability {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        enum virtio_video_buf_type buf_type;
> > +};
> > +\end{lstlisting}
> > +\begin{description}
> > +\item[\field{buf_type}] is the buffer type that the driver asks
> > +information about. The driver MUST set either
> > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > +
> > +The device responds a capability by using \field{struct
> > +virtio_video_get_capability_resp}.
> > +\begin{lstlisting}
> > +enum virtio_video_format {
> > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > +        /* Raw formats */
> > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > +
> > +        /* Compressed formats */
> > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > +};
> > +
> > +enum virtio_video_profile {
> > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > +
> > +        /* H.264 */
> > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > +        /* VP8 */
> > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > +
> > +        /* VP9 */
> > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > +};
> > +
> > +struct virtio_video_format_range {
> > +        le32 min;
> > +        le32 max;
> > +        le32 step;
> > +        u8 paddings[4];
> > +};
> > +
> > +struct virtio_video_format_desc {
> > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > +        le64 mask;
> > +        struct virtio_video_format_range width;
> > +        struct virtio_video_format_range height;
> > +        le32 num_rates;
> > +        u8 padding[4];
> > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > +};
> > +
> > +struct virtio_video_get_capability_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 num_descs;
> > +        /* Followed by struct virtio_video_format_desc desc[] */
> > +};
> > +\end{lstlisting}
> > +
> > +The format description \field{struct virtio_video_format_desc}
> > +includes the following fields:
> > +\begin{description}
> > +\item[\field{format}] specifies an image format. The device MUST set one
> > +  of \field{enum virtio_video_format}.
> > +\item[\field{profile}] specifies a profile of the compressed image format
> > +  specified in \field{format}. The driver SHOULD ignore this value if
> > +  \field{format} is a raw format.
>
> So how should this be used? The spec does not define any way to set profile for
> the device. It is very important for encoder.
>

Thank you for pointing this.
These points are overlooked, as I didn't care about encoder usage enough.

After thinking it again, I think it's not a very good idea to include
supported profiles and levels in a struct for capability.
This is because these values are available only for limited number of formats.
Also, it's true that we need to have a way to set these values as
Dmitry pointed.

Instead, it would make more sense to have additional three types of
controls for profiles, levels, and bitrates:
(1) QUERY_CONTROL: Query values supported by the device
(2) GET_CONTROL: Read a value that is set in the device
(3) SET_CONTROL: Set a value in the device

These operations are similar to V4L2 controls.
(1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
G_CTRL in V4L2, respectively.
Also, (3) would be similar to enum virtio_video_control_type in the
virtio-video v1 driver implementation in
https://markmail.org/message/dwghwdqsbl3gsjxu .

For QUERY_CONTROL, my idea is like this:

enum virtio_video_control_type {
  VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,

  VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
  VIRTIO_VIDEO_CONTROL_PROFILE,
  VIRTIO_VIDEO_CONTROL_LEVEL,
};

struct virtio_video_query_control {
  struct virtio_video_ctrl_hdr hdr;
  le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
  le32 length;
  /* Followed by additional data.
   * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
   * the device must pass a codec format like H264 or VP9.
   * The requred data must be defined in the specification.
   */
};

struct virtio_video_query_control_resp {
  struct virtio_video_ctrl_hdr hdr;
  le32 length;
  u8 padding[4];
  /* Followed by data corresponds to the specified control.
   * The type of data must be defined in the spec.
   * For example, if the driver queries profiles, this part should be
   * an array of supported profiles of a given format.
   */
};

WDYT?

> Also, shouldn't the profile come together with level? Would make sense for
> encoders.

Yeah. So, in the above idea of QUERY_CONTROL, profile should be
required when querying supported levels.

>
> > +\item[\field{mask}] is a bitset that represents the supported
> > +  combination of input and output format. If \textit{i}-th bit is set
> > +  in \field{mask} of \textit{j}-th \field{struct
> > +  virtio_video_format_desc} for input, the device supports encoding or
> > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > +  format.
> > +\item[\field{width, height}] represents a range of resolutions
> > +  supported by the device. If its \field{step} is not applicable, its
> > +  \field{min} is equal to its \field{max}.
> > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In
> > case of decoder, the driver SHOULD ignore this value.
> > +\item[\field{frame_rates}] is an array of supported frame rates.
> > +\end{description}
> > +
>
> I'd guess frame rates depend on the resolution as well. This dependency was
> clear in the v1 spec, but in the v2 there is no dependency anymore. I think we
> need to update this.

That's a good point. I missed that dependency when updating the structures.
So, let me update the structs like the following:

struct virtio_video_format_frame {
        struct virtio_video_format_range width;
        struct virtio_video_format_range height;
        le32 num_rates;
        u8 padding[4];
        /* Followed by struct virtio_video_format_range frame_rates[] */
};

struct virtio_video_format_desc {
        le64 mask;
        le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
        le32 planes_layout; /* See the thread [v2 0/1] */
        le32 num_frames;
        u8 padding[4];
        /* Followed by struct virtio_video_format_frame frames[] */
};

Best regards,
Keiichi.


>
> Best regards,
> Dmitry.
>
> > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_mem_type {
> > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > +};
> > +
> > +struct virtio_video_stream_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > +        char debug_name[64];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{in_mem_type}] is a type of buffer management for input
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{out_mem_type}] is a type of buffer management for output
> > +buffers. The driver MUST set a value in \field{enum
> > +virtio_video_mem_type}.
> > +\item[\field{debug_name}] is a text string for a debug purpose.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > +  queued buffers through the pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_stream_drain {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +};
> > +\end{lstlisting}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_mem_entry {
> > +        le64 addr;
> > +        le32 length;
> > +        u8 padding[4];
> > +};
> > +
> > +struct virtio_video_resource_create {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        le32 nr_entries;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{nr_entries}] number of \field{struct
> > +  virtio_video_mem_entry} memory entries.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > +  within the device.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_resource_destroy {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 resource_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{resource_id}] internal id of the resource.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > +queue.
> > +
> > +\begin{lstlisting}
> > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > +
> > +struct virtio_video_resource_queue {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        le32 resource_id;
> > +        le64 timestamp;
> > +        le32 nr_data_size;
> > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buf_type of the .
> > +\item[\field{resource_id}] internal id of the resource.
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > +\item[\field{data_size}] number of data bytes within a plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_buffer_flag {
> > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > +        /* Encoder only */
> > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > +};
> > +
> > +struct virtio_video_resource_queue_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le64 timestamp;
> > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > +        le32 size;  /* Encoded size */
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > +  for synchronisation.
> > +\item[\field{flags}] mark specific buffers in the sequence.
> > +\item[\field{size}] data size in the buffer (encoder only).
> > +\end{description}
> > +
> > +The device sends a response to the queue request asynchronously when
> > +it has finished processing the buffer.
> > +
> > +The device SHOULD mark a buffer that triggered a processing error with
> > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > +
> > +The device MUST mark the last buffer with the
> > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > +sequence.
> > +
> > +In case of encoder, to denote a particular frame type the devie MUST
> > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > +
> > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > +  buffers back from the input or the output queue of the device. The
> > +  device SHOULD return all of the buffers from the respective queue as
> > +  soon as possible without pushing the buffers through the processing
> > +  pipeline.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_queue_clear {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > +  output of a stream.
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_plane_format {
> > +        le32 plane_size;
> > +        le32 stride;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{plane_size}] size of the plane in bytes.
> > +\item[\field{stride}] stride used for the plane in bytes.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_params {
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > +        le32 frame_width;
> > +        le32 frame_height;
> > +        le32 min_buffers;
> > +        le32 max_buffers;
> > +        le32 frame_rate;
> > +        struct virtio_video_crop {
> > +                le32 left;
> > +                le32 top;
> > +                le32 width;
> > +                le32 height;
> > +        } crop;
> > +        le32 num_planes;
> > +        struct virtio_video_plane_format
> > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{frame_width}] the value to get/set.
> > +\item[\field{frame_height}] the value to get/set.
> > +\item[\field{pixel_format}] the value to get/set.
> > +\item[\field{min_buffers}] minimum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{max_buffers}] maximum buffers required to handle the
> > +  format (r/o).
> > +\item[\field{frame_rate}] the value to get/set.
> > +\item[\field{crop}] cropping (composing) rectangle.
> > +\item[\field{num_planes}] number of planes used to store pixel data
> > +(r/o).
> > +\item[\field{plane_formats}] description of each plane.
> > +\end{description}
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_get_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > +};
> > +
> > +struct virtio_video_get_params_resp {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{buf_type}] buffer type.
> > +\item[\field{params}] parameter values.
> > +\end{description}
> > +
> > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > +
> > +
> > +\begin{lstlisting}
> > +struct virtio_video_set_params {
> > +        struct virtio_video_ctrl_hdr hdr;
> > +        struct virtio_video_params params;
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{params}] parameters to set.
> > +\end{description}
> > +
> > +Setting stream parameters might have side effects within the device.
> > +For example, the device MAY perform alignment of width and height,
> > +change the number of planes it uses for the format, or do whatever
> > +changes that are required to continue normal operation using the
> > +updated parameters. It is up to the driver to check the parameter set
> > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > +
> > +\end{description}
> > +
> > +\subsubsection{Device Operation: eventq}
> > +
> > +The device can report events on the event queue. The driver initially
> > +populates the queue with device-writeable buffers. When the device
> > +needs to report an event, it fills a buffer and notifies the driver.
> > +The driver consumes the report and adds a new buffer to the virtqueue.
> > +
> > +\begin{lstlisting}
> > +enum virtio_video_event_type {
> > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > +        /* For all functions */
> > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > +        /* For decoder only */
> > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > +};
> > +
> > +struct virtio_video_event {
> > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > +        le32 stream_id;
> > +        u8 padding[4];
> > +};
> > +\end{lstlisting}
> > +
> > +\begin{description}
> > +\item[\field{event_type}] type of the triggered event .
> > +\item[\field{stream_id}] id of the source stream.
> > +\end{description}
> > +
> > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > +whenever it encounters new resolution data in the stream. This
> > +includes the case of the initial device configuration after metadata
> > +has been parsed and the case of dynamic resolution change.
>
>
Dmitry Sepp Jan. 7, 2020, 4:50 p.m. UTC | #28
Hi Keiichi,

thanks for the updates, please see my comments below.

On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> Hi Dmitry,
> 
> On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> 
wrote:
> > Hi,
> > 
> > a couple of new comments:
> > 
> > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > 
> > > The virtio video encoder device and decoder device provide
> > > functionalities
> > > to encode and decode video stream respectively.
> > > Though video encoder and decoder are provided as different devices, they
> > > use a same protocol.
> > > 
> > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > ---
> > > 
> > >  content.tex      |   1 +
> > >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> > >  2 files changed, 580 insertions(+)
> > >  create mode 100644 virtio-video.tex
> > > 
> > > diff --git a/content.tex b/content.tex
> > > index 556b373..9e56839 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > 
> > >  \input{virtio-fs.tex}
> > >  \input{virtio-rpmb.tex}
> > > 
> > > +\input{virtio-video.tex}
> > > 
> > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > 
> > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > new file mode 100644
> > > index 0000000..30e728d
> > > --- /dev/null
> > > +++ b/virtio-video.tex
> > > @@ -0,0 +1,579 @@
> > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > +
> > > +The virtio video encoder device and decoder device are virtual devices
> > > that +supports encoding and decoding respectively. Though the encoder
> > > and the decoder +are different devices, they use the same protocol.
> > > +
> > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device
> > > ID}
> > > +
> > > +\begin{description}
> > > +\item[30] encoder device
> > > +\item[31] decoder device
> > > +\end{description}
> > > +
> > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > Virtqueues} +
> > > +\begin{description}
> > > +\item[0] controlq - queue for sending control commands.
> > > +\item[1] eventq - queue for sending events happened in the device.
> > > +\end{description}
> > > +
> > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > Feature
> > > bits} +
> > > +\begin{description}
> > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used
> > > for
> > > video +  buffers.
> > > +\end{description}
> > > +
> > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > Device
> > > / Feature bits} +
> > > +The device MUST offer at least one of feature bits.
> > > +
> > > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > > Device / Device configuration layout} +
> > > +Video device configuration uses the following layout structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_config {
> > > +        le32 max_cap_len;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > +  MUST set this value.
> > > +\end{description}
> > > +
> > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > Device /
> > > Device Initialization} +
> > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > > Video Device / Device Initialization} +
> > > +The driver SHOULD query device capability by using the
> > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > > +setup.
> > > +
> > > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > > Device Operation} +
> > > +The driver allocates input and output buffers and queues the buffers
> > > +to the device. The device performs operations on the buffers according
> > > +to the function in question.
> > > +
> > > +\subsubsection{Device Operation: Create stream}
> > > +
> > > +To process buffers, the device needs to associate them with a certain
> > > +video stream (essentially, a context). Streams are created by
> > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > +determined by the device.
> > > +
> > > +\subsubsection{Device Operation: Create buffers}
> > > +
> > > +Buffers are used to store the actual data as well as the relevant
> > > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > > +be contiguous in guest physical memory.
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > +  resource that is backed by a buffer allocated from the driver's
> > > +  memory.
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > > +  is no longer needed.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Stream parameter control}
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > parameters
> > > for +  input and output streams from the device.
> > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to
> > > the +  device.
> > > +\item After setting stream parameters, the driver may issue
> > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output
> > > can be +  changed implicitly by the device during the set operation.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Process buffers}
> > > +
> > > +\begin{itemize*}
> > > +\item If the function and the buffer type require so, write data to
> > > +the buffer memory.
> > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > +processing in the device.
> > > +\item The request completes asynchronously when the device has
> > > +finished with the buffer.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Buffer processing control}
> > > +
> > > +\begin{itemize*}
> > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > > +  return all of the already queued buffers.
> > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > > +  already queued buffers from the input or the output queue. This also
> > > +  includes input or output buffers that can be currently owned by the
> > > +  device's processing pipeline.
> > > +\end{itemize*}
> > > +
> > > +\subsubsection{Device Operation: Asynchronous events}
> > > +
> > > +While processing buffers, the device can send asynchronous event
> > > +notifications to the driver. The behaviour depends on the exact
> > > +stream. For example, the decoder device sends a resolution change
> > > +event when it encounters new resolution metadata in the stream.
> > > +
> > > +\subsubsection{Device Operation: Request header}
> > > +
> > > +All requests and responses on the control virt queue have a fixed
> > > +header using the following layout structure and definitions:
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_ctrl_type {
> > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > +
> > > +        /* request */
> > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > +
> > > +        /* response */
> > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > +
> > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > +};
> > > +
> > > +struct virtio_video_ctrl_hdr {
> > > +        le32 type;
> > > +        le32 stream_id;
> > > +        le32 len; /* Length of the structure in bytes. */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{type}] is the type of the driver request or the device
> > > +response.
> > > +\item[\field{stream_id}] specifies a target stream.
> > > +\item[\field{len}] is the length of data in bytes, which includes
> > > +length of the header.
> > > +\end{description}
> > > +
> > > +\subsubsection{Device Operation: controlq}
> > > +
> > > +\begin{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > +supported formats.
> > > +
> > > +The driver uses \field{struct virtio_video_get_capability} to send a
> > > +query request.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_buf_type {
> > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > +};
> > > +
> > > +struct virtio_video_get_capability {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        enum virtio_video_buf_type buf_type;
> > > +};
> > > +\end{lstlisting}
> > > +\begin{description}
> > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > +information about. The driver MUST set either
> > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > +
> > > +The device responds a capability by using \field{struct
> > > +virtio_video_get_capability_resp}.
> > > +\begin{lstlisting}
> > > +enum virtio_video_format {
> > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > +        /* Raw formats */
> > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > +
> > > +        /* Compressed formats */
> > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > +};
> > > +
> > > +enum virtio_video_profile {
> > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > +
> > > +        /* H.264 */
> > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > +        /* VP8 */
> > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > +
> > > +        /* VP9 */
> > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > +};
> > > +
> > > +struct virtio_video_format_range {
> > > +        le32 min;
> > > +        le32 max;
> > > +        le32 step;
> > > +        u8 paddings[4];
> > > +};
> > > +
> > > +struct virtio_video_format_desc {
> > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > +        le64 mask;
> > > +        struct virtio_video_format_range width;
> > > +        struct virtio_video_format_range height;
> > > +        le32 num_rates;
> > > +        u8 padding[4];
> > > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > > +};
> > > +
> > > +struct virtio_video_get_capability_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 num_descs;
> > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +The format description \field{struct virtio_video_format_desc}
> > > +includes the following fields:
> > > +\begin{description}
> > > +\item[\field{format}] specifies an image format. The device MUST set
> > > one
> > > +  of \field{enum virtio_video_format}.
> > > +\item[\field{profile}] specifies a profile of the compressed image
> > > format
> > > +  specified in \field{format}. The driver SHOULD ignore this value if
> > > +  \field{format} is a raw format.
> > 
> > So how should this be used? The spec does not define any way to set
> > profile for the device. It is very important for encoder.
> 
> Thank you for pointing this.
> These points are overlooked, as I didn't care about encoder usage enough.
> 
> After thinking it again, I think it's not a very good idea to include
> supported profiles and levels in a struct for capability.
> This is because these values are available only for limited number of
> formats. Also, it's true that we need to have a way to set these values as
> Dmitry pointed.
> 

Yes, you are right. In fact, the approach of the v1 spec to keep controls 
separately was not correct.

> Instead, it would make more sense to have additional three types of
> controls for profiles, levels, and bitrates:
> (1) QUERY_CONTROL: Query values supported by the device
> (2) GET_CONTROL: Read a value that is set in the device
> (3) SET_CONTROL: Set a value in the device
> 
> These operations are similar to V4L2 controls.
> (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
> G_CTRL in V4L2, respectively.
> Also, (3) would be similar to enum virtio_video_control_type in the
> virtio-video v1 driver implementation in
> https://markmail.org/message/dwghwdqsbl3gsjxu .
> 
> For QUERY_CONTROL, my idea is like this:
> 
> enum virtio_video_control_type {
>   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> 
>   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
>   VIRTIO_VIDEO_CONTROL_PROFILE,
>   VIRTIO_VIDEO_CONTROL_LEVEL,
> };
> 
> struct virtio_video_query_control {
>   struct virtio_video_ctrl_hdr hdr;
>   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
>   le32 length;
>   /* Followed by additional data.
>    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
>    * the device must pass a codec format like H264 or VP9.
>    * The requred data must be defined in the specification.
>    */
> };
> 
> struct virtio_video_query_control_resp {
>   struct virtio_video_ctrl_hdr hdr;
>   le32 length;
>   u8 padding[4];
>   /* Followed by data corresponds to the specified control.
>    * The type of data must be defined in the spec.
>    * For example, if the driver queries profiles, this part should be
>    * an array of supported profiles of a given format.
>    */
> };
> 
> WDYT?
> 

I think virtio_video_control_type should make sense. But I would disagree with 
the need to have new QUERY_CONTROL and GET_CONTROL.

I assume the set of supported controls is fixed for some particular format on a 
given IP. So we'd propose to include controls into format descriptors, so we 
don't need to QUERY_CONTROL. This way (with 'virtio_video_format_list') we can 
define not-contiguos ranges, e.g. for profiles.

struct virtio_video_format_frame {
        /* As proposed in Keiichi's prev email */
};

virtio_video_format_list {
        le32 num_entries;
        u8 padding[4];
        /* Followed by le64 entries[] */
};

struct virtio_video_format_control {
        le32 type;
        u8 padding[4];
        struct virtio_video_format_list values;
};

struct virtio_video_format_desc {
        le64 mask;
        le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
        le32 planes_layout; /* See the thread [v2 0/1] */
        le32 num_frames;
        le32 num_controls;
        /* Followed by struct virtio_video_format_frame frames[] */
        /* Followed by struct virtio_video_format_control controls[] */
};

SET_CONTROL seems to be mandatory. If it succeeds, we can store the current 
value locally, so there is no need to have GET_CONTROL.

The only exception is the initial (default) control value in the device. But 
with the removal of 'function' and with addition of 'caps' instead, the way to 
provide defaults is gone. So I suppose for formats we'll be just using GET on 
driver start to get the 'defaults'. But the thing is that for formats there 
are other uses for GET, but for controls GET apparently does not make to much 
sense at runtime.

> > Also, shouldn't the profile come together with level? Would make sense for
> > encoders.
> 
> Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> required when querying supported levels.
> 

So probably should be enumerated together, not queried, as per the comment 
above.

> > > +\item[\field{mask}] is a bitset that represents the supported
> > > +  combination of input and output format. If \textit{i}-th bit is set
> > > +  in \field{mask} of \textit{j}-th \field{struct
> > > +  virtio_video_format_desc} for input, the device supports encoding or
> > > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > > +  format.
> > > +\item[\field{width, height}] represents a range of resolutions
> > > +  supported by the device. If its \field{step} is not applicable, its
> > > +  \field{min} is equal to its \field{max}.
> > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}.
> > > In
> > > case of decoder, the driver SHOULD ignore this value.
> > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > +\end{description}
> > > +
> > 
> > I'd guess frame rates depend on the resolution as well. This dependency
> > was
> > clear in the v1 spec, but in the v2 there is no dependency anymore. I
> > think we need to update this.
> 
> That's a good point. I missed that dependency when updating the structures.
> So, let me update the structs like the following:
> 
> struct virtio_video_format_frame {
>         struct virtio_video_format_range width;
>         struct virtio_video_format_range height;
>         le32 num_rates;
>         u8 padding[4];
>         /* Followed by struct virtio_video_format_range frame_rates[] */
> };
> 
> struct virtio_video_format_desc {
>         le64 mask;
>         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
>         le32 planes_layout; /* See the thread [v2 0/1] */
>         le32 num_frames;
>         u8 padding[4];
>         /* Followed by struct virtio_video_format_frame frames[] */
> };
>

Yes, I do agree with this approach.

Best regards,
Dmitry.
 
> Best regards,
> Keiichi.
> 
> > Best regards,
> > Dmitry.
> > 
> > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_mem_type {
> > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > +};
> > > +
> > > +struct virtio_video_stream_create {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > +        char debug_name[64];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > +buffers. The driver MUST set a value in \field{enum
> > > +virtio_video_mem_type}.
> > > +\item[\field{out_mem_type}] is a type of buffer management for output
> > > +buffers. The driver MUST set a value in \field{enum
> > > +virtio_video_mem_type}.
> > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_stream_destroy {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > +  queued buffers through the pipeline.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_stream_drain {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_mem_entry {
> > > +        le64 addr;
> > > +        le32 length;
> > > +        u8 padding[4];
> > > +};
> > > +
> > > +struct virtio_video_resource_create {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 resource_id;
> > > +        le32 nr_entries;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\item[\field{nr_entries}] number of \field{struct
> > > +  virtio_video_mem_entry} memory entries.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > > +  within the device.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_resource_destroy {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 resource_id;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > +queue.
> > > +
> > > +\begin{lstlisting}
> > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > +
> > > +struct virtio_video_resource_queue {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type;
> > > +        le32 resource_id;
> > > +        le64 timestamp;
> > > +        le32 nr_data_size;
> > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buf_type of the .
> > > +\item[\field{resource_id}] internal id of the resource.
> > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > +  for synchronisation.
> > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > +\item[\field{data_size}] number of data bytes within a plane.
> > > +\end{description}
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_buffer_flag {
> > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > +        /* Encoder only */
> > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > +};
> > > +
> > > +struct virtio_video_resource_queue_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le64 timestamp;
> > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > +        le32 size;  /* Encoded size */
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > +  for synchronisation.
> > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > +\item[\field{size}] data size in the buffer (encoder only).
> > > +\end{description}
> > > +
> > > +The device sends a response to the queue request asynchronously when
> > > +it has finished processing the buffer.
> > > +
> > > +The device SHOULD mark a buffer that triggered a processing error with
> > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > +
> > > +The device MUST mark the last buffer with the
> > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > +sequence.
> > > +
> > > +In case of encoder, to denote a particular frame type the devie MUST
> > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > +
> > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > +  buffers back from the input or the output queue of the device. The
> > > +  device SHOULD return all of the buffers from the respective queue as
> > > +  soon as possible without pushing the buffers through the processing
> > > +  pipeline.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_queue_clear {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buffer type.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > +  output of a stream.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_plane_format {
> > > +        le32 plane_size;
> > > +        le32 stride;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{plane_size}] size of the plane in bytes.
> > > +\item[\field{stride}] stride used for the plane in bytes.
> > > +\end{description}
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_params {
> > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > +        le32 frame_width;
> > > +        le32 frame_height;
> > > +        le32 min_buffers;
> > > +        le32 max_buffers;
> > > +        le32 frame_rate;
> > > +        struct virtio_video_crop {
> > > +                le32 left;
> > > +                le32 top;
> > > +                le32 width;
> > > +                le32 height;
> > > +        } crop;
> > > +        le32 num_planes;
> > > +        struct virtio_video_plane_format
> > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{frame_width}] the value to get/set.
> > > +\item[\field{frame_height}] the value to get/set.
> > > +\item[\field{pixel_format}] the value to get/set.
> > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > +  format (r/o).
> > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > +  format (r/o).
> > > +\item[\field{frame_rate}] the value to get/set.
> > > +\item[\field{crop}] cropping (composing) rectangle.
> > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > +(r/o).
> > > +\item[\field{plane_formats}] description of each plane.
> > > +\end{description}
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_get_params {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > +};
> > > +
> > > +struct virtio_video_get_params_resp {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        struct virtio_video_params params;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{buf_type}] buffer type.
> > > +\item[\field{params}] parameter values.
> > > +\end{description}
> > > +
> > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > +
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_video_set_params {
> > > +        struct virtio_video_ctrl_hdr hdr;
> > > +        struct virtio_video_params params;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{params}] parameters to set.
> > > +\end{description}
> > > +
> > > +Setting stream parameters might have side effects within the device.
> > > +For example, the device MAY perform alignment of width and height,
> > > +change the number of planes it uses for the format, or do whatever
> > > +changes that are required to continue normal operation using the
> > > +updated parameters. It is up to the driver to check the parameter set
> > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > +
> > > +\end{description}
> > > +
> > > +\subsubsection{Device Operation: eventq}
> > > +
> > > +The device can report events on the event queue. The driver initially
> > > +populates the queue with device-writeable buffers. When the device
> > > +needs to report an event, it fills a buffer and notifies the driver.
> > > +The driver consumes the report and adds a new buffer to the virtqueue.
> > > +
> > > +\begin{lstlisting}
> > > +enum virtio_video_event_type {
> > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > +        /* For all functions */
> > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > +        /* For decoder only */
> > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > +};
> > > +
> > > +struct virtio_video_event {
> > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > +        le32 stream_id;
> > > +        u8 padding[4];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\begin{description}
> > > +\item[\field{event_type}] type of the triggered event .
> > > +\item[\field{stream_id}] id of the source stream.
> > > +\end{description}
> > > +
> > > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > +whenever it encounters new resolution data in the stream. This
> > > +includes the case of the initial device configuration after metadata
> > > +has been parsed and the case of dynamic resolution change.
Keiichi Watanabe Jan. 8, 2020, 6:59 a.m. UTC | #29
Hi Dmitry,

On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi Keiichi,
>
> thanks for the updates, please see my comments below.
>
> On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> > Hi Dmitry,
> >
> > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> wrote:
> > > Hi,
> > >
> > > a couple of new comments:
> > >
> > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > >
> > > > The virtio video encoder device and decoder device provide
> > > > functionalities
> > > > to encode and decode video stream respectively.
> > > > Though video encoder and decoder are provided as different devices, they
> > > > use a same protocol.
> > > >
> > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > > ---
> > > >
> > > >  content.tex      |   1 +
> > > >  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
> > > >  2 files changed, 580 insertions(+)
> > > >  create mode 100644 virtio-video.tex
> > > >
> > > > diff --git a/content.tex b/content.tex
> > > > index 556b373..9e56839 100644
> > > > --- a/content.tex
> > > > +++ b/content.tex
> > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > >
> > > >  \input{virtio-fs.tex}
> > > >  \input{virtio-rpmb.tex}
> > > >
> > > > +\input{virtio-video.tex}
> > > >
> > > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > >
> > > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > > new file mode 100644
> > > > index 0000000..30e728d
> > > > --- /dev/null
> > > > +++ b/virtio-video.tex
> > > > @@ -0,0 +1,579 @@
> > > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > > +
> > > > +The virtio video encoder device and decoder device are virtual devices
> > > > that +supports encoding and decoding respectively. Though the encoder
> > > > and the decoder +are different devices, they use the same protocol.
> > > > +
> > > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device
> > > > ID}
> > > > +
> > > > +\begin{description}
> > > > +\item[30] encoder device
> > > > +\item[31] decoder device
> > > > +\end{description}
> > > > +
> > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > > Virtqueues} +
> > > > +\begin{description}
> > > > +\item[0] controlq - queue for sending control commands.
> > > > +\item[1] eventq - queue for sending events happened in the device.
> > > > +\end{description}
> > > > +
> > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > > Feature
> > > > bits} +
> > > > +\begin{description}
> > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used
> > > > for
> > > > video +  buffers.
> > > > +\end{description}
> > > > +
> > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > > Device
> > > > / Feature bits} +
> > > > +The device MUST offer at least one of feature bits.
> > > > +
> > > > +\subsection{Device configuration layout}\label{sec:Device Types / Video
> > > > Device / Device configuration layout} +
> > > > +Video device configuration uses the following layout structure:
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_config {
> > > > +        le32 max_cap_len;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> > > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > > +  MUST set this value.
> > > > +\end{description}
> > > > +
> > > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > > Device /
> > > > Device Initialization} +
> > > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types /
> > > > Video Device / Device Initialization} +
> > > > +The driver SHOULD query device capability by using the
> > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> > > > +setup.
> > > > +
> > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device /
> > > > Device Operation} +
> > > > +The driver allocates input and output buffers and queues the buffers
> > > > +to the device. The device performs operations on the buffers according
> > > > +to the function in question.
> > > > +
> > > > +\subsubsection{Device Operation: Create stream}
> > > > +
> > > > +To process buffers, the device needs to associate them with a certain
> > > > +video stream (essentially, a context). Streams are created by
> > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > > +determined by the device.
> > > > +
> > > > +\subsubsection{Device Operation: Create buffers}
> > > > +
> > > > +Buffers are used to store the actual data as well as the relevant
> > > > +metadata. Scatter lists are supported, so the buffer doesn't need to
> > > > +be contiguous in guest physical memory.
> > > > +
> > > > +\begin{itemize*}
> > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > > +  resource that is backed by a buffer allocated from the driver's
> > > > +  memory.
> > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> > > > +  is no longer needed.
> > > > +\end{itemize*}
> > > > +
> > > > +\subsubsection{Device Operation: Stream parameter control}
> > > > +
> > > > +\begin{itemize*}
> > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > > parameters
> > > > for +  input and output streams from the device.
> > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to
> > > > the +  device.
> > > > +\item After setting stream parameters, the driver may issue
> > > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output
> > > > can be +  changed implicitly by the device during the set operation.
> > > > +\end{itemize*}
> > > > +
> > > > +\subsubsection{Device Operation: Process buffers}
> > > > +
> > > > +\begin{itemize*}
> > > > +\item If the function and the buffer type require so, write data to
> > > > +the buffer memory.
> > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > > +processing in the device.
> > > > +\item The request completes asynchronously when the device has
> > > > +finished with the buffer.
> > > > +\end{itemize*}
> > > > +
> > > > +\subsubsection{Device Operation: Buffer processing control}
> > > > +
> > > > +\begin{itemize*}
> > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> > > > +  return all of the already queued buffers.
> > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> > > > +  already queued buffers from the input or the output queue. This also
> > > > +  includes input or output buffers that can be currently owned by the
> > > > +  device's processing pipeline.
> > > > +\end{itemize*}
> > > > +
> > > > +\subsubsection{Device Operation: Asynchronous events}
> > > > +
> > > > +While processing buffers, the device can send asynchronous event
> > > > +notifications to the driver. The behaviour depends on the exact
> > > > +stream. For example, the decoder device sends a resolution change
> > > > +event when it encounters new resolution metadata in the stream.
> > > > +
> > > > +\subsubsection{Device Operation: Request header}
> > > > +
> > > > +All requests and responses on the control virt queue have a fixed
> > > > +header using the following layout structure and definitions:
> > > > +
> > > > +\begin{lstlisting}
> > > > +enum virtio_video_ctrl_type {
> > > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > > +
> > > > +        /* request */
> > > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > > +
> > > > +        /* response */
> > > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > > +
> > > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > > +};
> > > > +
> > > > +struct virtio_video_ctrl_hdr {
> > > > +        le32 type;
> > > > +        le32 stream_id;
> > > > +        le32 len; /* Length of the structure in bytes. */
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{type}] is the type of the driver request or the device
> > > > +response.
> > > > +\item[\field{stream_id}] specifies a target stream.
> > > > +\item[\field{len}] is the length of data in bytes, which includes
> > > > +length of the header.
> > > > +\end{description}
> > > > +
> > > > +\subsubsection{Device Operation: controlq}
> > > > +
> > > > +\begin{description}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > > +supported formats.
> > > > +
> > > > +The driver uses \field{struct virtio_video_get_capability} to send a
> > > > +query request.
> > > > +
> > > > +\begin{lstlisting}
> > > > +enum virtio_video_buf_type {
> > > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > > +};
> > > > +
> > > > +struct virtio_video_get_capability {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        enum virtio_video_buf_type buf_type;
> > > > +};
> > > > +\end{lstlisting}
> > > > +\begin{description}
> > > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > > +information about. The driver MUST set either
> > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > > +
> > > > +The device responds a capability by using \field{struct
> > > > +virtio_video_get_capability_resp}.
> > > > +\begin{lstlisting}
> > > > +enum virtio_video_format {
> > > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > +        /* Raw formats */
> > > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > > +
> > > > +        /* Compressed formats */
> > > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > > +};
> > > > +
> > > > +enum virtio_video_profile {
> > > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > > +
> > > > +        /* H.264 */
> > > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > > +        /* VP8 */
> > > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > > +
> > > > +        /* VP9 */
> > > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > +};
> > > > +
> > > > +struct virtio_video_format_range {
> > > > +        le32 min;
> > > > +        le32 max;
> > > > +        le32 step;
> > > > +        u8 paddings[4];
> > > > +};
> > > > +
> > > > +struct virtio_video_format_desc {
> > > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > > +        le64 mask;
> > > > +        struct virtio_video_format_range width;
> > > > +        struct virtio_video_format_range height;
> > > > +        le32 num_rates;
> > > > +        u8 padding[4];
> > > > +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> > > > +};
> > > > +
> > > > +struct virtio_video_get_capability_resp {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 num_descs;
> > > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +The format description \field{struct virtio_video_format_desc}
> > > > +includes the following fields:
> > > > +\begin{description}
> > > > +\item[\field{format}] specifies an image format. The device MUST set
> > > > one
> > > > +  of \field{enum virtio_video_format}.
> > > > +\item[\field{profile}] specifies a profile of the compressed image
> > > > format
> > > > +  specified in \field{format}. The driver SHOULD ignore this value if
> > > > +  \field{format} is a raw format.
> > >
> > > So how should this be used? The spec does not define any way to set
> > > profile for the device. It is very important for encoder.
> >
> > Thank you for pointing this.
> > These points are overlooked, as I didn't care about encoder usage enough.
> >
> > After thinking it again, I think it's not a very good idea to include
> > supported profiles and levels in a struct for capability.
> > This is because these values are available only for limited number of
> > formats. Also, it's true that we need to have a way to set these values as
> > Dmitry pointed.
> >
>
> Yes, you are right. In fact, the approach of the v1 spec to keep controls
> separately was not correct.
>
> > Instead, it would make more sense to have additional three types of
> > controls for profiles, levels, and bitrates:
> > (1) QUERY_CONTROL: Query values supported by the device
> > (2) GET_CONTROL: Read a value that is set in the device
> > (3) SET_CONTROL: Set a value in the device
> >
> > These operations are similar to V4L2 controls.
> > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
> > G_CTRL in V4L2, respectively.
> > Also, (3) would be similar to enum virtio_video_control_type in the
> > virtio-video v1 driver implementation in
> > https://markmail.org/message/dwghwdqsbl3gsjxu .
> >
> > For QUERY_CONTROL, my idea is like this:
> >
> > enum virtio_video_control_type {
> >   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> >
> >   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
> >   VIRTIO_VIDEO_CONTROL_PROFILE,
> >   VIRTIO_VIDEO_CONTROL_LEVEL,
> > };
> >
> > struct virtio_video_query_control {
> >   struct virtio_video_ctrl_hdr hdr;
> >   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> >   le32 length;
> >   /* Followed by additional data.
> >    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
> >    * the device must pass a codec format like H264 or VP9.
> >    * The requred data must be defined in the specification.
> >    */
> > };
> >
> > struct virtio_video_query_control_resp {
> >   struct virtio_video_ctrl_hdr hdr;
> >   le32 length;
> >   u8 padding[4];
> >   /* Followed by data corresponds to the specified control.
> >    * The type of data must be defined in the spec.
> >    * For example, if the driver queries profiles, this part should be
> >    * an array of supported profiles of a given format.
> >    */
> > };
> >
> > WDYT?
> >
>
> I think virtio_video_control_type should make sense. But I would disagree with
> the need to have new QUERY_CONTROL and GET_CONTROL.
>
> I assume the set of supported controls is fixed for some particular format on a
> given IP. So we'd propose to include controls into format descriptors, so we
> don't need to QUERY_CONTROL. This way (with 'virtio_video_format_list') we can
> define not-contiguos ranges, e.g. for profiles.
>
> struct virtio_video_format_frame {
>         /* As proposed in Keiichi's prev email */
> };
>
> virtio_video_format_list {
>         le32 num_entries;
>         u8 padding[4];
>         /* Followed by le64 entries[] */
> };
>
> struct virtio_video_format_control {
>         le32 type;
>         u8 padding[4];
>         struct virtio_video_format_list values;
> };
>
> struct virtio_video_format_desc {
>         le64 mask;
>         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
>         le32 planes_layout; /* See the thread [v2 0/1] */
>         le32 num_frames;
>         le32 num_controls;
>         /* Followed by struct virtio_video_format_frame frames[] */
>         /* Followed by struct virtio_video_format_control controls[] */
> };

In my understanding, a set of supported levels depends on a profile.
H.264 spec says that levels are specified within each profile.
cf. "0.5 Profiles and levels" in https://www.itu.int/rec/T-REC-H.264-201906-I/en
(V4L2's QUERYMENU doesn't seem to provide a way to query levels for
each profile properly, though)

So, if we want to have supported profiles and levels in format_desc as
your idea, each supported profile should have a list of supported
levels.
I suppose it makes the structure of video_format_desc too complicated.
I'd like to avoid this complexity caused by some specific formats.

Instead, I'd like to keep virtio_video_format_desc minimal and have
QUERY_CONTROL to query format-specific values like profiles and
levels.
This design would make it easy to extend controls when we want to
support new types of formats. We will just need to define new
VIRTIO_VIDEO_CONTROL_*.
What do you think?

Best regards,
Keiichi

>
> SET_CONTROL seems to be mandatory. If it succeeds, we can store the current
> value locally, so there is no need to have GET_CONTROL.
>
> The only exception is the initial (default) control value in the device. But
> with the removal of 'function' and with addition of 'caps' instead, the way to
> provide defaults is gone. So I suppose for formats we'll be just using GET on
> driver start to get the 'defaults'. But the thing is that for formats there
> are other uses for GET, but for controls GET apparently does not make to much
> sense at runtime.
>
> > > Also, shouldn't the profile come together with level? Would make sense for
> > > encoders.
> >
> > Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> > required when querying supported levels.
> >
>
> So probably should be enumerated together, not queried, as per the comment
> above.
>
> > > > +\item[\field{mask}] is a bitset that represents the supported
> > > > +  combination of input and output format. If \textit{i}-th bit is set
> > > > +  in \field{mask} of \textit{j}-th \field{struct
> > > > +  virtio_video_format_desc} for input, the device supports encoding or
> > > > +  decoding from the \textit{j}-th input format to \textit{i}-th output
> > > > +  format.
> > > > +\item[\field{width, height}] represents a range of resolutions
> > > > +  supported by the device. If its \field{step} is not applicable, its
> > > > +  \field{min} is equal to its \field{max}.
> > > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}.
> > > > In
> > > > case of decoder, the driver SHOULD ignore this value.
> > > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > > +\end{description}
> > > > +
> > >
> > > I'd guess frame rates depend on the resolution as well. This dependency
> > > was
> > > clear in the v1 spec, but in the v2 there is no dependency anymore. I
> > > think we need to update this.
> >
> > That's a good point. I missed that dependency when updating the structures.
> > So, let me update the structs like the following:
> >
> > struct virtio_video_format_frame {
> >         struct virtio_video_format_range width;
> >         struct virtio_video_format_range height;
> >         le32 num_rates;
> >         u8 padding[4];
> >         /* Followed by struct virtio_video_format_range frame_rates[] */
> > };
> >
> > struct virtio_video_format_desc {
> >         le64 mask;
> >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> >         le32 planes_layout; /* See the thread [v2 0/1] */
> >         le32 num_frames;
> >         u8 padding[4];
> >         /* Followed by struct virtio_video_format_frame frames[] */
> > };
> >
>
> Yes, I do agree with this approach.
>
> Best regards,
> Dmitry.
>
> > Best regards,
> > Keiichi.
> >
> > > Best regards,
> > > Dmitry.
> > >
> > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > > +  within the device.
> > > > +
> > > > +\begin{lstlisting}
> > > > +enum virtio_video_mem_type {
> > > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > > +};
> > > > +
> > > > +struct virtio_video_stream_create {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> > > > +        char debug_name[64];
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > > +buffers. The driver MUST set a value in \field{enum
> > > > +virtio_video_mem_type}.
> > > > +\item[\field{out_mem_type}] is a type of buffer management for output
> > > > +buffers. The driver MUST set a value in \field{enum
> > > > +virtio_video_mem_type}.
> > > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > > +\end{description}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> > > > +  within the device.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_stream_destroy {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > > +  queued buffers through the pipeline.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_stream_drain {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > > +  within the device.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_mem_entry {
> > > > +        le64 addr;
> > > > +        le32 length;
> > > > +        u8 padding[4];
> > > > +};
> > > > +
> > > > +struct virtio_video_resource_create {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 resource_id;
> > > > +        le32 nr_entries;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{resource_id}] internal id of the resource.
> > > > +\item[\field{nr_entries}] number of \field{struct
> > > > +  virtio_video_mem_entry} memory entries.
> > > > +\end{description}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> > > > +  within the device.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_resource_destroy {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 resource_id;
> > > > +        u8 padding[4];
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{resource_id}] internal id of the resource.
> > > > +\end{description}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > > +queue.
> > > > +
> > > > +\begin{lstlisting}
> > > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > > +
> > > > +struct virtio_video_resource_queue {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 buf_type;
> > > > +        le32 resource_id;
> > > > +        le64 timestamp;
> > > > +        le32 nr_data_size;
> > > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{buf_type}] buf_type of the .
> > > > +\item[\field{resource_id}] internal id of the resource.
> > > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > > +  for synchronisation.
> > > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > > +\item[\field{data_size}] number of data bytes within a plane.
> > > > +\end{description}
> > > > +
> > > > +\begin{lstlisting}
> > > > +enum virtio_video_buffer_flag {
> > > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > > +        /* Encoder only */
> > > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > > +};
> > > > +
> > > > +struct virtio_video_resource_queue_resp {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le64 timestamp;
> > > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > > +        le32 size;  /* Encoded size */
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{timestamp}] an abstract sequence counter that can be used
> > > > +  for synchronisation.
> > > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > > +\item[\field{size}] data size in the buffer (encoder only).
> > > > +\end{description}
> > > > +
> > > > +The device sends a response to the queue request asynchronously when
> > > > +it has finished processing the buffer.
> > > > +
> > > > +The device SHOULD mark a buffer that triggered a processing error with
> > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > > +
> > > > +The device MUST mark the last buffer with the
> > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > +sequence.
> > > > +
> > > > +In case of encoder, to denote a particular frame type the devie MUST
> > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > > +  buffers back from the input or the output queue of the device. The
> > > > +  device SHOULD return all of the buffers from the respective queue as
> > > > +  soon as possible without pushing the buffers through the processing
> > > > +  pipeline.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_queue_clear {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 buf_type;
> > > > +        u8 padding[4];
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{buf_type}] buffer type.
> > > > +\end{description}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > > +  output of a stream.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_plane_format {
> > > > +        le32 plane_size;
> > > > +        le32 stride;
> > > > +        u8 padding[4];
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{plane_size}] size of the plane in bytes.
> > > > +\item[\field{stride}] stride used for the plane in bytes.
> > > > +\end{description}
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_params {
> > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > > +        le32 frame_width;
> > > > +        le32 frame_height;
> > > > +        le32 min_buffers;
> > > > +        le32 max_buffers;
> > > > +        le32 frame_rate;
> > > > +        struct virtio_video_crop {
> > > > +                le32 left;
> > > > +                le32 top;
> > > > +                le32 width;
> > > > +                le32 height;
> > > > +        } crop;
> > > > +        le32 num_planes;
> > > > +        struct virtio_video_plane_format
> > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{frame_width}] the value to get/set.
> > > > +\item[\field{frame_height}] the value to get/set.
> > > > +\item[\field{pixel_format}] the value to get/set.
> > > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > > +  format (r/o).
> > > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > > +  format (r/o).
> > > > +\item[\field{frame_rate}] the value to get/set.
> > > > +\item[\field{crop}] cropping (composing) rectangle.
> > > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > > +(r/o).
> > > > +\item[\field{plane_formats}] description of each plane.
> > > > +\end{description}
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_get_params {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > +};
> > > > +
> > > > +struct virtio_video_get_params_resp {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        struct virtio_video_params params;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{buf_type}] buffer type.
> > > > +\item[\field{params}] parameter values.
> > > > +\end{description}
> > > > +
> > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > > +
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_video_set_params {
> > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > +        struct virtio_video_params params;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{params}] parameters to set.
> > > > +\end{description}
> > > > +
> > > > +Setting stream parameters might have side effects within the device.
> > > > +For example, the device MAY perform alignment of width and height,
> > > > +change the number of planes it uses for the format, or do whatever
> > > > +changes that are required to continue normal operation using the
> > > > +updated parameters. It is up to the driver to check the parameter set
> > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > > +
> > > > +\end{description}
> > > > +
> > > > +\subsubsection{Device Operation: eventq}
> > > > +
> > > > +The device can report events on the event queue. The driver initially
> > > > +populates the queue with device-writeable buffers. When the device
> > > > +needs to report an event, it fills a buffer and notifies the driver.
> > > > +The driver consumes the report and adds a new buffer to the virtqueue.
> > > > +
> > > > +\begin{lstlisting}
> > > > +enum virtio_video_event_type {
> > > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > > +        /* For all functions */
> > > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > > +        /* For decoder only */
> > > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > > +};
> > > > +
> > > > +struct virtio_video_event {
> > > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > > +        le32 stream_id;
> > > > +        u8 padding[4];
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +\begin{description}
> > > > +\item[\field{event_type}] type of the triggered event .
> > > > +\item[\field{stream_id}] id of the source stream.
> > > > +\end{description}
> > > > +
> > > > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > > +whenever it encounters new resolution data in the stream. This
> > > > +includes the case of the initial device configuration after metadata
> > > > +has been parsed and the case of dynamic resolution change.
>
>
Dmitry Sepp Jan. 8, 2020, 10 a.m. UTC | #30
Hi Keiichi,

On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote:
> Hi Dmitry,
> 
> On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com> 
wrote:
> > Hi Keiichi,
> > 
> > thanks for the updates, please see my comments below.
> > 
> > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> > > Hi Dmitry,
> > > 
> > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp
> > > <dmitry.sepp@opensynergy.com>
> > 
> > wrote:
> > > > Hi,
> > > > 
> > > > a couple of new comments:
> > > > 
> > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > 
> > > > > The virtio video encoder device and decoder device provide
> > > > > functionalities
> > > > > to encode and decode video stream respectively.
> > > > > Though video encoder and decoder are provided as different devices,
> > > > > they
> > > > > use a same protocol.
> > > > > 
> > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > > > ---
> > > > > 
> > > > >  content.tex      |   1 +
> > > > >  virtio-video.tex | 579
> > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > >  2 files changed, 580 insertions(+)
> > > > >  create mode 100644 virtio-video.tex
> > > > > 
> > > > > diff --git a/content.tex b/content.tex
> > > > > index 556b373..9e56839 100644
> > > > > --- a/content.tex
> > > > > +++ b/content.tex
> > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > > > 
> > > > >  \input{virtio-fs.tex}
> > > > >  \input{virtio-rpmb.tex}
> > > > > 
> > > > > +\input{virtio-video.tex}
> > > > > 
> > > > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > > > 
> > > > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > > > new file mode 100644
> > > > > index 0000000..30e728d
> > > > > --- /dev/null
> > > > > +++ b/virtio-video.tex
> > > > > @@ -0,0 +1,579 @@
> > > > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > > > +
> > > > > +The virtio video encoder device and decoder device are virtual
> > > > > devices
> > > > > that +supports encoding and decoding respectively. Though the
> > > > > encoder
> > > > > and the decoder +are different devices, they use the same protocol.
> > > > > +
> > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device /
> > > > > Device
> > > > > ID}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[30] encoder device
> > > > > +\item[31] decoder device
> > > > > +\end{description}
> > > > > +
> > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > > > Virtqueues} +
> > > > > +\begin{description}
> > > > > +\item[0] controlq - queue for sending control commands.
> > > > > +\item[1] eventq - queue for sending events happened in the device.
> > > > > +\end{description}
> > > > > +
> > > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > > > Feature
> > > > > bits} +
> > > > > +\begin{description}
> > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be
> > > > > used
> > > > > for
> > > > > video +  buffers.
> > > > > +\end{description}
> > > > > +
> > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > > > Device
> > > > > / Feature bits} +
> > > > > +The device MUST offer at least one of feature bits.
> > > > > +
> > > > > +\subsection{Device configuration layout}\label{sec:Device Types /
> > > > > Video
> > > > > Device / Device configuration layout} +
> > > > > +Video device configuration uses the following layout structure:
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_config {
> > > > > +        le32 max_cap_len;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{max_cap_len}] defines the maximum length of a
> > > > > descriptor
> > > > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > > > +  MUST set this value.
> > > > > +\end{description}
> > > > > +
> > > > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > > > Device /
> > > > > Device Initialization} +
> > > > > +\devicenormative{\subsubsection}{Device Initialization}{Device
> > > > > Types /
> > > > > Video Device / Device Initialization} +
> > > > > +The driver SHOULD query device capability by using the
> > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the
> > > > > initial
> > > > > +setup.
> > > > > +
> > > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device
> > > > > /
> > > > > Device Operation} +
> > > > > +The driver allocates input and output buffers and queues the
> > > > > buffers
> > > > > +to the device. The device performs operations on the buffers
> > > > > according
> > > > > +to the function in question.
> > > > > +
> > > > > +\subsubsection{Device Operation: Create stream}
> > > > > +
> > > > > +To process buffers, the device needs to associate them with a
> > > > > certain
> > > > > +video stream (essentially, a context). Streams are created by
> > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > > > +determined by the device.
> > > > > +
> > > > > +\subsubsection{Device Operation: Create buffers}
> > > > > +
> > > > > +Buffers are used to store the actual data as well as the relevant
> > > > > +metadata. Scatter lists are supported, so the buffer doesn't need
> > > > > to
> > > > > +be contiguous in guest physical memory.
> > > > > +
> > > > > +\begin{itemize*}
> > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > > > +  resource that is backed by a buffer allocated from the driver's
> > > > > +  memory.
> > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource
> > > > > that
> > > > > +  is no longer needed.
> > > > > +\end{itemize*}
> > > > > +
> > > > > +\subsubsection{Device Operation: Stream parameter control}
> > > > > +
> > > > > +\begin{itemize*}
> > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > > > parameters
> > > > > for +  input and output streams from the device.
> > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream
> > > > > parameters to
> > > > > the +  device.
> > > > > +\item After setting stream parameters, the driver may issue
> > > > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and
> > > > > output
> > > > > can be +  changed implicitly by the device during the set operation.
> > > > > +\end{itemize*}
> > > > > +
> > > > > +\subsubsection{Device Operation: Process buffers}
> > > > > +
> > > > > +\begin{itemize*}
> > > > > +\item If the function and the buffer type require so, write data to
> > > > > +the buffer memory.
> > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > > > +processing in the device.
> > > > > +\item The request completes asynchronously when the device has
> > > > > +finished with the buffer.
> > > > > +\end{itemize*}
> > > > > +
> > > > > +\subsubsection{Device Operation: Buffer processing control}
> > > > > +
> > > > > +\begin{itemize*}
> > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process
> > > > > and
> > > > > +  return all of the already queued buffers.
> > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return
> > > > > back
> > > > > +  already queued buffers from the input or the output queue. This
> > > > > also
> > > > > +  includes input or output buffers that can be currently owned by
> > > > > the
> > > > > +  device's processing pipeline.
> > > > > +\end{itemize*}
> > > > > +
> > > > > +\subsubsection{Device Operation: Asynchronous events}
> > > > > +
> > > > > +While processing buffers, the device can send asynchronous event
> > > > > +notifications to the driver. The behaviour depends on the exact
> > > > > +stream. For example, the decoder device sends a resolution change
> > > > > +event when it encounters new resolution metadata in the stream.
> > > > > +
> > > > > +\subsubsection{Device Operation: Request header}
> > > > > +
> > > > > +All requests and responses on the control virt queue have a fixed
> > > > > +header using the following layout structure and definitions:
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +enum virtio_video_ctrl_type {
> > > > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > > > +
> > > > > +        /* request */
> > > > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > > > +
> > > > > +        /* response */
> > > > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > > > +
> > > > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_ctrl_hdr {
> > > > > +        le32 type;
> > > > > +        le32 stream_id;
> > > > > +        le32 len; /* Length of the structure in bytes. */
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{type}] is the type of the driver request or the device
> > > > > +response.
> > > > > +\item[\field{stream_id}] specifies a target stream.
> > > > > +\item[\field{len}] is the length of data in bytes, which includes
> > > > > +length of the header.
> > > > > +\end{description}
> > > > > +
> > > > > +\subsubsection{Device Operation: controlq}
> > > > > +
> > > > > +\begin{description}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > > > +supported formats.
> > > > > +
> > > > > +The driver uses \field{struct virtio_video_get_capability} to send
> > > > > a
> > > > > +query request.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +enum virtio_video_buf_type {
> > > > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_get_capability {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        enum virtio_video_buf_type buf_type;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +\begin{description}
> > > > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > > > +information about. The driver MUST set either
> > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > > > +
> > > > > +The device responds a capability by using \field{struct
> > > > > +virtio_video_get_capability_resp}.
> > > > > +\begin{lstlisting}
> > > > > +enum virtio_video_format {
> > > > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > > +        /* Raw formats */
> > > > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > > > +
> > > > > +        /* Compressed formats */
> > > > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > > > +};
> > > > > +
> > > > > +enum virtio_video_profile {
> > > > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > > > +
> > > > > +        /* H.264 */
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > > > +        /* VP8 */
> > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY =
> > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX =
> > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > > > +
> > > > > +        /* VP9 */
> > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_format_range {
> > > > > +        le32 min;
> > > > > +        le32 max;
> > > > > +        le32 step;
> > > > > +        u8 paddings[4];
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_format_desc {
> > > > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > > > +        le64 mask;
> > > > > +        struct virtio_video_format_range width;
> > > > > +        struct virtio_video_format_range height;
> > > > > +        le32 num_rates;
> > > > > +        u8 padding[4];
> > > > > +        /* Followed by struct virtio_video_frame_rate frame_rates[]
> > > > > */
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_get_capability_resp {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 num_descs;
> > > > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +The format description \field{struct virtio_video_format_desc}
> > > > > +includes the following fields:
> > > > > +\begin{description}
> > > > > +\item[\field{format}] specifies an image format. The device MUST
> > > > > set
> > > > > one
> > > > > +  of \field{enum virtio_video_format}.
> > > > > +\item[\field{profile}] specifies a profile of the compressed image
> > > > > format
> > > > > +  specified in \field{format}. The driver SHOULD ignore this value
> > > > > if
> > > > > +  \field{format} is a raw format.
> > > > 
> > > > So how should this be used? The spec does not define any way to set
> > > > profile for the device. It is very important for encoder.
> > > 
> > > Thank you for pointing this.
> > > These points are overlooked, as I didn't care about encoder usage
> > > enough.
> > > 
> > > After thinking it again, I think it's not a very good idea to include
> > > supported profiles and levels in a struct for capability.
> > > This is because these values are available only for limited number of
> > > formats. Also, it's true that we need to have a way to set these values
> > > as
> > > Dmitry pointed.
> > 
> > Yes, you are right. In fact, the approach of the v1 spec to keep controls
> > separately was not correct.
> > 
> > > Instead, it would make more sense to have additional three types of
> > > controls for profiles, levels, and bitrates:
> > > (1) QUERY_CONTROL: Query values supported by the device
> > > (2) GET_CONTROL: Read a value that is set in the device
> > > (3) SET_CONTROL: Set a value in the device
> > > 
> > > These operations are similar to V4L2 controls.
> > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
> > > G_CTRL in V4L2, respectively.
> > > Also, (3) would be similar to enum virtio_video_control_type in the
> > > virtio-video v1 driver implementation in
> > > https://markmail.org/message/dwghwdqsbl3gsjxu .
> > > 
> > > For QUERY_CONTROL, my idea is like this:
> > > 
> > > enum virtio_video_control_type {
> > > 
> > >   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> > >   
> > >   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
> > >   VIRTIO_VIDEO_CONTROL_PROFILE,
> > >   VIRTIO_VIDEO_CONTROL_LEVEL,
> > > 
> > > };
> > > 
> > > struct virtio_video_query_control {
> > > 
> > >   struct virtio_video_ctrl_hdr hdr;
> > >   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> > >   le32 length;
> > >   /* Followed by additional data.
> > >   
> > >    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
> > >    * the device must pass a codec format like H264 or VP9.
> > >    * The requred data must be defined in the specification.
> > >    */
> > > 
> > > };
> > > 
> > > struct virtio_video_query_control_resp {
> > > 
> > >   struct virtio_video_ctrl_hdr hdr;
> > >   le32 length;
> > >   u8 padding[4];
> > >   /* Followed by data corresponds to the specified control.
> > >   
> > >    * The type of data must be defined in the spec.
> > >    * For example, if the driver queries profiles, this part should be
> > >    * an array of supported profiles of a given format.
> > >    */
> > > 
> > > };
> > > 
> > > WDYT?
> > 
> > I think virtio_video_control_type should make sense. But I would disagree
> > with the need to have new QUERY_CONTROL and GET_CONTROL.
> > 
> > I assume the set of supported controls is fixed for some particular format
> > on a given IP. So we'd propose to include controls into format
> > descriptors, so we don't need to QUERY_CONTROL. This way (with
> > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. for
> > profiles.
> > 
> > struct virtio_video_format_frame {
> > 
> >         /* As proposed in Keiichi's prev email */
> > 
> > };
> > 
> > virtio_video_format_list {
> > 
> >         le32 num_entries;
> >         u8 padding[4];
> >         /* Followed by le64 entries[] */
> > 
> > };
> > 
> > struct virtio_video_format_control {
> > 
> >         le32 type;
> >         u8 padding[4];
> >         struct virtio_video_format_list values;
> > 
> > };
> > 
> > struct virtio_video_format_desc {
> > 
> >         le64 mask;
> >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> >         le32 planes_layout; /* See the thread [v2 0/1] */
> >         le32 num_frames;
> >         le32 num_controls;
> >         /* Followed by struct virtio_video_format_frame frames[] */
> >         /* Followed by struct virtio_video_format_control controls[] */
> > 
> > };
> 
> In my understanding, a set of supported levels depends on a profile.
> H.264 spec says that levels are specified within each profile.
> cf. "0.5 Profiles and levels" in
> https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU doesn't
> seem to provide a way to query levels for each profile properly, though)
> 
> So, if we want to have supported profiles and levels in format_desc as
> your idea, each supported profile should have a list of supported
> levels.
> I suppose it makes the structure of video_format_desc too complicated.
> I'd like to avoid this complexity caused by some specific formats.
> 
> Instead, I'd like to keep virtio_video_format_desc minimal and have
> QUERY_CONTROL to query format-specific values like profiles and
> levels.
> This design would make it easy to extend controls when we want to
> support new types of formats. We will just need to define new
> VIRTIO_VIDEO_CONTROL_*.
> What do you think?

Yes, I think we can keep it this way, this indeed provides more flexibility and 
this way we won't need to modify the 'format' parsing logic when more controls 
are added.

Best regards,
Dmitry.

> 
> Best regards,
> Keiichi
> 
> > SET_CONTROL seems to be mandatory. If it succeeds, we can store the
> > current
> > value locally, so there is no need to have GET_CONTROL.
> > 
> > The only exception is the initial (default) control value in the device.
> > But with the removal of 'function' and with addition of 'caps' instead,
> > the way to provide defaults is gone. So I suppose for formats we'll be
> > just using GET on driver start to get the 'defaults'. But the thing is
> > that for formats there are other uses for GET, but for controls GET
> > apparently does not make to much sense at runtime.
> > 
> > > > Also, shouldn't the profile come together with level? Would make sense
> > > > for
> > > > encoders.
> > > 
> > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> > > required when querying supported levels.
> > 
> > So probably should be enumerated together, not queried, as per the comment
> > above.
> > 
> > > > > +\item[\field{mask}] is a bitset that represents the supported
> > > > > +  combination of input and output format. If \textit{i}-th bit is
> > > > > set
> > > > > +  in \field{mask} of \textit{j}-th \field{struct
> > > > > +  virtio_video_format_desc} for input, the device supports encoding
> > > > > or
> > > > > +  decoding from the \textit{j}-th input format to \textit{i}-th
> > > > > output
> > > > > +  format.
> > > > > +\item[\field{width, height}] represents a range of resolutions
> > > > > +  supported by the device. If its \field{step} is not applicable,
> > > > > its
> > > > > +  \field{min} is equal to its \field{max}.
> > > > > +\item[\field{num_rates}] is the length of an array
> > > > > \field{frame_rates}.
> > > > > In
> > > > > case of decoder, the driver SHOULD ignore this value.
> > > > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > > > +\end{description}
> > > > > +
> > > > 
> > > > I'd guess frame rates depend on the resolution as well. This
> > > > dependency
> > > > was
> > > > clear in the v1 spec, but in the v2 there is no dependency anymore. I
> > > > think we need to update this.
> > > 
> > > That's a good point. I missed that dependency when updating the
> > > structures.
> > > So, let me update the structs like the following:
> > > 
> > > struct virtio_video_format_frame {
> > > 
> > >         struct virtio_video_format_range width;
> > >         struct virtio_video_format_range height;
> > >         le32 num_rates;
> > >         u8 padding[4];
> > >         /* Followed by struct virtio_video_format_range frame_rates[] */
> > > 
> > > };
> > > 
> > > struct virtio_video_format_desc {
> > > 
> > >         le64 mask;
> > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > >         le32 num_frames;
> > >         u8 padding[4];
> > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > 
> > > };
> > 
> > Yes, I do agree with this approach.
> > 
> > Best regards,
> > Dmitry.
> > 
> > > Best regards,
> > > Keiichi.
> > > 
> > > > Best regards,
> > > > Dmitry.
> > > > 
> > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > > > +  within the device.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +enum virtio_video_mem_type {
> > > > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_stream_create {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > */
> > > > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > */
> > > > > +        char debug_name[64];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > +virtio_video_mem_type}.
> > > > > +\item[\field{out_mem_type}] is a type of buffer management for
> > > > > output
> > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > +virtio_video_mem_type}.
> > > > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > > > +\end{description}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream
> > > > > (context)
> > > > > +  within the device.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_stream_destroy {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > > > +  queued buffers through the pipeline.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_stream_drain {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > > > +  within the device.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_mem_entry {
> > > > > +        le64 addr;
> > > > > +        le32 length;
> > > > > +        u8 padding[4];
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_resource_create {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 resource_id;
> > > > > +        le32 nr_entries;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > +\item[\field{nr_entries}] number of \field{struct
> > > > > +  virtio_video_mem_entry} memory entries.
> > > > > +\end{description}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource
> > > > > descriptor
> > > > > +  within the device.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_resource_destroy {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 resource_id;
> > > > > +        u8 padding[4];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > +\end{description}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > > > +queue.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > > > +
> > > > > +struct virtio_video_resource_queue {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 buf_type;
> > > > > +        le32 resource_id;
> > > > > +        le64 timestamp;
> > > > > +        le32 nr_data_size;
> > > > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{buf_type}] buf_type of the .
> > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > used
> > > > > +  for synchronisation.
> > > > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > > > +\item[\field{data_size}] number of data bytes within a plane.
> > > > > +\end{description}
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +enum virtio_video_buffer_flag {
> > > > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > > > +        /* Encoder only */
> > > > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_resource_queue_resp {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le64 timestamp;
> > > > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > > > +        le32 size;  /* Encoded size */
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > used
> > > > > +  for synchronisation.
> > > > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > > > +\item[\field{size}] data size in the buffer (encoder only).
> > > > > +\end{description}
> > > > > +
> > > > > +The device sends a response to the queue request asynchronously
> > > > > when
> > > > > +it has finished processing the buffer.
> > > > > +
> > > > > +The device SHOULD mark a buffer that triggered a processing error
> > > > > with
> > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > > > +
> > > > > +The device MUST mark the last buffer with the
> > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > +sequence.
> > > > > +
> > > > > +In case of encoder, to denote a particular frame type the devie
> > > > > MUST
> > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > > > +  buffers back from the input or the output queue of the device.
> > > > > The
> > > > > +  device SHOULD return all of the buffers from the respective queue
> > > > > as
> > > > > +  soon as possible without pushing the buffers through the
> > > > > processing
> > > > > +  pipeline.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_queue_clear {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 buf_type;
> > > > > +        u8 padding[4];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{buf_type}] buffer type.
> > > > > +\end{description}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > > > +  output of a stream.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_plane_format {
> > > > > +        le32 plane_size;
> > > > > +        le32 stride;
> > > > > +        u8 padding[4];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{plane_size}] size of the plane in bytes.
> > > > > +\item[\field{stride}] stride used for the plane in bytes.
> > > > > +\end{description}
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_params {
> > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > > > +        le32 frame_width;
> > > > > +        le32 frame_height;
> > > > > +        le32 min_buffers;
> > > > > +        le32 max_buffers;
> > > > > +        le32 frame_rate;
> > > > > +        struct virtio_video_crop {
> > > > > +                le32 left;
> > > > > +                le32 top;
> > > > > +                le32 width;
> > > > > +                le32 height;
> > > > > +        } crop;
> > > > > +        le32 num_planes;
> > > > > +        struct virtio_video_plane_format
> > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{frame_width}] the value to get/set.
> > > > > +\item[\field{frame_height}] the value to get/set.
> > > > > +\item[\field{pixel_format}] the value to get/set.
> > > > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > > > +  format (r/o).
> > > > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > > > +  format (r/o).
> > > > > +\item[\field{frame_rate}] the value to get/set.
> > > > > +\item[\field{crop}] cropping (composing) rectangle.
> > > > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > > > +(r/o).
> > > > > +\item[\field{plane_formats}] description of each plane.
> > > > > +\end{description}
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_get_params {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_get_params_resp {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        struct virtio_video_params params;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{buf_type}] buffer type.
> > > > > +\item[\field{params}] parameter values.
> > > > > +\end{description}
> > > > > +
> > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > > > +
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_video_set_params {
> > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > +        struct virtio_video_params params;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{params}] parameters to set.
> > > > > +\end{description}
> > > > > +
> > > > > +Setting stream parameters might have side effects within the
> > > > > device.
> > > > > +For example, the device MAY perform alignment of width and height,
> > > > > +change the number of planes it uses for the format, or do whatever
> > > > > +changes that are required to continue normal operation using the
> > > > > +updated parameters. It is up to the driver to check the parameter
> > > > > set
> > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > > > +
> > > > > +\end{description}
> > > > > +
> > > > > +\subsubsection{Device Operation: eventq}
> > > > > +
> > > > > +The device can report events on the event queue. The driver
> > > > > initially
> > > > > +populates the queue with device-writeable buffers. When the device
> > > > > +needs to report an event, it fills a buffer and notifies the
> > > > > driver.
> > > > > +The driver consumes the report and adds a new buffer to the
> > > > > virtqueue.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +enum virtio_video_event_type {
> > > > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > > > +        /* For all functions */
> > > > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > > > +        /* For decoder only */
> > > > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > > > +};
> > > > > +
> > > > > +struct virtio_video_event {
> > > > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > > > +        le32 stream_id;
> > > > > +        u8 padding[4];
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +\begin{description}
> > > > > +\item[\field{event_type}] type of the triggered event .
> > > > > +\item[\field{stream_id}] id of the source stream.
> > > > > +\end{description}
> > > > > +
> > > > > +The device MUST send
> > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > > > +whenever it encounters new resolution data in the stream. This
> > > > > +includes the case of the initial device configuration after
> > > > > metadata
> > > > > +has been parsed and the case of dynamic resolution change.
Keiichi Watanabe Jan. 8, 2020, 12:14 p.m. UTC | #31
Hi Dmitry,

On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
>
> Hi Keiichi,
>
> On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote:
> > Hi Dmitry,
> >
> > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> wrote:
> > > Hi Keiichi,
> > >
> > > thanks for the updates, please see my comments below.
> > >
> > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> > > > Hi Dmitry,
> > > >
> > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp
> > > > <dmitry.sepp@opensynergy.com>
> > >
> > > wrote:
> > > > > Hi,
> > > > >
> > > > > a couple of new comments:
> > > > >
> > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > >
> > > > > > The virtio video encoder device and decoder device provide
> > > > > > functionalities
> > > > > > to encode and decode video stream respectively.
> > > > > > Though video encoder and decoder are provided as different devices,
> > > > > > they
> > > > > > use a same protocol.
> > > > > >
> > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > > > > ---
> > > > > >
> > > > > >  content.tex      |   1 +
> > > > > >  virtio-video.tex | 579
> > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > >  2 files changed, 580 insertions(+)
> > > > > >  create mode 100644 virtio-video.tex
> > > > > >
> > > > > > diff --git a/content.tex b/content.tex
> > > > > > index 556b373..9e56839 100644
> > > > > > --- a/content.tex
> > > > > > +++ b/content.tex
> > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > > > >
> > > > > >  \input{virtio-fs.tex}
> > > > > >  \input{virtio-rpmb.tex}
> > > > > >
> > > > > > +\input{virtio-video.tex}
> > > > > >
> > > > > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > > > >
> > > > > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > > > > new file mode 100644
> > > > > > index 0000000..30e728d
> > > > > > --- /dev/null
> > > > > > +++ b/virtio-video.tex
> > > > > > @@ -0,0 +1,579 @@
> > > > > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > > > > +
> > > > > > +The virtio video encoder device and decoder device are virtual
> > > > > > devices
> > > > > > that +supports encoding and decoding respectively. Though the
> > > > > > encoder
> > > > > > and the decoder +are different devices, they use the same protocol.
> > > > > > +
> > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device /
> > > > > > Device
> > > > > > ID}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[30] encoder device
> > > > > > +\item[31] decoder device
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > > > > Virtqueues} +
> > > > > > +\begin{description}
> > > > > > +\item[0] controlq - queue for sending control commands.
> > > > > > +\item[1] eventq - queue for sending events happened in the device.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > > > > Feature
> > > > > > bits} +
> > > > > > +\begin{description}
> > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be
> > > > > > used
> > > > > > for
> > > > > > video +  buffers.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > > > > Device
> > > > > > / Feature bits} +
> > > > > > +The device MUST offer at least one of feature bits.
> > > > > > +
> > > > > > +\subsection{Device configuration layout}\label{sec:Device Types /
> > > > > > Video
> > > > > > Device / Device configuration layout} +
> > > > > > +Video device configuration uses the following layout structure:
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_config {
> > > > > > +        le32 max_cap_len;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{max_cap_len}] defines the maximum length of a
> > > > > > descriptor
> > > > > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > > > > +  MUST set this value.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > > > > Device /
> > > > > > Device Initialization} +
> > > > > > +\devicenormative{\subsubsection}{Device Initialization}{Device
> > > > > > Types /
> > > > > > Video Device / Device Initialization} +
> > > > > > +The driver SHOULD query device capability by using the
> > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the
> > > > > > initial
> > > > > > +setup.
> > > > > > +
> > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device
> > > > > > /
> > > > > > Device Operation} +
> > > > > > +The driver allocates input and output buffers and queues the
> > > > > > buffers
> > > > > > +to the device. The device performs operations on the buffers
> > > > > > according
> > > > > > +to the function in question.
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Create stream}
> > > > > > +
> > > > > > +To process buffers, the device needs to associate them with a
> > > > > > certain
> > > > > > +video stream (essentially, a context). Streams are created by
> > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > > > > +determined by the device.
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Create buffers}
> > > > > > +
> > > > > > +Buffers are used to store the actual data as well as the relevant
> > > > > > +metadata. Scatter lists are supported, so the buffer doesn't need
> > > > > > to
> > > > > > +be contiguous in guest physical memory.
> > > > > > +
> > > > > > +\begin{itemize*}
> > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > > > > +  resource that is backed by a buffer allocated from the driver's
> > > > > > +  memory.
> > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource
> > > > > > that
> > > > > > +  is no longer needed.
> > > > > > +\end{itemize*}
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Stream parameter control}
> > > > > > +
> > > > > > +\begin{itemize*}
> > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > > > > parameters
> > > > > > for +  input and output streams from the device.
> > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream
> > > > > > parameters to
> > > > > > the +  device.
> > > > > > +\item After setting stream parameters, the driver may issue
> > > > > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and
> > > > > > output
> > > > > > can be +  changed implicitly by the device during the set operation.
> > > > > > +\end{itemize*}
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Process buffers}
> > > > > > +
> > > > > > +\begin{itemize*}
> > > > > > +\item If the function and the buffer type require so, write data to
> > > > > > +the buffer memory.
> > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > > > > +processing in the device.
> > > > > > +\item The request completes asynchronously when the device has
> > > > > > +finished with the buffer.
> > > > > > +\end{itemize*}
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Buffer processing control}
> > > > > > +
> > > > > > +\begin{itemize*}
> > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process
> > > > > > and
> > > > > > +  return all of the already queued buffers.
> > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return
> > > > > > back
> > > > > > +  already queued buffers from the input or the output queue. This
> > > > > > also
> > > > > > +  includes input or output buffers that can be currently owned by
> > > > > > the
> > > > > > +  device's processing pipeline.
> > > > > > +\end{itemize*}
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Asynchronous events}
> > > > > > +
> > > > > > +While processing buffers, the device can send asynchronous event
> > > > > > +notifications to the driver. The behaviour depends on the exact
> > > > > > +stream. For example, the decoder device sends a resolution change
> > > > > > +event when it encounters new resolution metadata in the stream.
> > > > > > +
> > > > > > +\subsubsection{Device Operation: Request header}
> > > > > > +
> > > > > > +All requests and responses on the control virt queue have a fixed
> > > > > > +header using the following layout structure and definitions:
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +enum virtio_video_ctrl_type {
> > > > > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > > > > +
> > > > > > +        /* request */
> > > > > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > > > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > > > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > > > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > > > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > > > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > > > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > > > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > > > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > > > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > > > > +
> > > > > > +        /* response */
> > > > > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > > > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > > > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > > > > +
> > > > > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > > > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_ctrl_hdr {
> > > > > > +        le32 type;
> > > > > > +        le32 stream_id;
> > > > > > +        le32 len; /* Length of the structure in bytes. */
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{type}] is the type of the driver request or the device
> > > > > > +response.
> > > > > > +\item[\field{stream_id}] specifies a target stream.
> > > > > > +\item[\field{len}] is the length of data in bytes, which includes
> > > > > > +length of the header.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\subsubsection{Device Operation: controlq}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > > > > +supported formats.
> > > > > > +
> > > > > > +The driver uses \field{struct virtio_video_get_capability} to send
> > > > > > a
> > > > > > +query request.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +enum virtio_video_buf_type {
> > > > > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > > > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_get_capability {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        enum virtio_video_buf_type buf_type;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +\begin{description}
> > > > > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > > > > +information about. The driver MUST set either
> > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > > > > +
> > > > > > +The device responds a capability by using \field{struct
> > > > > > +virtio_video_get_capability_resp}.
> > > > > > +\begin{lstlisting}
> > > > > > +enum virtio_video_format {
> > > > > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > > > +        /* Raw formats */
> > > > > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > > > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > > > > +
> > > > > > +        /* Compressed formats */
> > > > > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > > > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > > > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > > > > +};
> > > > > > +
> > > > > > +enum virtio_video_profile {
> > > > > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > > > > +
> > > > > > +        /* H.264 */
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > > > > +        /* VP8 */
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY =
> > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX =
> > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > > > > +
> > > > > > +        /* VP9 */
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_format_range {
> > > > > > +        le32 min;
> > > > > > +        le32 max;
> > > > > > +        le32 step;
> > > > > > +        u8 paddings[4];
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_format_desc {
> > > > > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > > > > +        le64 mask;
> > > > > > +        struct virtio_video_format_range width;
> > > > > > +        struct virtio_video_format_range height;
> > > > > > +        le32 num_rates;
> > > > > > +        u8 padding[4];
> > > > > > +        /* Followed by struct virtio_video_frame_rate frame_rates[]
> > > > > > */
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_get_capability_resp {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 num_descs;
> > > > > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +The format description \field{struct virtio_video_format_desc}
> > > > > > +includes the following fields:
> > > > > > +\begin{description}
> > > > > > +\item[\field{format}] specifies an image format. The device MUST
> > > > > > set
> > > > > > one
> > > > > > +  of \field{enum virtio_video_format}.
> > > > > > +\item[\field{profile}] specifies a profile of the compressed image
> > > > > > format
> > > > > > +  specified in \field{format}. The driver SHOULD ignore this value
> > > > > > if
> > > > > > +  \field{format} is a raw format.
> > > > >
> > > > > So how should this be used? The spec does not define any way to set
> > > > > profile for the device. It is very important for encoder.
> > > >
> > > > Thank you for pointing this.
> > > > These points are overlooked, as I didn't care about encoder usage
> > > > enough.
> > > >
> > > > After thinking it again, I think it's not a very good idea to include
> > > > supported profiles and levels in a struct for capability.
> > > > This is because these values are available only for limited number of
> > > > formats. Also, it's true that we need to have a way to set these values
> > > > as
> > > > Dmitry pointed.
> > >
> > > Yes, you are right. In fact, the approach of the v1 spec to keep controls
> > > separately was not correct.
> > >
> > > > Instead, it would make more sense to have additional three types of
> > > > controls for profiles, levels, and bitrates:
> > > > (1) QUERY_CONTROL: Query values supported by the device
> > > > (2) GET_CONTROL: Read a value that is set in the device
> > > > (3) SET_CONTROL: Set a value in the device
> > > >
> > > > These operations are similar to V4L2 controls.
> > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
> > > > G_CTRL in V4L2, respectively.
> > > > Also, (3) would be similar to enum virtio_video_control_type in the
> > > > virtio-video v1 driver implementation in
> > > > https://markmail.org/message/dwghwdqsbl3gsjxu .
> > > >
> > > > For QUERY_CONTROL, my idea is like this:
> > > >
> > > > enum virtio_video_control_type {
> > > >
> > > >   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> > > >
> > > >   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
> > > >   VIRTIO_VIDEO_CONTROL_PROFILE,
> > > >   VIRTIO_VIDEO_CONTROL_LEVEL,
> > > >
> > > > };
> > > >
> > > > struct virtio_video_query_control {
> > > >
> > > >   struct virtio_video_ctrl_hdr hdr;
> > > >   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> > > >   le32 length;
> > > >   /* Followed by additional data.
> > > >
> > > >    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
> > > >    * the device must pass a codec format like H264 or VP9.
> > > >    * The requred data must be defined in the specification.
> > > >    */
> > > >
> > > > };
> > > >
> > > > struct virtio_video_query_control_resp {
> > > >
> > > >   struct virtio_video_ctrl_hdr hdr;
> > > >   le32 length;
> > > >   u8 padding[4];
> > > >   /* Followed by data corresponds to the specified control.
> > > >
> > > >    * The type of data must be defined in the spec.
> > > >    * For example, if the driver queries profiles, this part should be
> > > >    * an array of supported profiles of a given format.
> > > >    */
> > > >
> > > > };
> > > >
> > > > WDYT?
> > >
> > > I think virtio_video_control_type should make sense. But I would disagree
> > > with the need to have new QUERY_CONTROL and GET_CONTROL.
> > >
> > > I assume the set of supported controls is fixed for some particular format
> > > on a given IP. So we'd propose to include controls into format
> > > descriptors, so we don't need to QUERY_CONTROL. This way (with
> > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. for
> > > profiles.
> > >
> > > struct virtio_video_format_frame {
> > >
> > >         /* As proposed in Keiichi's prev email */
> > >
> > > };
> > >
> > > virtio_video_format_list {
> > >
> > >         le32 num_entries;
> > >         u8 padding[4];
> > >         /* Followed by le64 entries[] */
> > >
> > > };
> > >
> > > struct virtio_video_format_control {
> > >
> > >         le32 type;
> > >         u8 padding[4];
> > >         struct virtio_video_format_list values;
> > >
> > > };
> > >
> > > struct virtio_video_format_desc {
> > >
> > >         le64 mask;
> > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > >         le32 num_frames;
> > >         le32 num_controls;
> > >         /* Followed by struct virtio_video_format_frame frames[] */
> > >         /* Followed by struct virtio_video_format_control controls[] */
> > >
> > > };
> >
> > In my understanding, a set of supported levels depends on a profile.
> > H.264 spec says that levels are specified within each profile.
> > cf. "0.5 Profiles and levels" in
> > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU doesn't
> > seem to provide a way to query levels for each profile properly, though)
> >
> > So, if we want to have supported profiles and levels in format_desc as
> > your idea, each supported profile should have a list of supported
> > levels.
> > I suppose it makes the structure of video_format_desc too complicated.
> > I'd like to avoid this complexity caused by some specific formats.
> >
> > Instead, I'd like to keep virtio_video_format_desc minimal and have
> > QUERY_CONTROL to query format-specific values like profiles and
> > levels.
> > This design would make it easy to extend controls when we want to
> > support new types of formats. We will just need to define new
> > VIRTIO_VIDEO_CONTROL_*.
> > What do you think?
>
> Yes, I think we can keep it this way, this indeed provides more flexibility and
> this way we won't need to modify the 'format' parsing logic when more controls
> are added.

Thanks! I'm preparing the next version of the patch.

I have one concern about terminology.
We have two different things called "controls" now:
1) Messages passed via controlq. i.e. "ctrl" in 'enum
virtio_video_ctrl_type' and 'virtio_video_ctrl_hdr'.
2) Commands for profiles, levels, and bitrates similar to V4L2
controls. e.g.., "CONTROL" in VIRTIO_VIDEO_CONTROL_*

I feel calling both "controls" is somewhat confusing and I am
wondering if we can't rename either of them.

For example, how about renaming 2) to "command"? Then, we'll have
{QUERY ,GET, SET}_COMMAND.
It may sound strange, though.
Do you have any idea?

Best regards,
Keiichi

>
> Best regards,
> Dmitry.
>
> >
> > Best regards,
> > Keiichi
> >
> > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the
> > > current
> > > value locally, so there is no need to have GET_CONTROL.
> > >
> > > The only exception is the initial (default) control value in the device.
> > > But with the removal of 'function' and with addition of 'caps' instead,
> > > the way to provide defaults is gone. So I suppose for formats we'll be
> > > just using GET on driver start to get the 'defaults'. But the thing is
> > > that for formats there are other uses for GET, but for controls GET
> > > apparently does not make to much sense at runtime.
> > >
> > > > > Also, shouldn't the profile come together with level? Would make sense
> > > > > for
> > > > > encoders.
> > > >
> > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> > > > required when querying supported levels.
> > >
> > > So probably should be enumerated together, not queried, as per the comment
> > > above.
> > >
> > > > > > +\item[\field{mask}] is a bitset that represents the supported
> > > > > > +  combination of input and output format. If \textit{i}-th bit is
> > > > > > set
> > > > > > +  in \field{mask} of \textit{j}-th \field{struct
> > > > > > +  virtio_video_format_desc} for input, the device supports encoding
> > > > > > or
> > > > > > +  decoding from the \textit{j}-th input format to \textit{i}-th
> > > > > > output
> > > > > > +  format.
> > > > > > +\item[\field{width, height}] represents a range of resolutions
> > > > > > +  supported by the device. If its \field{step} is not applicable,
> > > > > > its
> > > > > > +  \field{min} is equal to its \field{max}.
> > > > > > +\item[\field{num_rates}] is the length of an array
> > > > > > \field{frame_rates}.
> > > > > > In
> > > > > > case of decoder, the driver SHOULD ignore this value.
> > > > > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > > > > +\end{description}
> > > > > > +
> > > > >
> > > > > I'd guess frame rates depend on the resolution as well. This
> > > > > dependency
> > > > > was
> > > > > clear in the v1 spec, but in the v2 there is no dependency anymore. I
> > > > > think we need to update this.
> > > >
> > > > That's a good point. I missed that dependency when updating the
> > > > structures.
> > > > So, let me update the structs like the following:
> > > >
> > > > struct virtio_video_format_frame {
> > > >
> > > >         struct virtio_video_format_range width;
> > > >         struct virtio_video_format_range height;
> > > >         le32 num_rates;
> > > >         u8 padding[4];
> > > >         /* Followed by struct virtio_video_format_range frame_rates[] */
> > > >
> > > > };
> > > >
> > > > struct virtio_video_format_desc {
> > > >
> > > >         le64 mask;
> > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > >         le32 num_frames;
> > > >         u8 padding[4];
> > > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > >
> > > > };
> > >
> > > Yes, I do agree with this approach.
> > >
> > > Best regards,
> > > Dmitry.
> > >
> > > > Best regards,
> > > > Keiichi.
> > > >
> > > > > Best regards,
> > > > > Dmitry.
> > > > >
> > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > > > > +  within the device.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +enum virtio_video_mem_type {
> > > > > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_stream_create {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > > */
> > > > > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > > */
> > > > > > +        char debug_name[64];
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > +virtio_video_mem_type}.
> > > > > > +\item[\field{out_mem_type}] is a type of buffer management for
> > > > > > output
> > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > +virtio_video_mem_type}.
> > > > > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream
> > > > > > (context)
> > > > > > +  within the device.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_stream_destroy {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > > > > +  queued buffers through the pipeline.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_stream_drain {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > > > > +  within the device.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_mem_entry {
> > > > > > +        le64 addr;
> > > > > > +        le32 length;
> > > > > > +        u8 padding[4];
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_resource_create {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 resource_id;
> > > > > > +        le32 nr_entries;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > +\item[\field{nr_entries}] number of \field{struct
> > > > > > +  virtio_video_mem_entry} memory entries.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource
> > > > > > descriptor
> > > > > > +  within the device.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_resource_destroy {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 resource_id;
> > > > > > +        u8 padding[4];
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > > > > +queue.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > > > > +
> > > > > > +struct virtio_video_resource_queue {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 buf_type;
> > > > > > +        le32 resource_id;
> > > > > > +        le64 timestamp;
> > > > > > +        le32 nr_data_size;
> > > > > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{buf_type}] buf_type of the .
> > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > > used
> > > > > > +  for synchronisation.
> > > > > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > > > > +\item[\field{data_size}] number of data bytes within a plane.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +enum virtio_video_buffer_flag {
> > > > > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > > > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > > > > +        /* Encoder only */
> > > > > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > > > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > > > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_resource_queue_resp {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le64 timestamp;
> > > > > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > > > > +        le32 size;  /* Encoded size */
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > > used
> > > > > > +  for synchronisation.
> > > > > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > > > > +\item[\field{size}] data size in the buffer (encoder only).
> > > > > > +\end{description}
> > > > > > +
> > > > > > +The device sends a response to the queue request asynchronously
> > > > > > when
> > > > > > +it has finished processing the buffer.
> > > > > > +
> > > > > > +The device SHOULD mark a buffer that triggered a processing error
> > > > > > with
> > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > > > > +
> > > > > > +The device MUST mark the last buffer with the
> > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > > +sequence.
> > > > > > +
> > > > > > +In case of encoder, to denote a particular frame type the devie
> > > > > > MUST
> > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > > > > +  buffers back from the input or the output queue of the device.
> > > > > > The
> > > > > > +  device SHOULD return all of the buffers from the respective queue
> > > > > > as
> > > > > > +  soon as possible without pushing the buffers through the
> > > > > > processing
> > > > > > +  pipeline.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_queue_clear {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 buf_type;
> > > > > > +        u8 padding[4];
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > > > > +  output of a stream.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_plane_format {
> > > > > > +        le32 plane_size;
> > > > > > +        le32 stride;
> > > > > > +        u8 padding[4];
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{plane_size}] size of the plane in bytes.
> > > > > > +\item[\field{stride}] stride used for the plane in bytes.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_params {
> > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > > > > +        le32 frame_width;
> > > > > > +        le32 frame_height;
> > > > > > +        le32 min_buffers;
> > > > > > +        le32 max_buffers;
> > > > > > +        le32 frame_rate;
> > > > > > +        struct virtio_video_crop {
> > > > > > +                le32 left;
> > > > > > +                le32 top;
> > > > > > +                le32 width;
> > > > > > +                le32 height;
> > > > > > +        } crop;
> > > > > > +        le32 num_planes;
> > > > > > +        struct virtio_video_plane_format
> > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{frame_width}] the value to get/set.
> > > > > > +\item[\field{frame_height}] the value to get/set.
> > > > > > +\item[\field{pixel_format}] the value to get/set.
> > > > > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > > > > +  format (r/o).
> > > > > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > > > > +  format (r/o).
> > > > > > +\item[\field{frame_rate}] the value to get/set.
> > > > > > +\item[\field{crop}] cropping (composing) rectangle.
> > > > > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > > > > +(r/o).
> > > > > > +\item[\field{plane_formats}] description of each plane.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_get_params {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_get_params_resp {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        struct virtio_video_params params;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > +\item[\field{params}] parameter values.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > > > > +
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +struct virtio_video_set_params {
> > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > +        struct virtio_video_params params;
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{params}] parameters to set.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +Setting stream parameters might have side effects within the
> > > > > > device.
> > > > > > +For example, the device MAY perform alignment of width and height,
> > > > > > +change the number of planes it uses for the format, or do whatever
> > > > > > +changes that are required to continue normal operation using the
> > > > > > +updated parameters. It is up to the driver to check the parameter
> > > > > > set
> > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > > > > +
> > > > > > +\end{description}
> > > > > > +
> > > > > > +\subsubsection{Device Operation: eventq}
> > > > > > +
> > > > > > +The device can report events on the event queue. The driver
> > > > > > initially
> > > > > > +populates the queue with device-writeable buffers. When the device
> > > > > > +needs to report an event, it fills a buffer and notifies the
> > > > > > driver.
> > > > > > +The driver consumes the report and adds a new buffer to the
> > > > > > virtqueue.
> > > > > > +
> > > > > > +\begin{lstlisting}
> > > > > > +enum virtio_video_event_type {
> > > > > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > > > > +        /* For all functions */
> > > > > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > > > > +        /* For decoder only */
> > > > > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > > > > +};
> > > > > > +
> > > > > > +struct virtio_video_event {
> > > > > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > > > > +        le32 stream_id;
> > > > > > +        u8 padding[4];
> > > > > > +};
> > > > > > +\end{lstlisting}
> > > > > > +
> > > > > > +\begin{description}
> > > > > > +\item[\field{event_type}] type of the triggered event .
> > > > > > +\item[\field{stream_id}] id of the source stream.
> > > > > > +\end{description}
> > > > > > +
> > > > > > +The device MUST send
> > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > > > > +whenever it encounters new resolution data in the stream. This
> > > > > > +includes the case of the initial device configuration after
> > > > > > metadata
> > > > > > +has been parsed and the case of dynamic resolution change.
>
>
Keiichi Watanabe Jan. 8, 2020, 12:23 p.m. UTC | #32
Hi Dmitry,

On Wed, Dec 18, 2019 at 10:02 PM Keiichi Watanabe <keiichiw@chromium.org> wrote:
>
> From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
>
> The virtio video encoder device and decoder device provide functionalities to
> encode and decode video stream respectively.
> Though video encoder and decoder are provided as different devices, they use a
> same protocol.
>
> Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> ---
>  content.tex      |   1 +
>  virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 580 insertions(+)
>  create mode 100644 virtio-video.tex
>
> diff --git a/content.tex b/content.tex
> index 556b373..9e56839 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
>  \input{virtio-vsock.tex}
>  \input{virtio-fs.tex}
>  \input{virtio-rpmb.tex}
> +\input{virtio-video.tex}
>
>  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>
> diff --git a/virtio-video.tex b/virtio-video.tex
> new file mode 100644
> index 0000000..30e728d
> --- /dev/null
> +++ b/virtio-video.tex
> @@ -0,0 +1,579 @@
> +\section{Video Device}\label{sec:Device Types / Video Device}
> +
> +The virtio video encoder device and decoder device are virtual devices that
> +supports encoding and decoding respectively. Though the encoder and the decoder
> +are different devices, they use the same protocol.
> +
> +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID}
> +
> +\begin{description}
> +\item[30] encoder device
> +\item[31] decoder device
> +\end{description}
> +
> +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues}
> +
> +\begin{description}
> +\item[0] controlq - queue for sending control commands.
> +\item[1] eventq - queue for sending events happened in the device.
> +\end{description}
> +
> +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature bits}
> +
> +\begin{description}
> +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for video
> +  buffers.
> +\end{description}
> +
> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits}
> +
> +The device MUST offer at least one of feature bits.
> +
> +\subsection{Device configuration layout}\label{sec:Device Types / Video Device / Device configuration layout}
> +
> +Video device configuration uses the following layout structure:
> +
> +\begin{lstlisting}
> +struct virtio_video_config {
> +        le32 max_cap_len;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{max_cap_len}] defines the maximum length of a descriptor
> +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> +  MUST set this value.
> +\end{description}
> +
> +\subsection{Device Initialization}\label{sec:Device Types / Video Device / Device Initialization}
> +
> +\devicenormative{\subsubsection}{Device Initialization}{Device Types / Video Device / Device Initialization}
> +
> +The driver SHOULD query device capability by using the
> +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial
> +setup.
> +
> +\subsection{Device Operation}\label{sec:Device Types / Video Device / Device Operation}
> +
> +The driver allocates input and output buffers and queues the buffers
> +to the device. The device performs operations on the buffers according
> +to the function in question.
> +
> +\subsubsection{Device Operation: Create stream}
> +
> +To process buffers, the device needs to associate them with a certain
> +video stream (essentially, a context). Streams are created by
> +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> +determined by the device.
> +
> +\subsubsection{Device Operation: Create buffers}
> +
> +Buffers are used to store the actual data as well as the relevant
> +metadata. Scatter lists are supported, so the buffer doesn't need to
> +be contiguous in guest physical memory.
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> +  resource that is backed by a buffer allocated from the driver's
> +  memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that
> +  is no longer needed.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Stream parameter control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters for
> +  input and output streams from the device.
> +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the
> +  device.
> +\item After setting stream parameters, the driver may issue
> +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can be
> +  changed implicitly by the device during the set operation.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Process buffers}
> +
> +\begin{itemize*}
> +\item If the function and the buffer type require so, write data to
> +the buffer memory.
> +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> +processing in the device.
> +\item The request completes asynchronously when the device has
> +finished with the buffer.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Buffer processing control}
> +
> +\begin{itemize*}
> +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and
> +  return all of the already queued buffers.
> +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back
> +  already queued buffers from the input or the output queue. This also
> +  includes input or output buffers that can be currently owned by the
> +  device's processing pipeline.
> +\end{itemize*}
> +
> +\subsubsection{Device Operation: Asynchronous events}
> +
> +While processing buffers, the device can send asynchronous event
> +notifications to the driver. The behaviour depends on the exact
> +stream. For example, the decoder device sends a resolution change
> +event when it encounters new resolution metadata in the stream.
> +
> +\subsubsection{Device Operation: Request header}
> +
> +All requests and responses on the control virt queue have a fixed
> +header using the following layout structure and definitions:
> +
> +\begin{lstlisting}
> +enum virtio_video_ctrl_type {
> +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> +
> +        /* request */
> +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> +        VIRTIO_VIDEO_T_STREAM_CREATE,
> +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> +        VIRTIO_VIDEO_T_SET_PARAMS,
> +        VIRTIO_VIDEO_T_GET_PARAMS,
> +
> +        /* response */
> +        VIRTIO_VIDEO_S_OK = 0x0200,
> +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> +
> +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> +};

I have a question about naming of these constants.
What's the meaning of "T" and "S" for requests and responses, respectively?
Though I copied these values from your first version, I'm wondering if
we can't call VIRTIO_VIDEO_REQ_* and VIRTIO_VIDEO_RESP_* instead.

Best regards,
Keiichi

> +
> +struct virtio_video_ctrl_hdr {
> +        le32 type;
> +        le32 stream_id;
> +        le32 len; /* Length of the structure in bytes. */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{type}] is the type of the driver request or the device
> +response.
> +\item[\field{stream_id}] specifies a target stream.
> +\item[\field{len}] is the length of data in bytes, which includes
> +length of the header.
> +\end{description}
> +
> +\subsubsection{Device Operation: controlq}
> +
> +\begin{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> +supported formats.
> +
> +The driver uses \field{struct virtio_video_get_capability} to send a
> +query request.
> +
> +\begin{lstlisting}
> +enum virtio_video_buf_type {
> +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> +};
> +
> +struct virtio_video_get_capability {
> +        struct virtio_video_ctrl_hdr hdr;
> +        enum virtio_video_buf_type buf_type;
> +};
> +\end{lstlisting}
> +\begin{description}
> +\item[\field{buf_type}] is the buffer type that the driver asks
> +information about. The driver MUST set either
> +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}.
> +\end{description}
> +
> +The device responds a capability by using \field{struct
> +virtio_video_get_capability_resp}.
> +\begin{lstlisting}
> +enum virtio_video_format {
> +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> +        /* Raw formats */
> +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> +        VIRTIO_VIDEO_FORMAT_YUV420,
> +        VIRTIO_VIDEO_FORMAT_YVU420,
> +
> +        /* Compressed formats */
> +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> +};
> +
> +enum virtio_video_profile {
> +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> +
> +        /* H.264 */
> +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> +        VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_BASELINE,
> +        VIRTIO_VIDEO_PROFILE_H264_MAIN,
> +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +        VIRTIO_VIDEO_PROFILE_H264_MAX = VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +
> +        /* VP8 */
> +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> +        VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY,
> +
> +        /* VP9 */
> +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +        VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +};
> +
> +struct virtio_video_format_range {
> +        le32 min;
> +        le32 max;
> +        le32 step;
> +        u8 paddings[4];
> +};
> +
> +struct virtio_video_format_desc {
> +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> +        le64 mask;
> +        struct virtio_video_format_range width;
> +        struct virtio_video_format_range height;
> +        le32 num_rates;
> +        u8 padding[4];
> +        /* Followed by struct virtio_video_frame_rate frame_rates[] */
> +};
> +
> +struct virtio_video_get_capability_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 num_descs;
> +        /* Followed by struct virtio_video_format_desc desc[] */
> +};
> +\end{lstlisting}
> +
> +The format description \field{struct virtio_video_format_desc}
> +includes the following fields:
> +\begin{description}
> +\item[\field{format}] specifies an image format. The device MUST set one
> +  of \field{enum virtio_video_format}.
> +\item[\field{profile}] specifies a profile of the compressed image format
> +  specified in \field{format}. The driver SHOULD ignore this value if
> +  \field{format} is a raw format.
> +\item[\field{mask}] is a bitset that represents the supported
> +  combination of input and output format. If \textit{i}-th bit is set
> +  in \field{mask} of \textit{j}-th \field{struct
> +  virtio_video_format_desc} for input, the device supports encoding or
> +  decoding from the \textit{j}-th input format to \textit{i}-th output
> +  format.
> +\item[\field{width, height}] represents a range of resolutions
> +  supported by the device. If its \field{step} is not applicable, its
> +  \field{min} is equal to its \field{max}.
> +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In case of decoder, the driver SHOULD ignore this value.
> +\item[\field{frame_rates}] is an array of supported frame rates.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +enum virtio_video_mem_type {
> +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> +};
> +
> +struct virtio_video_stream_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +        char debug_name[64];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{in_mem_type}] is a type of buffer management for input
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{out_mem_type}] is a type of buffer management for output
> +buffers. The driver MUST set a value in \field{enum
> +virtio_video_mem_type}.
> +\item[\field{debug_name}] is a text string for a debug purpose.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context)
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> +  queued buffers through the pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_stream_drain {
> +        struct virtio_video_ctrl_hdr hdr;
> +};
> +\end{lstlisting}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_mem_entry {
> +        le64 addr;
> +        le32 length;
> +        u8 padding[4];
> +};
> +
> +struct virtio_video_resource_create {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        le32 nr_entries;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{nr_entries}] number of \field{struct
> +  virtio_video_mem_entry} memory entries.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor
> +  within the device.
> +
> +\begin{lstlisting}
> +struct virtio_video_resource_destroy {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 resource_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{resource_id}] internal id of the resource.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> +queue.
> +
> +\begin{lstlisting}
> +#define VIRTIO_VIDEO_MAX_PLANES 8
> +
> +struct virtio_video_resource_queue {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        le32 resource_id;
> +        le64 timestamp;
> +        le32 nr_data_size;
> +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buf_type of the .
> +\item[\field{resource_id}] internal id of the resource.
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{nr_data_size}] number of \field{data_size} entries.
> +\item[\field{data_size}] number of data bytes within a plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +enum virtio_video_buffer_flag {
> +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> +        /* Encoder only */
> +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> +};
> +
> +struct virtio_video_resource_queue_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le64 timestamp;
> +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> +        le32 size;  /* Encoded size */
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{timestamp}] an abstract sequence counter that can be used
> +  for synchronisation.
> +\item[\field{flags}] mark specific buffers in the sequence.
> +\item[\field{size}] data size in the buffer (encoder only).
> +\end{description}
> +
> +The device sends a response to the queue request asynchronously when
> +it has finished processing the buffer.
> +
> +The device SHOULD mark a buffer that triggered a processing error with
> +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> +
> +The device MUST mark the last buffer with the
> +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> +sequence.
> +
> +In case of encoder, to denote a particular frame type the devie MUST
> +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> +
> +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> +  buffers back from the input or the output queue of the device. The
> +  device SHOULD return all of the buffers from the respective queue as
> +  soon as possible without pushing the buffers through the processing
> +  pipeline.
> +
> +\begin{lstlisting}
> +struct virtio_video_queue_clear {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> +  output of a stream.
> +
> +\begin{lstlisting}
> +struct virtio_video_plane_format {
> +        le32 plane_size;
> +        le32 stride;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{plane_size}] size of the plane in bytes.
> +\item[\field{stride}] stride used for the plane in bytes.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_params {
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> +        le32 frame_width;
> +        le32 frame_height;
> +        le32 min_buffers;
> +        le32 max_buffers;
> +        le32 frame_rate;
> +        struct virtio_video_crop {
> +                le32 left;
> +                le32 top;
> +                le32 width;
> +                le32 height;
> +        } crop;
> +        le32 num_planes;
> +        struct virtio_video_plane_format plane_formats[VIRTIO_VIDEO_MAX_PLANES];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{frame_width}] the value to get/set.
> +\item[\field{frame_height}] the value to get/set.
> +\item[\field{pixel_format}] the value to get/set.
> +\item[\field{min_buffers}] minimum buffers required to handle the
> +  format (r/o).
> +\item[\field{max_buffers}] maximum buffers required to handle the
> +  format (r/o).
> +\item[\field{frame_rate}] the value to get/set.
> +\item[\field{crop}] cropping (composing) rectangle.
> +\item[\field{num_planes}] number of planes used to store pixel data
> +(r/o).
> +\item[\field{plane_formats}] description of each plane.
> +\end{description}
> +
> +\begin{lstlisting}
> +struct virtio_video_get_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> +};
> +
> +struct virtio_video_get_params_resp {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{buf_type}] buffer type.
> +\item[\field{params}] parameter values.
> +\end{description}
> +
> +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> +
> +
> +\begin{lstlisting}
> +struct virtio_video_set_params {
> +        struct virtio_video_ctrl_hdr hdr;
> +        struct virtio_video_params params;
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{params}] parameters to set.
> +\end{description}
> +
> +Setting stream parameters might have side effects within the device.
> +For example, the device MAY perform alignment of width and height,
> +change the number of planes it uses for the format, or do whatever
> +changes that are required to continue normal operation using the
> +updated parameters. It is up to the driver to check the parameter set
> +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> +
> +\end{description}
> +
> +\subsubsection{Device Operation: eventq}
> +
> +The device can report events on the event queue. The driver initially
> +populates the queue with device-writeable buffers. When the device
> +needs to report an event, it fills a buffer and notifies the driver.
> +The driver consumes the report and adds a new buffer to the virtqueue.
> +
> +\begin{lstlisting}
> +enum virtio_video_event_type {
> +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> +        /* For all functions */
> +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> +        /* For decoder only */
> +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> +};
> +
> +struct virtio_video_event {
> +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> +        le32 stream_id;
> +        u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +\begin{description}
> +\item[\field{event_type}] type of the triggered event .
> +\item[\field{stream_id}] id of the source stream.
> +\end{description}
> +
> +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> +whenever it encounters new resolution data in the stream. This
> +includes the case of the initial device configuration after metadata
> +has been parsed and the case of dynamic resolution change.
> --
> 2.24.1.735.g03f4e72817-goog
>
Tomasz Figa Jan. 8, 2020, 12:46 p.m. UTC | #33
On Wed, Jan 8, 2020 at 9:15 PM Keiichi Watanabe <keiichiw@chromium.org> wrote:
>
> Hi Dmitry,
>
> On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
> >
> > Hi Keiichi,
> >
> > On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote:
> > > Hi Dmitry,
> > >
> > > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > wrote:
> > > > Hi Keiichi,
> > > >
> > > > thanks for the updates, please see my comments below.
> > > >
> > > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> > > > > Hi Dmitry,
> > > > >
> > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp
> > > > > <dmitry.sepp@opensynergy.com>
> > > >
> > > > wrote:
> > > > > > Hi,
> > > > > >
> > > > > > a couple of new comments:
> > > > > >
> > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > >
> > > > > > > The virtio video encoder device and decoder device provide
> > > > > > > functionalities
> > > > > > > to encode and decode video stream respectively.
> > > > > > > Though video encoder and decoder are provided as different devices,
> > > > > > > they
> > > > > > > use a same protocol.
> > > > > > >
> > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > > > > > ---
> > > > > > >
> > > > > > >  content.tex      |   1 +
> > > > > > >  virtio-video.tex | 579
> > > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > >  2 files changed, 580 insertions(+)
> > > > > > >  create mode 100644 virtio-video.tex
> > > > > > >
> > > > > > > diff --git a/content.tex b/content.tex
> > > > > > > index 556b373..9e56839 100644
> > > > > > > --- a/content.tex
> > > > > > > +++ b/content.tex
> > > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > > > > >
> > > > > > >  \input{virtio-fs.tex}
> > > > > > >  \input{virtio-rpmb.tex}
> > > > > > >
> > > > > > > +\input{virtio-video.tex}
> > > > > > >
> > > > > > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > > > > >
> > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > > > > > new file mode 100644
> > > > > > > index 0000000..30e728d
> > > > > > > --- /dev/null
> > > > > > > +++ b/virtio-video.tex
> > > > > > > @@ -0,0 +1,579 @@
> > > > > > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > > > > > +
> > > > > > > +The virtio video encoder device and decoder device are virtual
> > > > > > > devices
> > > > > > > that +supports encoding and decoding respectively. Though the
> > > > > > > encoder
> > > > > > > and the decoder +are different devices, they use the same protocol.
> > > > > > > +
> > > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device /
> > > > > > > Device
> > > > > > > ID}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[30] encoder device
> > > > > > > +\item[31] decoder device
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > > > > > Virtqueues} +
> > > > > > > +\begin{description}
> > > > > > > +\item[0] controlq - queue for sending control commands.
> > > > > > > +\item[1] eventq - queue for sending events happened in the device.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > > > > > Feature
> > > > > > > bits} +
> > > > > > > +\begin{description}
> > > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be
> > > > > > > used
> > > > > > > for
> > > > > > > video +  buffers.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > > > > > Device
> > > > > > > / Feature bits} +
> > > > > > > +The device MUST offer at least one of feature bits.
> > > > > > > +
> > > > > > > +\subsection{Device configuration layout}\label{sec:Device Types /
> > > > > > > Video
> > > > > > > Device / Device configuration layout} +
> > > > > > > +Video device configuration uses the following layout structure:
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_config {
> > > > > > > +        le32 max_cap_len;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{max_cap_len}] defines the maximum length of a
> > > > > > > descriptor
> > > > > > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > > > > > +  MUST set this value.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > > > > > Device /
> > > > > > > Device Initialization} +
> > > > > > > +\devicenormative{\subsubsection}{Device Initialization}{Device
> > > > > > > Types /
> > > > > > > Video Device / Device Initialization} +
> > > > > > > +The driver SHOULD query device capability by using the
> > > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the
> > > > > > > initial
> > > > > > > +setup.
> > > > > > > +
> > > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device
> > > > > > > /
> > > > > > > Device Operation} +
> > > > > > > +The driver allocates input and output buffers and queues the
> > > > > > > buffers
> > > > > > > +to the device. The device performs operations on the buffers
> > > > > > > according
> > > > > > > +to the function in question.
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Create stream}
> > > > > > > +
> > > > > > > +To process buffers, the device needs to associate them with a
> > > > > > > certain
> > > > > > > +video stream (essentially, a context). Streams are created by
> > > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > > > > > +determined by the device.
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Create buffers}
> > > > > > > +
> > > > > > > +Buffers are used to store the actual data as well as the relevant
> > > > > > > +metadata. Scatter lists are supported, so the buffer doesn't need
> > > > > > > to
> > > > > > > +be contiguous in guest physical memory.
> > > > > > > +
> > > > > > > +\begin{itemize*}
> > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > > > > > +  resource that is backed by a buffer allocated from the driver's
> > > > > > > +  memory.
> > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource
> > > > > > > that
> > > > > > > +  is no longer needed.
> > > > > > > +\end{itemize*}
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Stream parameter control}
> > > > > > > +
> > > > > > > +\begin{itemize*}
> > > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > > > > > parameters
> > > > > > > for +  input and output streams from the device.
> > > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream
> > > > > > > parameters to
> > > > > > > the +  device.
> > > > > > > +\item After setting stream parameters, the driver may issue
> > > > > > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and
> > > > > > > output
> > > > > > > can be +  changed implicitly by the device during the set operation.
> > > > > > > +\end{itemize*}
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Process buffers}
> > > > > > > +
> > > > > > > +\begin{itemize*}
> > > > > > > +\item If the function and the buffer type require so, write data to
> > > > > > > +the buffer memory.
> > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > > > > > +processing in the device.
> > > > > > > +\item The request completes asynchronously when the device has
> > > > > > > +finished with the buffer.
> > > > > > > +\end{itemize*}
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Buffer processing control}
> > > > > > > +
> > > > > > > +\begin{itemize*}
> > > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process
> > > > > > > and
> > > > > > > +  return all of the already queued buffers.
> > > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return
> > > > > > > back
> > > > > > > +  already queued buffers from the input or the output queue. This
> > > > > > > also
> > > > > > > +  includes input or output buffers that can be currently owned by
> > > > > > > the
> > > > > > > +  device's processing pipeline.
> > > > > > > +\end{itemize*}
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Asynchronous events}
> > > > > > > +
> > > > > > > +While processing buffers, the device can send asynchronous event
> > > > > > > +notifications to the driver. The behaviour depends on the exact
> > > > > > > +stream. For example, the decoder device sends a resolution change
> > > > > > > +event when it encounters new resolution metadata in the stream.
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: Request header}
> > > > > > > +
> > > > > > > +All requests and responses on the control virt queue have a fixed
> > > > > > > +header using the following layout structure and definitions:
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +enum virtio_video_ctrl_type {
> > > > > > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > > > > > +
> > > > > > > +        /* request */
> > > > > > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > > > > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > > > > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > > > > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > > > > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > > > > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > > > > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > > > > > +
> > > > > > > +        /* response */
> > > > > > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > > > > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > > > > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > > > > > +
> > > > > > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > > > > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_ctrl_hdr {
> > > > > > > +        le32 type;
> > > > > > > +        le32 stream_id;
> > > > > > > +        le32 len; /* Length of the structure in bytes. */
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{type}] is the type of the driver request or the device
> > > > > > > +response.
> > > > > > > +\item[\field{stream_id}] specifies a target stream.
> > > > > > > +\item[\field{len}] is the length of data in bytes, which includes
> > > > > > > +length of the header.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: controlq}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > > > > > +supported formats.
> > > > > > > +
> > > > > > > +The driver uses \field{struct virtio_video_get_capability} to send
> > > > > > > a
> > > > > > > +query request.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +enum virtio_video_buf_type {
> > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_get_capability {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        enum virtio_video_buf_type buf_type;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > > > > > +information about. The driver MUST set either
> > > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > > > > > +
> > > > > > > +The device responds a capability by using \field{struct
> > > > > > > +virtio_video_get_capability_resp}.
> > > > > > > +\begin{lstlisting}
> > > > > > > +enum virtio_video_format {
> > > > > > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > > > > +        /* Raw formats */
> > > > > > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > > > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > > > > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > > > > > +
> > > > > > > +        /* Compressed formats */
> > > > > > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > > > > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > > > > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > > > > > +};
> > > > > > > +
> > > > > > > +enum virtio_video_profile {
> > > > > > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > > > > > +
> > > > > > > +        /* H.264 */
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > > > > > +        /* VP8 */
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY =
> > > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX =
> > > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > > > > > +
> > > > > > > +        /* VP9 */
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_format_range {
> > > > > > > +        le32 min;
> > > > > > > +        le32 max;
> > > > > > > +        le32 step;
> > > > > > > +        u8 paddings[4];
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_format_desc {
> > > > > > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > > > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > > > > > +        le64 mask;
> > > > > > > +        struct virtio_video_format_range width;
> > > > > > > +        struct virtio_video_format_range height;
> > > > > > > +        le32 num_rates;
> > > > > > > +        u8 padding[4];
> > > > > > > +        /* Followed by struct virtio_video_frame_rate frame_rates[]
> > > > > > > */
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_get_capability_resp {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 num_descs;
> > > > > > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +The format description \field{struct virtio_video_format_desc}
> > > > > > > +includes the following fields:
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{format}] specifies an image format. The device MUST
> > > > > > > set
> > > > > > > one
> > > > > > > +  of \field{enum virtio_video_format}.
> > > > > > > +\item[\field{profile}] specifies a profile of the compressed image
> > > > > > > format
> > > > > > > +  specified in \field{format}. The driver SHOULD ignore this value
> > > > > > > if
> > > > > > > +  \field{format} is a raw format.
> > > > > >
> > > > > > So how should this be used? The spec does not define any way to set
> > > > > > profile for the device. It is very important for encoder.
> > > > >
> > > > > Thank you for pointing this.
> > > > > These points are overlooked, as I didn't care about encoder usage
> > > > > enough.
> > > > >
> > > > > After thinking it again, I think it's not a very good idea to include
> > > > > supported profiles and levels in a struct for capability.
> > > > > This is because these values are available only for limited number of
> > > > > formats. Also, it's true that we need to have a way to set these values
> > > > > as
> > > > > Dmitry pointed.
> > > >
> > > > Yes, you are right. In fact, the approach of the v1 spec to keep controls
> > > > separately was not correct.
> > > >
> > > > > Instead, it would make more sense to have additional three types of
> > > > > controls for profiles, levels, and bitrates:
> > > > > (1) QUERY_CONTROL: Query values supported by the device
> > > > > (2) GET_CONTROL: Read a value that is set in the device
> > > > > (3) SET_CONTROL: Set a value in the device
> > > > >
> > > > > These operations are similar to V4L2 controls.
> > > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
> > > > > G_CTRL in V4L2, respectively.
> > > > > Also, (3) would be similar to enum virtio_video_control_type in the
> > > > > virtio-video v1 driver implementation in
> > > > > https://markmail.org/message/dwghwdqsbl3gsjxu .
> > > > >
> > > > > For QUERY_CONTROL, my idea is like this:
> > > > >
> > > > > enum virtio_video_control_type {
> > > > >
> > > > >   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> > > > >
> > > > >   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
> > > > >   VIRTIO_VIDEO_CONTROL_PROFILE,
> > > > >   VIRTIO_VIDEO_CONTROL_LEVEL,
> > > > >
> > > > > };
> > > > >
> > > > > struct virtio_video_query_control {
> > > > >
> > > > >   struct virtio_video_ctrl_hdr hdr;
> > > > >   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> > > > >   le32 length;
> > > > >   /* Followed by additional data.
> > > > >
> > > > >    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
> > > > >    * the device must pass a codec format like H264 or VP9.
> > > > >    * The requred data must be defined in the specification.
> > > > >    */
> > > > >
> > > > > };
> > > > >
> > > > > struct virtio_video_query_control_resp {
> > > > >
> > > > >   struct virtio_video_ctrl_hdr hdr;
> > > > >   le32 length;
> > > > >   u8 padding[4];
> > > > >   /* Followed by data corresponds to the specified control.
> > > > >
> > > > >    * The type of data must be defined in the spec.
> > > > >    * For example, if the driver queries profiles, this part should be
> > > > >    * an array of supported profiles of a given format.
> > > > >    */
> > > > >
> > > > > };
> > > > >
> > > > > WDYT?
> > > >
> > > > I think virtio_video_control_type should make sense. But I would disagree
> > > > with the need to have new QUERY_CONTROL and GET_CONTROL.
> > > >
> > > > I assume the set of supported controls is fixed for some particular format
> > > > on a given IP. So we'd propose to include controls into format
> > > > descriptors, so we don't need to QUERY_CONTROL. This way (with
> > > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. for
> > > > profiles.
> > > >
> > > > struct virtio_video_format_frame {
> > > >
> > > >         /* As proposed in Keiichi's prev email */
> > > >
> > > > };
> > > >
> > > > virtio_video_format_list {
> > > >
> > > >         le32 num_entries;
> > > >         u8 padding[4];
> > > >         /* Followed by le64 entries[] */
> > > >
> > > > };
> > > >
> > > > struct virtio_video_format_control {
> > > >
> > > >         le32 type;
> > > >         u8 padding[4];
> > > >         struct virtio_video_format_list values;
> > > >
> > > > };
> > > >
> > > > struct virtio_video_format_desc {
> > > >
> > > >         le64 mask;
> > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > >         le32 num_frames;
> > > >         le32 num_controls;
> > > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > >         /* Followed by struct virtio_video_format_control controls[] */
> > > >
> > > > };
> > >
> > > In my understanding, a set of supported levels depends on a profile.
> > > H.264 spec says that levels are specified within each profile.
> > > cf. "0.5 Profiles and levels" in
> > > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU doesn't
> > > seem to provide a way to query levels for each profile properly, though)
> > >
> > > So, if we want to have supported profiles and levels in format_desc as
> > > your idea, each supported profile should have a list of supported
> > > levels.
> > > I suppose it makes the structure of video_format_desc too complicated.
> > > I'd like to avoid this complexity caused by some specific formats.
> > >
> > > Instead, I'd like to keep virtio_video_format_desc minimal and have
> > > QUERY_CONTROL to query format-specific values like profiles and
> > > levels.
> > > This design would make it easy to extend controls when we want to
> > > support new types of formats. We will just need to define new
> > > VIRTIO_VIDEO_CONTROL_*.
> > > What do you think?
> >
> > Yes, I think we can keep it this way, this indeed provides more flexibility and
> > this way we won't need to modify the 'format' parsing logic when more controls
> > are added.
>
> Thanks! I'm preparing the next version of the patch.
>
> I have one concern about terminology.
> We have two different things called "controls" now:
> 1) Messages passed via controlq. i.e. "ctrl" in 'enum
> virtio_video_ctrl_type' and 'virtio_video_ctrl_hdr'.
> 2) Commands for profiles, levels, and bitrates similar to V4L2
> controls. e.g.., "CONTROL" in VIRTIO_VIDEO_CONTROL_*
>
> I feel calling both "controls" is somewhat confusing and I am
> wondering if we can't rename either of them.
>
> For example, how about renaming 2) to "command"? Then, we'll have
> {QUERY ,GET, SET}_COMMAND.
> It may sound strange, though.
> Do you have any idea?

"Command" sound like executing an action, but in this context we're
thinking more about parameters or, exactly, controls. The term
"control" is not limited to V4L2 and actually used in other subsystems
as well, for example the Android Camera HAL.

How about renaming 1) to "commandq" and then having the other symbols
use the abbreviated "cmd"?

Best regards,
Tomasz

>
> Best regards,
> Keiichi
>
> >
> > Best regards,
> > Dmitry.
> >
> > >
> > > Best regards,
> > > Keiichi
> > >
> > > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the
> > > > current
> > > > value locally, so there is no need to have GET_CONTROL.
> > > >
> > > > The only exception is the initial (default) control value in the device.
> > > > But with the removal of 'function' and with addition of 'caps' instead,
> > > > the way to provide defaults is gone. So I suppose for formats we'll be
> > > > just using GET on driver start to get the 'defaults'. But the thing is
> > > > that for formats there are other uses for GET, but for controls GET
> > > > apparently does not make to much sense at runtime.
> > > >
> > > > > > Also, shouldn't the profile come together with level? Would make sense
> > > > > > for
> > > > > > encoders.
> > > > >
> > > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> > > > > required when querying supported levels.
> > > >
> > > > So probably should be enumerated together, not queried, as per the comment
> > > > above.
> > > >
> > > > > > > +\item[\field{mask}] is a bitset that represents the supported
> > > > > > > +  combination of input and output format. If \textit{i}-th bit is
> > > > > > > set
> > > > > > > +  in \field{mask} of \textit{j}-th \field{struct
> > > > > > > +  virtio_video_format_desc} for input, the device supports encoding
> > > > > > > or
> > > > > > > +  decoding from the \textit{j}-th input format to \textit{i}-th
> > > > > > > output
> > > > > > > +  format.
> > > > > > > +\item[\field{width, height}] represents a range of resolutions
> > > > > > > +  supported by the device. If its \field{step} is not applicable,
> > > > > > > its
> > > > > > > +  \field{min} is equal to its \field{max}.
> > > > > > > +\item[\field{num_rates}] is the length of an array
> > > > > > > \field{frame_rates}.
> > > > > > > In
> > > > > > > case of decoder, the driver SHOULD ignore this value.
> > > > > > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > >
> > > > > > I'd guess frame rates depend on the resolution as well. This
> > > > > > dependency
> > > > > > was
> > > > > > clear in the v1 spec, but in the v2 there is no dependency anymore. I
> > > > > > think we need to update this.
> > > > >
> > > > > That's a good point. I missed that dependency when updating the
> > > > > structures.
> > > > > So, let me update the structs like the following:
> > > > >
> > > > > struct virtio_video_format_frame {
> > > > >
> > > > >         struct virtio_video_format_range width;
> > > > >         struct virtio_video_format_range height;
> > > > >         le32 num_rates;
> > > > >         u8 padding[4];
> > > > >         /* Followed by struct virtio_video_format_range frame_rates[] */
> > > > >
> > > > > };
> > > > >
> > > > > struct virtio_video_format_desc {
> > > > >
> > > > >         le64 mask;
> > > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > > >         le32 num_frames;
> > > > >         u8 padding[4];
> > > > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > > >
> > > > > };
> > > >
> > > > Yes, I do agree with this approach.
> > > >
> > > > Best regards,
> > > > Dmitry.
> > > >
> > > > > Best regards,
> > > > > Keiichi.
> > > > >
> > > > > > Best regards,
> > > > > > Dmitry.
> > > > > >
> > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > > > > > +  within the device.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +enum virtio_video_mem_type {
> > > > > > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_stream_create {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > > > */
> > > > > > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > > > */
> > > > > > > +        char debug_name[64];
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > > +virtio_video_mem_type}.
> > > > > > > +\item[\field{out_mem_type}] is a type of buffer management for
> > > > > > > output
> > > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > > +virtio_video_mem_type}.
> > > > > > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream
> > > > > > > (context)
> > > > > > > +  within the device.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_stream_destroy {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > > > > > +  queued buffers through the pipeline.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_stream_drain {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > > > > > +  within the device.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_mem_entry {
> > > > > > > +        le64 addr;
> > > > > > > +        le32 length;
> > > > > > > +        u8 padding[4];
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_resource_create {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 resource_id;
> > > > > > > +        le32 nr_entries;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > +\item[\field{nr_entries}] number of \field{struct
> > > > > > > +  virtio_video_mem_entry} memory entries.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource
> > > > > > > descriptor
> > > > > > > +  within the device.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_resource_destroy {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 resource_id;
> > > > > > > +        u8 padding[4];
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > > > > > +queue.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > > > > > +
> > > > > > > +struct virtio_video_resource_queue {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 buf_type;
> > > > > > > +        le32 resource_id;
> > > > > > > +        le64 timestamp;
> > > > > > > +        le32 nr_data_size;
> > > > > > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{buf_type}] buf_type of the .
> > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > > > used
> > > > > > > +  for synchronisation.
> > > > > > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > > > > > +\item[\field{data_size}] number of data bytes within a plane.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +enum virtio_video_buffer_flag {
> > > > > > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > > > > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > > > > > +        /* Encoder only */
> > > > > > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > > > > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > > > > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_resource_queue_resp {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le64 timestamp;
> > > > > > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > > > > > +        le32 size;  /* Encoded size */
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > > > used
> > > > > > > +  for synchronisation.
> > > > > > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > > > > > +\item[\field{size}] data size in the buffer (encoder only).
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +The device sends a response to the queue request asynchronously
> > > > > > > when
> > > > > > > +it has finished processing the buffer.
> > > > > > > +
> > > > > > > +The device SHOULD mark a buffer that triggered a processing error
> > > > > > > with
> > > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > > > > > +
> > > > > > > +The device MUST mark the last buffer with the
> > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > > > +sequence.
> > > > > > > +
> > > > > > > +In case of encoder, to denote a particular frame type the devie
> > > > > > > MUST
> > > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > > > > > +  buffers back from the input or the output queue of the device.
> > > > > > > The
> > > > > > > +  device SHOULD return all of the buffers from the respective queue
> > > > > > > as
> > > > > > > +  soon as possible without pushing the buffers through the
> > > > > > > processing
> > > > > > > +  pipeline.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_queue_clear {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 buf_type;
> > > > > > > +        u8 padding[4];
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > > > > > +  output of a stream.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_plane_format {
> > > > > > > +        le32 plane_size;
> > > > > > > +        le32 stride;
> > > > > > > +        u8 padding[4];
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{plane_size}] size of the plane in bytes.
> > > > > > > +\item[\field{stride}] stride used for the plane in bytes.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_params {
> > > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > > > > > +        le32 frame_width;
> > > > > > > +        le32 frame_height;
> > > > > > > +        le32 min_buffers;
> > > > > > > +        le32 max_buffers;
> > > > > > > +        le32 frame_rate;
> > > > > > > +        struct virtio_video_crop {
> > > > > > > +                le32 left;
> > > > > > > +                le32 top;
> > > > > > > +                le32 width;
> > > > > > > +                le32 height;
> > > > > > > +        } crop;
> > > > > > > +        le32 num_planes;
> > > > > > > +        struct virtio_video_plane_format
> > > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{frame_width}] the value to get/set.
> > > > > > > +\item[\field{frame_height}] the value to get/set.
> > > > > > > +\item[\field{pixel_format}] the value to get/set.
> > > > > > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > > > > > +  format (r/o).
> > > > > > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > > > > > +  format (r/o).
> > > > > > > +\item[\field{frame_rate}] the value to get/set.
> > > > > > > +\item[\field{crop}] cropping (composing) rectangle.
> > > > > > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > > > > > +(r/o).
> > > > > > > +\item[\field{plane_formats}] description of each plane.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_get_params {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_get_params_resp {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        struct virtio_video_params params;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > > +\item[\field{params}] parameter values.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > > > > > +
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +struct virtio_video_set_params {
> > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > +        struct virtio_video_params params;
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{params}] parameters to set.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +Setting stream parameters might have side effects within the
> > > > > > > device.
> > > > > > > +For example, the device MAY perform alignment of width and height,
> > > > > > > +change the number of planes it uses for the format, or do whatever
> > > > > > > +changes that are required to continue normal operation using the
> > > > > > > +updated parameters. It is up to the driver to check the parameter
> > > > > > > set
> > > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > > > > > +
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +\subsubsection{Device Operation: eventq}
> > > > > > > +
> > > > > > > +The device can report events on the event queue. The driver
> > > > > > > initially
> > > > > > > +populates the queue with device-writeable buffers. When the device
> > > > > > > +needs to report an event, it fills a buffer and notifies the
> > > > > > > driver.
> > > > > > > +The driver consumes the report and adds a new buffer to the
> > > > > > > virtqueue.
> > > > > > > +
> > > > > > > +\begin{lstlisting}
> > > > > > > +enum virtio_video_event_type {
> > > > > > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > > > > > +        /* For all functions */
> > > > > > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > > > > > +        /* For decoder only */
> > > > > > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > > > > > +};
> > > > > > > +
> > > > > > > +struct virtio_video_event {
> > > > > > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > > > > > +        le32 stream_id;
> > > > > > > +        u8 padding[4];
> > > > > > > +};
> > > > > > > +\end{lstlisting}
> > > > > > > +
> > > > > > > +\begin{description}
> > > > > > > +\item[\field{event_type}] type of the triggered event .
> > > > > > > +\item[\field{stream_id}] id of the source stream.
> > > > > > > +\end{description}
> > > > > > > +
> > > > > > > +The device MUST send
> > > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > > > > > +whenever it encounters new resolution data in the stream. This
> > > > > > > +includes the case of the initial device configuration after
> > > > > > > metadata
> > > > > > > +has been parsed and the case of dynamic resolution change.
> >
> >
Keiichi Watanabe Jan. 8, 2020, 1:05 p.m. UTC | #34
On Wed, Jan 8, 2020 at 9:46 PM Tomasz Figa <tfiga@chromium.org> wrote:
>
> On Wed, Jan 8, 2020 at 9:15 PM Keiichi Watanabe <keiichiw@chromium.org> wrote:
> >
> > Hi Dmitry,
> >
> > On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote:
> > >
> > > Hi Keiichi,
> > >
> > > On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote:
> > > > Hi Dmitry,
> > > >
> > > > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > wrote:
> > > > > Hi Keiichi,
> > > > >
> > > > > thanks for the updates, please see my comments below.
> > > > >
> > > > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> > > > > > Hi Dmitry,
> > > > > >
> > > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp
> > > > > > <dmitry.sepp@opensynergy.com>
> > > > >
> > > > > wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > a couple of new comments:
> > > > > > >
> > > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote:
> > > > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > > >
> > > > > > > > The virtio video encoder device and decoder device provide
> > > > > > > > functionalities
> > > > > > > > to encode and decode video stream respectively.
> > > > > > > > Though video encoder and decoder are provided as different devices,
> > > > > > > > they
> > > > > > > > use a same protocol.
> > > > > > > >
> > > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > > > > > > ---
> > > > > > > >
> > > > > > > >  content.tex      |   1 +
> > > > > > > >  virtio-video.tex | 579
> > > > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > >  2 files changed, 580 insertions(+)
> > > > > > > >  create mode 100644 virtio-video.tex
> > > > > > > >
> > > > > > > > diff --git a/content.tex b/content.tex
> > > > > > > > index 556b373..9e56839 100644
> > > > > > > > --- a/content.tex
> > > > > > > > +++ b/content.tex
> > > > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > > > > > >
> > > > > > > >  \input{virtio-fs.tex}
> > > > > > > >  \input{virtio-rpmb.tex}
> > > > > > > >
> > > > > > > > +\input{virtio-video.tex}
> > > > > > > >
> > > > > > > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > > > > > > >
> > > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > > > > > > new file mode 100644
> > > > > > > > index 0000000..30e728d
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/virtio-video.tex
> > > > > > > > @@ -0,0 +1,579 @@
> > > > > > > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > > > > > > +
> > > > > > > > +The virtio video encoder device and decoder device are virtual
> > > > > > > > devices
> > > > > > > > that +supports encoding and decoding respectively. Though the
> > > > > > > > encoder
> > > > > > > > and the decoder +are different devices, they use the same protocol.
> > > > > > > > +
> > > > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device /
> > > > > > > > Device
> > > > > > > > ID}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[30] encoder device
> > > > > > > > +\item[31] decoder device
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device /
> > > > > > > > Virtqueues} +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[0] controlq - queue for sending control commands.
> > > > > > > > +\item[1] eventq - queue for sending events happened in the device.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device /
> > > > > > > > Feature
> > > > > > > > bits} +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be
> > > > > > > > used
> > > > > > > > for
> > > > > > > > video +  buffers.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video
> > > > > > > > Device
> > > > > > > > / Feature bits} +
> > > > > > > > +The device MUST offer at least one of feature bits.
> > > > > > > > +
> > > > > > > > +\subsection{Device configuration layout}\label{sec:Device Types /
> > > > > > > > Video
> > > > > > > > Device / Device configuration layout} +
> > > > > > > > +Video device configuration uses the following layout structure:
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_config {
> > > > > > > > +        le32 max_cap_len;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{max_cap_len}] defines the maximum length of a
> > > > > > > > descriptor
> > > > > > > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device
> > > > > > > > +  MUST set this value.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsection{Device Initialization}\label{sec:Device Types / Video
> > > > > > > > Device /
> > > > > > > > Device Initialization} +
> > > > > > > > +\devicenormative{\subsubsection}{Device Initialization}{Device
> > > > > > > > Types /
> > > > > > > > Video Device / Device Initialization} +
> > > > > > > > +The driver SHOULD query device capability by using the
> > > > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the
> > > > > > > > initial
> > > > > > > > +setup.
> > > > > > > > +
> > > > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device
> > > > > > > > /
> > > > > > > > Device Operation} +
> > > > > > > > +The driver allocates input and output buffers and queues the
> > > > > > > > buffers
> > > > > > > > +to the device. The device performs operations on the buffers
> > > > > > > > according
> > > > > > > > +to the function in question.
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Create stream}
> > > > > > > > +
> > > > > > > > +To process buffers, the device needs to associate them with a
> > > > > > > > certain
> > > > > > > > +video stream (essentially, a context). Streams are created by
> > > > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > > > > > > +determined by the device.
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Create buffers}
> > > > > > > > +
> > > > > > > > +Buffers are used to store the actual data as well as the relevant
> > > > > > > > +metadata. Scatter lists are supported, so the buffer doesn't need
> > > > > > > > to
> > > > > > > > +be contiguous in guest physical memory.
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video
> > > > > > > > +  resource that is backed by a buffer allocated from the driver's
> > > > > > > > +  memory.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource
> > > > > > > > that
> > > > > > > > +  is no longer needed.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Stream parameter control}
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > > > > > > parameters
> > > > > > > > for +  input and output streams from the device.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream
> > > > > > > > parameters to
> > > > > > > > the +  device.
> > > > > > > > +\item After setting stream parameters, the driver may issue
> > > > > > > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and
> > > > > > > > output
> > > > > > > > can be +  changed implicitly by the device during the set operation.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Process buffers}
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item If the function and the buffer type require so, write data to
> > > > > > > > +the buffer memory.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for
> > > > > > > > +processing in the device.
> > > > > > > > +\item The request completes asynchronously when the device has
> > > > > > > > +finished with the buffer.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Buffer processing control}
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process
> > > > > > > > and
> > > > > > > > +  return all of the already queued buffers.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return
> > > > > > > > back
> > > > > > > > +  already queued buffers from the input or the output queue. This
> > > > > > > > also
> > > > > > > > +  includes input or output buffers that can be currently owned by
> > > > > > > > the
> > > > > > > > +  device's processing pipeline.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Asynchronous events}
> > > > > > > > +
> > > > > > > > +While processing buffers, the device can send asynchronous event
> > > > > > > > +notifications to the driver. The behaviour depends on the exact
> > > > > > > > +stream. For example, the decoder device sends a resolution change
> > > > > > > > +event when it encounters new resolution metadata in the stream.
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Request header}
> > > > > > > > +
> > > > > > > > +All requests and responses on the control virt queue have a fixed
> > > > > > > > +header using the following layout structure and definitions:
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_ctrl_type {
> > > > > > > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > > > > > > +
> > > > > > > > +        /* request */
> > > > > > > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > > > > > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > > > > > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > > > > > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > > > > > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > > > > > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > > > > > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > > > > > > +
> > > > > > > > +        /* response */
> > > > > > > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > > > > > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > > > > > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > > > > > > +
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_ctrl_hdr {
> > > > > > > > +        le32 type;
> > > > > > > > +        le32 stream_id;
> > > > > > > > +        le32 len; /* Length of the structure in bytes. */
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{type}] is the type of the driver request or the device
> > > > > > > > +response.
> > > > > > > > +\item[\field{stream_id}] specifies a target stream.
> > > > > > > > +\item[\field{len}] is the length of data in bytes, which includes
> > > > > > > > +length of the header.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: controlq}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about
> > > > > > > > +supported formats.
> > > > > > > > +
> > > > > > > > +The driver uses \field{struct virtio_video_get_capability} to send
> > > > > > > > a
> > > > > > > > +query request.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_buf_type {
> > > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_get_capability {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        enum virtio_video_buf_type buf_type;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{buf_type}] is the buffer type that the driver asks
> > > > > > > > +information about. The driver MUST set either
> > > > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > > > > > > +
> > > > > > > > +The device responds a capability by using \field{struct
> > > > > > > > +virtio_video_get_capability_resp}.
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_format {
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > > > > > +        /* Raw formats */
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > > > > > > +
> > > > > > > > +        /* Compressed formats */
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +enum virtio_video_profile {
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > > > > > > +
> > > > > > > > +        /* H.264 */
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > > > > > > +        /* VP8 */
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > > > > > > +
> > > > > > > > +        /* VP9 */
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_format_range {
> > > > > > > > +        le32 min;
> > > > > > > > +        le32 max;
> > > > > > > > +        le32 step;
> > > > > > > > +        u8 paddings[4];
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_format_desc {
> > > > > > > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > > > > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */
> > > > > > > > +        le64 mask;
> > > > > > > > +        struct virtio_video_format_range width;
> > > > > > > > +        struct virtio_video_format_range height;
> > > > > > > > +        le32 num_rates;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +        /* Followed by struct virtio_video_frame_rate frame_rates[]
> > > > > > > > */
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_get_capability_resp {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 num_descs;
> > > > > > > > +        /* Followed by struct virtio_video_format_desc desc[] */
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +The format description \field{struct virtio_video_format_desc}
> > > > > > > > +includes the following fields:
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{format}] specifies an image format. The device MUST
> > > > > > > > set
> > > > > > > > one
> > > > > > > > +  of \field{enum virtio_video_format}.
> > > > > > > > +\item[\field{profile}] specifies a profile of the compressed image
> > > > > > > > format
> > > > > > > > +  specified in \field{format}. The driver SHOULD ignore this value
> > > > > > > > if
> > > > > > > > +  \field{format} is a raw format.
> > > > > > >
> > > > > > > So how should this be used? The spec does not define any way to set
> > > > > > > profile for the device. It is very important for encoder.
> > > > > >
> > > > > > Thank you for pointing this.
> > > > > > These points are overlooked, as I didn't care about encoder usage
> > > > > > enough.
> > > > > >
> > > > > > After thinking it again, I think it's not a very good idea to include
> > > > > > supported profiles and levels in a struct for capability.
> > > > > > This is because these values are available only for limited number of
> > > > > > formats. Also, it's true that we need to have a way to set these values
> > > > > > as
> > > > > > Dmitry pointed.
> > > > >
> > > > > Yes, you are right. In fact, the approach of the v1 spec to keep controls
> > > > > separately was not correct.
> > > > >
> > > > > > Instead, it would make more sense to have additional three types of
> > > > > > controls for profiles, levels, and bitrates:
> > > > > > (1) QUERY_CONTROL: Query values supported by the device
> > > > > > (2) GET_CONTROL: Read a value that is set in the device
> > > > > > (3) SET_CONTROL: Set a value in the device
> > > > > >
> > > > > > These operations are similar to V4L2 controls.
> > > > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and
> > > > > > G_CTRL in V4L2, respectively.
> > > > > > Also, (3) would be similar to enum virtio_video_control_type in the
> > > > > > virtio-video v1 driver implementation in
> > > > > > https://markmail.org/message/dwghwdqsbl3gsjxu .
> > > > > >
> > > > > > For QUERY_CONTROL, my idea is like this:
> > > > > >
> > > > > > enum virtio_video_control_type {
> > > > > >
> > > > > >   VIRTIO_VIDEO_CONTROL_UNDEFINED = 0,
> > > > > >
> > > > > >   VIRTIO_VIDEO_CONTROL_BITRATE = 0x100,
> > > > > >   VIRTIO_VIDEO_CONTROL_PROFILE,
> > > > > >   VIRTIO_VIDEO_CONTROL_LEVEL,
> > > > > >
> > > > > > };
> > > > > >
> > > > > > struct virtio_video_query_control {
> > > > > >
> > > > > >   struct virtio_video_ctrl_hdr hdr;
> > > > > >   le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> > > > > >   le32 length;
> > > > > >   /* Followed by additional data.
> > > > > >
> > > > > >    * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE,
> > > > > >    * the device must pass a codec format like H264 or VP9.
> > > > > >    * The requred data must be defined in the specification.
> > > > > >    */
> > > > > >
> > > > > > };
> > > > > >
> > > > > > struct virtio_video_query_control_resp {
> > > > > >
> > > > > >   struct virtio_video_ctrl_hdr hdr;
> > > > > >   le32 length;
> > > > > >   u8 padding[4];
> > > > > >   /* Followed by data corresponds to the specified control.
> > > > > >
> > > > > >    * The type of data must be defined in the spec.
> > > > > >    * For example, if the driver queries profiles, this part should be
> > > > > >    * an array of supported profiles of a given format.
> > > > > >    */
> > > > > >
> > > > > > };
> > > > > >
> > > > > > WDYT?
> > > > >
> > > > > I think virtio_video_control_type should make sense. But I would disagree
> > > > > with the need to have new QUERY_CONTROL and GET_CONTROL.
> > > > >
> > > > > I assume the set of supported controls is fixed for some particular format
> > > > > on a given IP. So we'd propose to include controls into format
> > > > > descriptors, so we don't need to QUERY_CONTROL. This way (with
> > > > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. for
> > > > > profiles.
> > > > >
> > > > > struct virtio_video_format_frame {
> > > > >
> > > > >         /* As proposed in Keiichi's prev email */
> > > > >
> > > > > };
> > > > >
> > > > > virtio_video_format_list {
> > > > >
> > > > >         le32 num_entries;
> > > > >         u8 padding[4];
> > > > >         /* Followed by le64 entries[] */
> > > > >
> > > > > };
> > > > >
> > > > > struct virtio_video_format_control {
> > > > >
> > > > >         le32 type;
> > > > >         u8 padding[4];
> > > > >         struct virtio_video_format_list values;
> > > > >
> > > > > };
> > > > >
> > > > > struct virtio_video_format_desc {
> > > > >
> > > > >         le64 mask;
> > > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > > >         le32 num_frames;
> > > > >         le32 num_controls;
> > > > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > > >         /* Followed by struct virtio_video_format_control controls[] */
> > > > >
> > > > > };
> > > >
> > > > In my understanding, a set of supported levels depends on a profile.
> > > > H.264 spec says that levels are specified within each profile.
> > > > cf. "0.5 Profiles and levels" in
> > > > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU doesn't
> > > > seem to provide a way to query levels for each profile properly, though)
> > > >
> > > > So, if we want to have supported profiles and levels in format_desc as
> > > > your idea, each supported profile should have a list of supported
> > > > levels.
> > > > I suppose it makes the structure of video_format_desc too complicated.
> > > > I'd like to avoid this complexity caused by some specific formats.
> > > >
> > > > Instead, I'd like to keep virtio_video_format_desc minimal and have
> > > > QUERY_CONTROL to query format-specific values like profiles and
> > > > levels.
> > > > This design would make it easy to extend controls when we want to
> > > > support new types of formats. We will just need to define new
> > > > VIRTIO_VIDEO_CONTROL_*.
> > > > What do you think?
> > >
> > > Yes, I think we can keep it this way, this indeed provides more flexibility and
> > > this way we won't need to modify the 'format' parsing logic when more controls
> > > are added.
> >
> > Thanks! I'm preparing the next version of the patch.
> >
> > I have one concern about terminology.
> > We have two different things called "controls" now:
> > 1) Messages passed via controlq. i.e. "ctrl" in 'enum
> > virtio_video_ctrl_type' and 'virtio_video_ctrl_hdr'.
> > 2) Commands for profiles, levels, and bitrates similar to V4L2
> > controls. e.g.., "CONTROL" in VIRTIO_VIDEO_CONTROL_*
> >
> > I feel calling both "controls" is somewhat confusing and I am
> > wondering if we can't rename either of them.
> >
> > For example, how about renaming 2) to "command"? Then, we'll have
> > {QUERY ,GET, SET}_COMMAND.
> > It may sound strange, though.
> > Do you have any idea?
>
> "Command" sound like executing an action, but in this context we're
> thinking more about parameters or, exactly, controls. The term
> "control" is not limited to V4L2 and actually used in other subsystems
> as well, for example the Android Camera HAL.
>
> How about renaming 1) to "commandq" and then having the other symbols
> use the abbreviated "cmd"?

Thanks! Keep calling 2) "control" and renaming 1) sounds better now.
Actually, virtio-gpu's requests are called "command". (The queue name
is "controlq", though)

One concern is that the name "controlq" is commonly used among some
virtio devices such as virtio-net and virtio-gpu. However, it won't
make more sense to stick to using the same queue name as other virtio
protocols.

Keiichi

>
> Best regards,
> Tomasz
>
> >
> > Best regards,
> > Keiichi
> >
> > >
> > > Best regards,
> > > Dmitry.
> > >
> > > >
> > > > Best regards,
> > > > Keiichi
> > > >
> > > > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the
> > > > > current
> > > > > value locally, so there is no need to have GET_CONTROL.
> > > > >
> > > > > The only exception is the initial (default) control value in the device.
> > > > > But with the removal of 'function' and with addition of 'caps' instead,
> > > > > the way to provide defaults is gone. So I suppose for formats we'll be
> > > > > just using GET on driver start to get the 'defaults'. But the thing is
> > > > > that for formats there are other uses for GET, but for controls GET
> > > > > apparently does not make to much sense at runtime.
> > > > >
> > > > > > > Also, shouldn't the profile come together with level? Would make sense
> > > > > > > for
> > > > > > > encoders.
> > > > > >
> > > > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be
> > > > > > required when querying supported levels.
> > > > >
> > > > > So probably should be enumerated together, not queried, as per the comment
> > > > > above.
> > > > >
> > > > > > > > +\item[\field{mask}] is a bitset that represents the supported
> > > > > > > > +  combination of input and output format. If \textit{i}-th bit is
> > > > > > > > set
> > > > > > > > +  in \field{mask} of \textit{j}-th \field{struct
> > > > > > > > +  virtio_video_format_desc} for input, the device supports encoding
> > > > > > > > or
> > > > > > > > +  decoding from the \textit{j}-th input format to \textit{i}-th
> > > > > > > > output
> > > > > > > > +  format.
> > > > > > > > +\item[\field{width, height}] represents a range of resolutions
> > > > > > > > +  supported by the device. If its \field{step} is not applicable,
> > > > > > > > its
> > > > > > > > +  \field{min} is equal to its \field{max}.
> > > > > > > > +\item[\field{num_rates}] is the length of an array
> > > > > > > > \field{frame_rates}.
> > > > > > > > In
> > > > > > > > case of decoder, the driver SHOULD ignore this value.
> > > > > > > > +\item[\field{frame_rates}] is an array of supported frame rates.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > >
> > > > > > > I'd guess frame rates depend on the resolution as well. This
> > > > > > > dependency
> > > > > > > was
> > > > > > > clear in the v1 spec, but in the v2 there is no dependency anymore. I
> > > > > > > think we need to update this.
> > > > > >
> > > > > > That's a good point. I missed that dependency when updating the
> > > > > > structures.
> > > > > > So, let me update the structs like the following:
> > > > > >
> > > > > > struct virtio_video_format_frame {
> > > > > >
> > > > > >         struct virtio_video_format_range width;
> > > > > >         struct virtio_video_format_range height;
> > > > > >         le32 num_rates;
> > > > > >         u8 padding[4];
> > > > > >         /* Followed by struct virtio_video_format_range frame_rates[] */
> > > > > >
> > > > > > };
> > > > > >
> > > > > > struct virtio_video_format_desc {
> > > > > >
> > > > > >         le64 mask;
> > > > > >         le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> > > > > >         le32 planes_layout; /* See the thread [v2 0/1] */
> > > > > >         le32 num_frames;
> > > > > >         u8 padding[4];
> > > > > >         /* Followed by struct virtio_video_format_frame frames[] */
> > > > > >
> > > > > > };
> > > > >
> > > > > Yes, I do agree with this approach.
> > > > >
> > > > > Best regards,
> > > > > Dmitry.
> > > > >
> > > > > > Best regards,
> > > > > > Keiichi.
> > > > > >
> > > > > > > Best regards,
> > > > > > > Dmitry.
> > > > > > >
> > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context)
> > > > > > > > +  within the device.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_mem_type {
> > > > > > > > +        VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_stream_create {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 in_mem_type;  /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > > > > */
> > > > > > > > +        le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types
> > > > > > > > */
> > > > > > > > +        char debug_name[64];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{in_mem_type}] is a type of buffer management for input
> > > > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > > > +virtio_video_mem_type}.
> > > > > > > > +\item[\field{out_mem_type}] is a type of buffer management for
> > > > > > > > output
> > > > > > > > +buffers. The driver MUST set a value in \field{enum
> > > > > > > > +virtio_video_mem_type}.
> > > > > > > > +\item[\field{debug_name}] is a text string for a debug purpose.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream
> > > > > > > > (context)
> > > > > > > > +  within the device.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_stream_destroy {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the
> > > > > > > > +  queued buffers through the pipeline.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_stream_drain {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor
> > > > > > > > +  within the device.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_mem_entry {
> > > > > > > > +        le64 addr;
> > > > > > > > +        le32 length;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_resource_create {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 resource_id;
> > > > > > > > +        le32 nr_entries;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > > +\item[\field{nr_entries}] number of \field{struct
> > > > > > > > +  virtio_video_mem_entry} memory entries.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource
> > > > > > > > descriptor
> > > > > > > > +  within the device.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_resource_destroy {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 resource_id;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's
> > > > > > > > +queue.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8
> > > > > > > > +
> > > > > > > > +struct virtio_video_resource_queue {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 buf_type;
> > > > > > > > +        le32 resource_id;
> > > > > > > > +        le64 timestamp;
> > > > > > > > +        le32 nr_data_size;
> > > > > > > > +        le32 data_size[VIRTIO_VIDEO_MAX_PLANES];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{buf_type}] buf_type of the .
> > > > > > > > +\item[\field{resource_id}] internal id of the resource.
> > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > > > > used
> > > > > > > > +  for synchronisation.
> > > > > > > > +\item[\field{nr_data_size}] number of \field{data_size} entries.
> > > > > > > > +\item[\field{data_size}] number of data bytes within a plane.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_buffer_flag {
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_F_ERR        = 0x0001,
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_F_EOS        = 0x0002,
> > > > > > > > +        /* Encoder only */
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_IFRAME        = 0x0004,
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_PFRAME        = 0x0008,
> > > > > > > > +        VIRTIO_VIDEO_BUFFER_BFRAME        = 0x0010,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_resource_queue_resp {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le64 timestamp;
> > > > > > > > +        le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */
> > > > > > > > +        le32 size;  /* Encoded size */
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be
> > > > > > > > used
> > > > > > > > +  for synchronisation.
> > > > > > > > +\item[\field{flags}] mark specific buffers in the sequence.
> > > > > > > > +\item[\field{size}] data size in the buffer (encoder only).
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +The device sends a response to the queue request asynchronously
> > > > > > > > when
> > > > > > > > +it has finished processing the buffer.
> > > > > > > > +
> > > > > > > > +The device SHOULD mark a buffer that triggered a processing error
> > > > > > > > with
> > > > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag.
> > > > > > > > +
> > > > > > > > +The device MUST mark the last buffer with the
> > > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain
> > > > > > > > +sequence.
> > > > > > > > +
> > > > > > > > +In case of encoder, to denote a particular frame type the devie
> > > > > > > > MUST
> > > > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME,
> > > > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME.
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued
> > > > > > > > +  buffers back from the input or the output queue of the device.
> > > > > > > > The
> > > > > > > > +  device SHOULD return all of the buffers from the respective queue
> > > > > > > > as
> > > > > > > > +  soon as possible without pushing the buffers through the
> > > > > > > > processing
> > > > > > > > +  pipeline.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_queue_clear {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 buf_type;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the
> > > > > > > > +  output of a stream.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_plane_format {
> > > > > > > > +        le32 plane_size;
> > > > > > > > +        le32 stride;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{plane_size}] size of the plane in bytes.
> > > > > > > > +\item[\field{stride}] stride used for the plane in bytes.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_params {
> > > > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > > > > +        le32 fourcc;   /* One of VIRTIO_VIDEO_FOURCC_* types */
> > > > > > > > +        le32 frame_width;
> > > > > > > > +        le32 frame_height;
> > > > > > > > +        le32 min_buffers;
> > > > > > > > +        le32 max_buffers;
> > > > > > > > +        le32 frame_rate;
> > > > > > > > +        struct virtio_video_crop {
> > > > > > > > +                le32 left;
> > > > > > > > +                le32 top;
> > > > > > > > +                le32 width;
> > > > > > > > +                le32 height;
> > > > > > > > +        } crop;
> > > > > > > > +        le32 num_planes;
> > > > > > > > +        struct virtio_video_plane_format
> > > > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{frame_width}] the value to get/set.
> > > > > > > > +\item[\field{frame_height}] the value to get/set.
> > > > > > > > +\item[\field{pixel_format}] the value to get/set.
> > > > > > > > +\item[\field{min_buffers}] minimum buffers required to handle the
> > > > > > > > +  format (r/o).
> > > > > > > > +\item[\field{max_buffers}] maximum buffers required to handle the
> > > > > > > > +  format (r/o).
> > > > > > > > +\item[\field{frame_rate}] the value to get/set.
> > > > > > > > +\item[\field{crop}] cropping (composing) rectangle.
> > > > > > > > +\item[\field{num_planes}] number of planes used to store pixel data
> > > > > > > > +(r/o).
> > > > > > > > +\item[\field{plane_formats}] description of each plane.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_get_params {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_get_params_resp {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        struct virtio_video_params params;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{buf_type}] buffer type.
> > > > > > > > +\item[\field{params}] parameter values.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream.
> > > > > > > > +
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_set_params {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        struct virtio_video_params params;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{params}] parameters to set.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +Setting stream parameters might have side effects within the
> > > > > > > > device.
> > > > > > > > +For example, the device MAY perform alignment of width and height,
> > > > > > > > +change the number of planes it uses for the format, or do whatever
> > > > > > > > +changes that are required to continue normal operation using the
> > > > > > > > +updated parameters. It is up to the driver to check the parameter
> > > > > > > > set
> > > > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued.
> > > > > > > > +
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: eventq}
> > > > > > > > +
> > > > > > > > +The device can report events on the event queue. The driver
> > > > > > > > initially
> > > > > > > > +populates the queue with device-writeable buffers. When the device
> > > > > > > > +needs to report an event, it fills a buffer and notifies the
> > > > > > > > driver.
> > > > > > > > +The driver consumes the report and adds a new buffer to the
> > > > > > > > virtqueue.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_event_type {
> > > > > > > > +        VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0,
> > > > > > > > +        /* For all functions */
> > > > > > > > +        VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100,
> > > > > > > > +        /* For decoder only */
> > > > > > > > +        VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_event {
> > > > > > > > +        le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */
> > > > > > > > +        le32 stream_id;
> > > > > > > > +        u8 padding[4];
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{event_type}] type of the triggered event .
> > > > > > > > +\item[\field{stream_id}] id of the source stream.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +The device MUST send
> > > > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED
> > > > > > > > +whenever it encounters new resolution data in the stream. This
> > > > > > > > +includes the case of the initial device configuration after
> > > > > > > > metadata
> > > > > > > > +has been parsed and the case of dynamic resolution change.
> > >
> > >
Dmitry Sepp Jan. 8, 2020, 1:11 p.m. UTC | #35
Hi Tomasz, Keiichi,

On Mittwoch, 8. Januar 2020 13:46:25 CET Tomasz Figa wrote:
> On Wed, Jan 8, 2020 at 9:15 PM Keiichi Watanabe <keiichiw@chromium.org> 
wrote:
> > Hi Dmitry,
> > 
> > On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> 
wrote:
> > > Hi Keiichi,
> > > 
> > > On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote:
> > > > Hi Dmitry,
> > > > 
> > > > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp
> > > > <dmitry.sepp@opensynergy.com>
> > > 
> > > wrote:
> > > > > Hi Keiichi,
> > > > > 
> > > > > thanks for the updates, please see my comments below.
> > > > > 
> > > > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote:
> > > > > > Hi Dmitry,
> > > > > > 
> > > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp
> > > > > > <dmitry.sepp@opensynergy.com>
> > > > > 
> > > > > wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > a couple of new comments:
> > > > > > > 
> > > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe 
wrote:
> > > > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > > > 
> > > > > > > > The virtio video encoder device and decoder device provide
> > > > > > > > functionalities
> > > > > > > > to encode and decode video stream respectively.
> > > > > > > > Though video encoder and decoder are provided as different
> > > > > > > > devices,
> > > > > > > > they
> > > > > > > > use a same protocol.
> > > > > > > > 
> > > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com>
> > > > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> > > > > > > > ---
> > > > > > > > 
> > > > > > > >  content.tex      |   1 +
> > > > > > > >  virtio-video.tex | 579
> > > > > > > >  +++++++++++++++++++++++++++++++++++++++++++++++
> > > > > > > >  2 files changed, 580 insertions(+)
> > > > > > > >  create mode 100644 virtio-video.tex
> > > > > > > > 
> > > > > > > > diff --git a/content.tex b/content.tex
> > > > > > > > index 556b373..9e56839 100644
> > > > > > > > --- a/content.tex
> > > > > > > > +++ b/content.tex
> > > > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing
> > > > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex}
> > > > > > > > 
> > > > > > > >  \input{virtio-fs.tex}
> > > > > > > >  \input{virtio-rpmb.tex}
> > > > > > > > 
> > > > > > > > +\input{virtio-video.tex}
> > > > > > > > 
> > > > > > > >  \chapter{Reserved Feature Bits}\label{sec:Reserved Feature
> > > > > > > >  Bits}
> > > > > > > > 
> > > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex
> > > > > > > > new file mode 100644
> > > > > > > > index 0000000..30e728d
> > > > > > > > --- /dev/null
> > > > > > > > +++ b/virtio-video.tex
> > > > > > > > @@ -0,0 +1,579 @@
> > > > > > > > +\section{Video Device}\label{sec:Device Types / Video Device}
> > > > > > > > +
> > > > > > > > +The virtio video encoder device and decoder device are
> > > > > > > > virtual
> > > > > > > > devices
> > > > > > > > that +supports encoding and decoding respectively. Though the
> > > > > > > > encoder
> > > > > > > > and the decoder +are different devices, they use the same
> > > > > > > > protocol.
> > > > > > > > +
> > > > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device
> > > > > > > > /
> > > > > > > > Device
> > > > > > > > ID}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[30] encoder device
> > > > > > > > +\item[31] decoder device
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device
> > > > > > > > /
> > > > > > > > Virtqueues} +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[0] controlq - queue for sending control commands.
> > > > > > > > +\item[1] eventq - queue for sending events happened in the
> > > > > > > > device.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video
> > > > > > > > Device /
> > > > > > > > Feature
> > > > > > > > bits} +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages
> > > > > > > > can be
> > > > > > > > used
> > > > > > > > for
> > > > > > > > video +  buffers.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types /
> > > > > > > > Video
> > > > > > > > Device
> > > > > > > > / Feature bits} +
> > > > > > > > +The device MUST offer at least one of feature bits.
> > > > > > > > +
> > > > > > > > +\subsection{Device configuration layout}\label{sec:Device
> > > > > > > > Types /
> > > > > > > > Video
> > > > > > > > Device / Device configuration layout} +
> > > > > > > > +Video device configuration uses the following layout
> > > > > > > > structure:
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +struct virtio_video_config {
> > > > > > > > +        le32 max_cap_len;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{max_cap_len}] defines the maximum length of a
> > > > > > > > descriptor
> > > > > > > > +  required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The
> > > > > > > > device
> > > > > > > > +  MUST set this value.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsection{Device Initialization}\label{sec:Device Types /
> > > > > > > > Video
> > > > > > > > Device /
> > > > > > > > Device Initialization} +
> > > > > > > > +\devicenormative{\subsubsection}{Device
> > > > > > > > Initialization}{Device
> > > > > > > > Types /
> > > > > > > > Video Device / Device Initialization} +
> > > > > > > > +The driver SHOULD query device capability by using the
> > > > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for
> > > > > > > > the
> > > > > > > > initial
> > > > > > > > +setup.
> > > > > > > > +
> > > > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video
> > > > > > > > Device
> > > > > > > > /
> > > > > > > > Device Operation} +
> > > > > > > > +The driver allocates input and output buffers and queues the
> > > > > > > > buffers
> > > > > > > > +to the device. The device performs operations on the buffers
> > > > > > > > according
> > > > > > > > +to the function in question.
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Create stream}
> > > > > > > > +
> > > > > > > > +To process buffers, the device needs to associate them with a
> > > > > > > > certain
> > > > > > > > +video stream (essentially, a context). Streams are created by
> > > > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters
> > > > > > > > +determined by the device.
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Create buffers}
> > > > > > > > +
> > > > > > > > +Buffers are used to store the actual data as well as the
> > > > > > > > relevant
> > > > > > > > +metadata. Scatter lists are supported, so the buffer doesn't
> > > > > > > > need
> > > > > > > > to
> > > > > > > > +be contiguous in guest physical memory.
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio
> > > > > > > > video
> > > > > > > > +  resource that is backed by a buffer allocated from the
> > > > > > > > driver's
> > > > > > > > +  memory.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a
> > > > > > > > resource
> > > > > > > > that
> > > > > > > > +  is no longer needed.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Stream parameter control}
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream
> > > > > > > > parameters
> > > > > > > > for +  input and output streams from the device.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream
> > > > > > > > parameters to
> > > > > > > > the +  device.
> > > > > > > > +\item After setting stream parameters, the driver may issue
> > > > > > > > +  VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input
> > > > > > > > and
> > > > > > > > output
> > > > > > > > can be +  changed implicitly by the device during the set
> > > > > > > > operation.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Process buffers}
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item If the function and the buffer type require so, write
> > > > > > > > data to
> > > > > > > > +the buffer memory.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer
> > > > > > > > for
> > > > > > > > +processing in the device.
> > > > > > > > +\item The request completes asynchronously when the device
> > > > > > > > has
> > > > > > > > +finished with the buffer.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Buffer processing control}
> > > > > > > > +
> > > > > > > > +\begin{itemize*}
> > > > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to
> > > > > > > > process
> > > > > > > > and
> > > > > > > > +  return all of the already queued buffers.
> > > > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to
> > > > > > > > return
> > > > > > > > back
> > > > > > > > +  already queued buffers from the input or the output queue.
> > > > > > > > This
> > > > > > > > also
> > > > > > > > +  includes input or output buffers that can be currently
> > > > > > > > owned by
> > > > > > > > the
> > > > > > > > +  device's processing pipeline.
> > > > > > > > +\end{itemize*}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Asynchronous events}
> > > > > > > > +
> > > > > > > > +While processing buffers, the device can send asynchronous
> > > > > > > > event
> > > > > > > > +notifications to the driver. The behaviour depends on the
> > > > > > > > exact
> > > > > > > > +stream. For example, the decoder device sends a resolution
> > > > > > > > change
> > > > > > > > +event when it encounters new resolution metadata in the
> > > > > > > > stream.
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: Request header}
> > > > > > > > +
> > > > > > > > +All requests and responses on the control virt queue have a
> > > > > > > > fixed
> > > > > > > > +header using the following layout structure and definitions:
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_ctrl_type {
> > > > > > > > +        VIRTIO_VIDEO_CTRL_UNDEFINED = 0,
> > > > > > > > +
> > > > > > > > +        /* request */
> > > > > > > > +        VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100,
> > > > > > > > +        VIRTIO_VIDEO_T_STREAM_CREATE,
> > > > > > > > +        VIRTIO_VIDEO_T_STREAM_DESTROY,
> > > > > > > > +        VIRTIO_VIDEO_T_STREAM_DRAIN,
> > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_CREATE,
> > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_DESTROY,
> > > > > > > > +        VIRTIO_VIDEO_T_RESOURCE_QUEUE,
> > > > > > > > +        VIRTIO_VIDEO_T_QUEUE_CLEAR,
> > > > > > > > +        VIRTIO_VIDEO_T_SET_PARAMS,
> > > > > > > > +        VIRTIO_VIDEO_T_GET_PARAMS,
> > > > > > > > +
> > > > > > > > +        /* response */
> > > > > > > > +        VIRTIO_VIDEO_S_OK = 0x0200,
> > > > > > > > +        VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE,
> > > > > > > > +        VIRTIO_VIDEO_S_OK_GET_PARAMS,
> > > > > > > > +
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID,
> > > > > > > > +        VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_ctrl_hdr {
> > > > > > > > +        le32 type;
> > > > > > > > +        le32 stream_id;
> > > > > > > > +        le32 len; /* Length of the structure in bytes. */
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{type}] is the type of the driver request or the
> > > > > > > > device
> > > > > > > > +response.
> > > > > > > > +\item[\field{stream_id}] specifies a target stream.
> > > > > > > > +\item[\field{len}] is the length of data in bytes, which
> > > > > > > > includes
> > > > > > > > +length of the header.
> > > > > > > > +\end{description}
> > > > > > > > +
> > > > > > > > +\subsubsection{Device Operation: controlq}
> > > > > > > > +
> > > > > > > > +\begin{description}
> > > > > > > > +
> > > > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information
> > > > > > > > about
> > > > > > > > +supported formats.
> > > > > > > > +
> > > > > > > > +The driver uses \field{struct virtio_video_get_capability} to
> > > > > > > > send
> > > > > > > > a
> > > > > > > > +query request.
> > > > > > > > +
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_buf_type {
> > > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_INPUT,
> > > > > > > > +        VIRTIO_VIDEO_BUF_TYPE_OUTPUT,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_get_capability {
> > > > > > > > +        struct virtio_video_ctrl_hdr hdr;
> > > > > > > > +        enum virtio_video_buf_type buf_type;
> > > > > > > > +};
> > > > > > > > +\end{lstlisting}
> > > > > > > > +\begin{description}
> > > > > > > > +\item[\field{buf_type}] is the buffer type that the driver
> > > > > > > > asks
> > > > > > > > +information about. The driver MUST set either
> > > > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or
> > > > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description}
> > > > > > > > +
> > > > > > > > +The device responds a capability by using \field{struct
> > > > > > > > +virtio_video_get_capability_resp}.
> > > > > > > > +\begin{lstlisting}
> > > > > > > > +enum virtio_video_format {
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_UNDEFINED = 0,
> > > > > > > > +        /* Raw formats */
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_NV12 = 1,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_YUV420,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_YVU420,
> > > > > > > > +
> > > > > > > > +        /* Compressed formats */
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_H264 = 0x1001,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_VP8 =  0x1002,
> > > > > > > > +        VIRTIO_VIDEO_FORMAT_VP9 =  0x1003,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +enum virtio_video_profile {
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_UNDEFINED = 0,
> > > > > > > > +
> > > > > > > > +        /* H.264 */
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_BASELINE =
> > > > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, +
> > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_H264_MAX =
> > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, +
> > > > > > > > +        /* VP8 */
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_ANY =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP8_MAX =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY,
> > > > > > > > +
> > > > > > > > +        /* VP9 */
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > > +        VIRTIO_VIDEO_PROFILE_VP9_MAX =
> > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_format_range {
> > > > > > > > +        le32 min;
> > > > > > > > +        le32 max;
> > > > > > > > +        le32 step;
> > > > > > > > +        u8 paddings[4];
> > > > > > > > +};
> > > > > > > > +
> > > > > > > > +struct virtio_video_format_desc {
> > > > > > > > +        le32 format;  /* One of VIRTIO_VIDEO_FORMAT_* types
> > > > > > > > */
> > > > > > > > +        le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types
> > > > > > > > */
> >