| Message ID | 20191218130214.170703-2-keiichiw@chromium.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | VirtIO video device specification | expand |
Hi, > +The device MUST mark the last buffer with the > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > +sequence. No, that would build a race condition into the protocol. The device could complete the last buffer after the driver has sent the drain command but before the device saw it. So the flag would not be reliable. I also can't see why the flag is needed in the first place. The driver should know which buffers are queued still and be able to figure whenever the drain is complete or not without depending on that flag. So I'd suggest to simply drop it. That is the only issue I've spotted in the protocol on a first quick look. There are a few places where the spec text could be improved. I'll try to set aside some time to go over this in detail, but I can't promise I'll find the time to do that before xmas and new year. cheers, Gerd
On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> wrote: > > Hi, > > > +The device MUST mark the last buffer with the > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > +sequence. > > No, that would build a race condition into the protocol. The device > could complete the last buffer after the driver has sent the drain > command but before the device saw it. So the flag would not be > reliable. > > I also can't see why the flag is needed in the first place. The driver > should know which buffers are queued still and be able to figure > whenever the drain is complete or not without depending on that flag. > So I'd suggest to simply drop it. Unfortunately video decoders are not that simple. There are always going to be some buffers on the decoder side used as reference frames. Only the decoder knows when to release them, as it continues decoding the stream. Moreover, with certain features of certain codecs (e.g. framebuffer reordering in H.264), there could be decoded frames that need to be held by decoder, because later bitstream may contain earlier frames. How we defined this in the V4L2 stateful decoder interface is that if the decoder happened to return the last framebuffer before the drain request arrived, it would return one more, empty. From our experience that was the easiest thing to deal with from both the driver and the application, so I believe the same should apply to the host device implementation and the guest driver. > > That is the only issue I've spotted in the protocol on a first quick > look. There are a few places where the spec text could be improved. > I'll try to set aside some time to go over this in detail, but I can't > promise I'll find the time to do that before xmas and new year. Thanks a lot for review. Best regards, Tomasz
> > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > The virtio video encoder device and decoder device provide functionalities to > encode and decode video stream respectively. > Though video encoder and decoder are provided as different devices, they use > a > same protocol. > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > --- > content.tex | 1 + > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 580 insertions(+) > create mode 100644 virtio-video.tex > > diff --git a/content.tex b/content.tex > index 556b373..9e56839 100644 > --- a/content.tex > +++ b/content.tex > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > Requirements}\label{sec:Device > \input{virtio-vsock.tex} > \input{virtio-fs.tex} > \input{virtio-rpmb.tex} > +\input{virtio-video.tex} > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > diff --git a/virtio-video.tex b/virtio-video.tex > new file mode 100644 > index 0000000..30e728d > --- /dev/null > +++ b/virtio-video.tex > @@ -0,0 +1,579 @@ > +\section{Video Device}\label{sec:Device Types / Video Device} > + > +The virtio video encoder device and decoder device are virtual devices that > +supports encoding and decoding respectively. Though the encoder and the > decoder > +are different devices, they use the same protocol. > + > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID} > + > +\begin{description} > +\item[30] encoder device > +\item[31] decoder device > +\end{description} > + > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues} > + > +\begin{description} > +\item[0] controlq - queue for sending control commands. > +\item[1] eventq - queue for sending events happened in the device. > +\end{description} > + > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature > bits} > + > +\begin{description} > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for > video > + buffers. > +\end{description} > + > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / > Feature bits} > + > +The device MUST offer at least one of feature bits. > + > +\subsection{Device configuration layout}\label{sec:Device Types / Video > Device / Device configuration layout} > + > +Video device configuration uses the following layout structure: > + > +\begin{lstlisting} > +struct virtio_video_config { > + le32 max_cap_len; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > + MUST set this value. > +\end{description} > + > +\subsection{Device Initialization}\label{sec:Device Types / Video Device / > Device Initialization} > + > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / Video > Device / Device Initialization} > + > +The driver SHOULD query device capability by using the > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > +setup. > + > +\subsection{Device Operation}\label{sec:Device Types / Video Device / Device > Operation} > + > +The driver allocates input and output buffers and queues the buffers > +to the device. The device performs operations on the buffers according > +to the function in question. > + > +\subsubsection{Device Operation: Create stream} > + > +To process buffers, the device needs to associate them with a certain > +video stream (essentially, a context). Streams are created by > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > +determined by the device. > + > +\subsubsection{Device Operation: Create buffers} > + > +Buffers are used to store the actual data as well as the relevant > +metadata. Scatter lists are supported, so the buffer doesn't need to > +be contiguous in guest physical memory. > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > + resource that is backed by a buffer allocated from the driver's > + memory. > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > + is no longer needed. > +\end{itemize*} > + > +\subsubsection{Device Operation: Stream parameter control} > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters for > + input and output streams from the device. > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the > + device. > +\item After setting stream parameters, the driver may issue > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can > be > + changed implicitly by the device during the set operation. > +\end{itemize*} > + > +\subsubsection{Device Operation: Process buffers} > + > +\begin{itemize*} > +\item If the function and the buffer type require so, write data to > +the buffer memory. > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > +processing in the device. > +\item The request completes asynchronously when the device has > +finished with the buffer. > +\end{itemize*} > + > +\subsubsection{Device Operation: Buffer processing control} > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > + return all of the already queued buffers. > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > + already queued buffers from the input or the output queue. This also > + includes input or output buffers that can be currently owned by the > + device's processing pipeline. > +\end{itemize*} > + > +\subsubsection{Device Operation: Asynchronous events} > + > +While processing buffers, the device can send asynchronous event > +notifications to the driver. The behaviour depends on the exact > +stream. For example, the decoder device sends a resolution change > +event when it encounters new resolution metadata in the stream. > + > +\subsubsection{Device Operation: Request header} > + > +All requests and responses on the control virt queue have a fixed > +header using the following layout structure and definitions: > + > +\begin{lstlisting} > +enum virtio_video_ctrl_type { > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > + > + /* request */ > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > + VIRTIO_VIDEO_T_STREAM_CREATE, > + VIRTIO_VIDEO_T_STREAM_DESTROY, > + VIRTIO_VIDEO_T_STREAM_DRAIN, > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > + VIRTIO_VIDEO_T_SET_PARAMS, > + VIRTIO_VIDEO_T_GET_PARAMS, > + > + /* response */ > + VIRTIO_VIDEO_S_OK = 0x0200, > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > + > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > +}; > + > +struct virtio_video_ctrl_hdr { > + le32 type; > + le32 stream_id; > + le32 len; /* Length of the structure in bytes. */ > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{type}] is the type of the driver request or the device > +response. > +\item[\field{stream_id}] specifies a target stream. > +\item[\field{len}] is the length of data in bytes, which includes > +length of the header. I suppose is implicit that it's the size of the container structure. > +\end{description} > + > +\subsubsection{Device Operation: controlq} > + > +\begin{description} > + > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > +supported formats. > + > +The driver uses \field{struct virtio_video_get_capability} to send a > +query request. > + > +\begin{lstlisting} > +enum virtio_video_buf_type { > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > +}; > + > +struct virtio_video_get_capability { > + struct virtio_video_ctrl_hdr hdr; > + enum virtio_video_buf_type buf_type; Here you used an enumeration but later you are using just le32 type with a comment. Why this difference? > +}; > +\end{lstlisting} > +\begin{description} > +\item[\field{buf_type}] is the buffer type that the driver asks > +information about. The driver MUST set either > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. > +\end{description} > + > +The device responds a capability by using \field{struct > +virtio_video_get_capability_resp}. > +\begin{lstlisting} > +enum virtio_video_format { > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, Really minor: maybe you want to insert an empty line to be coherent with other enumerations? > + /* Raw formats */ > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > + VIRTIO_VIDEO_FORMAT_YUV420, > + VIRTIO_VIDEO_FORMAT_YVU420, > + > + /* Compressed formats */ > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > +}; > + > +enum virtio_video_profile { > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > + > + /* H.264 */ > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > VIRTIO_VIDEO_PROFILE_H264_BASELINE, Maybe you want VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_MIN, like others? > + VIRTIO_VIDEO_PROFILE_H264_MAIN, > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > + VIRTIO_VIDEO_PROFILE_H264_MAX = > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > + > + /* VP8 */ > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > + > + /* VP9 */ > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > + VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > +}; > + > +struct virtio_video_format_range { > + le32 min; > + le32 max; > + le32 step; > + u8 paddings[4]; Here you padded at 8-byte like many other structures but not for virtio_video_ctrl_hdr. Is this expected? > +}; > + > +struct virtio_video_format_desc { > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > + le64 mask; This will limit the formats to 64 while "num_descs" below is a 32 bit. > + struct virtio_video_format_range width; > + struct virtio_video_format_range height; > + le32 num_rates; > + u8 padding[4]; > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > +}; > + > +struct virtio_video_get_capability_resp { > + struct virtio_video_ctrl_hdr hdr; > + le32 num_descs; > + /* Followed by struct virtio_video_format_desc desc[] */ > +}; > +\end{lstlisting} > + > +The format description \field{struct virtio_video_format_desc} > +includes the following fields: > +\begin{description} > +\item[\field{format}] specifies an image format. The device MUST set one > + of \field{enum virtio_video_format}. > +\item[\field{profile}] specifies a profile of the compressed image format > + specified in \field{format}. The driver SHOULD ignore this value if > + \field{format} is a raw format. Why not documenting that MUST be VIRTIO_VIDEO_PROFILE_UNDEFINED for raw formats so the field could be extended in the future? > +\item[\field{mask}] is a bitset that represents the supported > + combination of input and output format. If \textit{i}-th bit is set > + in \field{mask} of \textit{j}-th \field{struct > + virtio_video_format_desc} for input, the device supports encoding or > + decoding from the \textit{j}-th input format to \textit{i}-th output > + format. > +\item[\field{width, height}] represents a range of resolutions > + supported by the device. If its \field{step} is not applicable, its > + \field{min} is equal to its \field{max}. > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In > case of decoder, the driver SHOULD ignore this value. > +\item[\field{frame_rates}] is an array of supported frame rates. I suppose here we are talking about bitrates, right? Could be confused by FPS. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > + within the device. > + > +\begin{lstlisting} > +enum virtio_video_mem_type { > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > +}; > + > +struct virtio_video_stream_create { > + struct virtio_video_ctrl_hdr hdr; > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > + char debug_name[64]; > +}; This structure has a size not multiple of 8-bytes. Not an issue, but the same apply to other structures that instead are padded to 8-byte size. > +\end{lstlisting} > + > +\begin{description} > +\item[\field{in_mem_type}] is a type of buffer management for input > +buffers. The driver MUST set a value in \field{enum > +virtio_video_mem_type}. > +\item[\field{out_mem_type}] is a type of buffer management for output > +buffers. The driver MUST set a value in \field{enum > +virtio_video_mem_type}. > +\item[\field{debug_name}] is a text string for a debug purpose. Must be NUL-terminated ? UTF-8 ? > +\end{description} > + > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_stream_destroy { > + struct virtio_video_ctrl_hdr hdr; > +}; > +\end{lstlisting} > + > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > + queued buffers through the pipeline. > + > +\begin{lstlisting} > +struct virtio_video_stream_drain { > + struct virtio_video_ctrl_hdr hdr; > +}; > +\end{lstlisting} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_mem_entry { > + le64 addr; > + le32 length; > + u8 padding[4]; > +}; > + > +struct virtio_video_resource_create { > + struct virtio_video_ctrl_hdr hdr; > + le32 resource_id; > + le32 nr_entries; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{resource_id}] internal id of the resource. Is it input or output? > +\item[\field{nr_entries}] number of \field{struct > + virtio_video_mem_entry} memory entries. I suppose that the structure is followed by an array of virtio_video_mem_entry after a padding of 4-bytes for alignment. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_resource_destroy { > + struct virtio_video_ctrl_hdr hdr; > + le32 resource_id; > + u8 padding[4]; Is this padding for future extensions? > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{resource_id}] internal id of the resource. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > +queue. > + > +\begin{lstlisting} > +#define VIRTIO_VIDEO_MAX_PLANES 8 > + > +struct virtio_video_resource_queue { > + struct virtio_video_ctrl_hdr hdr; > + le32 buf_type; > + le32 resource_id; > + le64 timestamp; I suppose you would like this field aligned to avoid hidden paddings. > + le32 nr_data_size; > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{buf_type}] buf_type of the . > +\item[\field{resource_id}] internal id of the resource. > +\item[\field{timestamp}] an abstract sequence counter that can be used > + for synchronisation. > +\item[\field{nr_data_size}] number of \field{data_size} entries. > +\item[\field{data_size}] number of data bytes within a plane. > +\end{description} > + ... omissis ... > -- > 2.24.1.735.g03f4e72817-goog > > Frediano
On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote: > On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> wrote: > > > > Hi, > > > > > +The device MUST mark the last buffer with the > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > > +sequence. > > > > No, that would build a race condition into the protocol. The device > > could complete the last buffer after the driver has sent the drain > > command but before the device saw it. So the flag would not be > > reliable. > > > > I also can't see why the flag is needed in the first place. The driver > > should know which buffers are queued still and be able to figure > > whenever the drain is complete or not without depending on that flag. > > So I'd suggest to simply drop it. > > Unfortunately video decoders are not that simple. There are always > going to be some buffers on the decoder side used as reference frames. > Only the decoder knows when to release them, as it continues decoding > the stream. Not clearly defined in the spec: When is the decoder supposed to send the response for a queue request? When it finished decoding (i.e. frame is ready for playback), or when it doesn't need the buffer any more for decoding (i.e. buffer can be re-queued or pages can be released)? > How we defined this in the V4L2 stateful decoder interface is that if > the decoder happened to return the last framebuffer before the drain > request arrived, it would return one more, empty. Ok. That is not clear from the spec. I've read the drain request as as "please stop decoding and complete all queue requests which are in the virtqueue, even when you didn't process them yet". In which case completing the last pending queue request would clearly indicate the draining request is complete. Also completing the drain request should only happen when the operation is finished. I think the various states a buffer can have and how queuing & deciding & draining changes these states should be clarified in the specification. cheers, Gerd
Hi Gerd, On Mittwoch, 18. Dezember 2019 14:40:37 CET Gerd Hoffmann wrote: > Hi, > > > +The device MUST mark the last buffer with the > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > +sequence. > > No, that would build a race condition into the protocol. The device > could complete the last buffer after the driver has sent the drain > command but before the device saw it. So the flag would not be > reliable. No, then it means the device was not in drain, but, for example, hit a resolution change in the stream and tells us that this is the last buffer with the old resolution. > > I also can't see why the flag is needed in the first place. The driver > should know which buffers are queued still and be able to figure > whenever the drain is complete or not without depending on that flag. > So I'd suggest to simply drop it. This flag is used not for drain only. In marks the completion of whatever specific buffer sequence, like a full end-of-stream, resolution change, drain etc. We also need this to handle nested sequences. For instance, a resolution change event might happen while in drain. Regards, Dmitry. > > That is the only issue I've spotted in the protocol on a first quick > look. There are a few places where the spec text could be improved. > I'll try to set aside some time to go over this in detail, but I can't > promise I'll find the time to do that before xmas and new year. > > cheers, > Gerd
Hi, On Donnerstag, 19. Dezember 2019 08:46:39 CET Gerd Hoffmann wrote: > On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote: > > On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> wrote: > > > Hi, > > > > > > > +The device MUST mark the last buffer with the > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > > > +sequence. > > > > > > No, that would build a race condition into the protocol. The device > > > could complete the last buffer after the driver has sent the drain > > > command but before the device saw it. So the flag would not be > > > reliable. > > > > > > I also can't see why the flag is needed in the first place. The driver > > > should know which buffers are queued still and be able to figure > > > whenever the drain is complete or not without depending on that flag. > > > So I'd suggest to simply drop it. > > > > Unfortunately video decoders are not that simple. There are always > > going to be some buffers on the decoder side used as reference frames. > > Only the decoder knows when to release them, as it continues decoding > > the stream. > > Not clearly defined in the spec: When is the decoder supposed to send > the response for a queue request? When it finished decoding (i.e. frame > is ready for playback), or when it doesn't need the buffer any more for > decoding (i.e. buffer can be re-queued or pages can be released)? In my eyes the both statements mean almost the same and both are valid. I think whatever underlying libraries are used for decoding on the device side, they simply won't return us the buffer as long as the HW device needs them to continue its normal operation. So your first sentence applies to output buffers, the second - to input buffers. My understanding is as follows: we send the response for a queue request as soon as the HW device on the host side passes the buffer ownership back to the client (like when VIDIOC_DQBUF has returned a buffer). Thanks for reviewing! Regards, Dmitry. > > > How we defined this in the V4L2 stateful decoder interface is that if > > the decoder happened to return the last framebuffer before the drain > > request arrived, it would return one more, empty. > > Ok. That is not clear from the spec. I've read the drain request as as > "please stop decoding and complete all queue requests which are in the > virtqueue, even when you didn't process them yet". In which case > completing the last pending queue request would clearly indicate the > draining request is complete. Also completing the drain request should > only happen when the operation is finished. > > I think the various states a buffer can have and how queuing & deciding > & draining changes these states should be clarified in the > specification. > > cheers, > Gerd
On Thu, Dec 19, 2019 at 6:48 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi, > > On Donnerstag, 19. Dezember 2019 08:46:39 CET Gerd Hoffmann wrote: > > On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote: > > > On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> wrote: > > > > Hi, > > > > > > > > > +The device MUST mark the last buffer with the > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > > > > +sequence. > > > > > > > > No, that would build a race condition into the protocol. The device > > > > could complete the last buffer after the driver has sent the drain > > > > command but before the device saw it. So the flag would not be > > > > reliable. > > > > > > > > I also can't see why the flag is needed in the first place. The driver > > > > should know which buffers are queued still and be able to figure > > > > whenever the drain is complete or not without depending on that flag. > > > > So I'd suggest to simply drop it. > > > > > > Unfortunately video decoders are not that simple. There are always > > > going to be some buffers on the decoder side used as reference frames. > > > Only the decoder knows when to release them, as it continues decoding > > > the stream. > > > > Not clearly defined in the spec: When is the decoder supposed to send > > the response for a queue request? When it finished decoding (i.e. frame > > is ready for playback), or when it doesn't need the buffer any more for > > decoding (i.e. buffer can be re-queued or pages can be released)? > In my eyes the both statements mean almost the same and both are valid. I > think whatever underlying libraries are used for decoding on the device side, > they simply won't return us the buffer as long as the HW device needs them to > continue its normal operation. So your first sentence applies to output buffers, > the second - to input buffers. > > My understanding is as follows: we send the response for a queue request as > soon as the HW device on the host side passes the buffer ownership back to the > client (like when VIDIOC_DQBUF has returned a buffer). That's how it's defined in V4L2 and what makes the most sense from the video decoding point of view, as one wants to display frames as soon as they are available. However that still doesn't let the driver know which buffers will be dequeued when. A simple example of this scenario is when the guest is done displaying a frame and requeues the buffer back to the decoder. Then the decoder will not choose it for decoding next frames into as long as the frame in that buffer is still used as a reference frame, even if one sends the drain request. > > Thanks for reviewing! > > Regards, > Dmitry. > > > > > > How we defined this in the V4L2 stateful decoder interface is that if > > > the decoder happened to return the last framebuffer before the drain > > > request arrived, it would return one more, empty. > > > > Ok. That is not clear from the spec. I've read the drain request as as > > "please stop decoding and complete all queue requests which are in the > > virtqueue, even when you didn't process them yet". In which case > > completing the last pending queue request would clearly indicate the > > draining request is complete. Also completing the drain request should > > only happen when the operation is finished. > > > > I think the various states a buffer can have and how queuing & deciding > > & draining changes these states should be clarified in the > > specification. > > > > cheers, > > Gerd > >
On Thu, Dec 19, 2019 at 6:26 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi Gerd, > > On Mittwoch, 18. Dezember 2019 14:40:37 CET Gerd Hoffmann wrote: > > Hi, > > > > > +The device MUST mark the last buffer with the > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > > +sequence. > > > > No, that would build a race condition into the protocol. The device > > could complete the last buffer after the driver has sent the drain > > command but before the device saw it. So the flag would not be > > reliable. > No, then it means the device was not in drain, but, for example, hit a > resolution change in the stream and tells us that this is the last buffer with > the old resolution. > > > > > I also can't see why the flag is needed in the first place. The driver > > should know which buffers are queued still and be able to figure > > whenever the drain is complete or not without depending on that flag. > > So I'd suggest to simply drop it. > This flag is used not for drain only. In marks the completion of whatever > specific buffer sequence, like a full end-of-stream, resolution change, drain > etc. We also need this to handle nested sequences. For instance, a resolution > change event might happen while in drain. Good point. The resolution change event is in a different virtqueue, so it's not serialized with the request completions.
Hi Tomasz, On Donnerstag, 19. Dezember 2019 10:59:02 CET Tomasz Figa wrote: > On Thu, Dec 19, 2019 at 6:48 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi, > > > > On Donnerstag, 19. Dezember 2019 08:46:39 CET Gerd Hoffmann wrote: > > > On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote: > > > > On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> wrote: > > > > > Hi, > > > > > > > > > > > +The device MUST mark the last buffer with the > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > > > > > +sequence. > > > > > > > > > > No, that would build a race condition into the protocol. The device > > > > > could complete the last buffer after the driver has sent the drain > > > > > command but before the device saw it. So the flag would not be > > > > > reliable. > > > > > > > > > > I also can't see why the flag is needed in the first place. The > > > > > driver > > > > > should know which buffers are queued still and be able to figure > > > > > whenever the drain is complete or not without depending on that > > > > > flag. > > > > > So I'd suggest to simply drop it. > > > > > > > > Unfortunately video decoders are not that simple. There are always > > > > going to be some buffers on the decoder side used as reference frames. > > > > Only the decoder knows when to release them, as it continues decoding > > > > the stream. > > > > > > Not clearly defined in the spec: When is the decoder supposed to send > > > the response for a queue request? When it finished decoding (i.e. frame > > > is ready for playback), or when it doesn't need the buffer any more for > > > decoding (i.e. buffer can be re-queued or pages can be released)? > > > > In my eyes the both statements mean almost the same and both are valid. I > > think whatever underlying libraries are used for decoding on the device > > side, they simply won't return us the buffer as long as the HW device > > needs them to continue its normal operation. So your first sentence > > applies to output buffers, the second - to input buffers. > > > > My understanding is as follows: we send the response for a queue request > > as > > soon as the HW device on the host side passes the buffer ownership back to > > the client (like when VIDIOC_DQBUF has returned a buffer). > > That's how it's defined in V4L2 and what makes the most sense from the > video decoding point of view, as one wants to display frames as soon > as they are available. > > However that still doesn't let the driver know which buffers will be > dequeued when. A simple example of this scenario is when the guest is > done displaying a frame and requeues the buffer back to the decoder. > Then the decoder will not choose it for decoding next frames into as > long as the frame in that buffer is still used as a reference frame, > even if one sends the drain request. It might be that I'm getting your point wrong, but do you mean some hardware can mark a buffer as ready to be displayed yet still using the underlying memory to decode other frames? This means, if you occasionally/intentionally write to the buffer you mess up the whole decoding pipeline. That would be strange at least... Regds, Dmitry. > > > Thanks for reviewing! > > > > Regards, > > Dmitry. > > > > > > How we defined this in the V4L2 stateful decoder interface is that if > > > > the decoder happened to return the last framebuffer before the drain > > > > request arrived, it would return one more, empty. > > > > > > Ok. That is not clear from the spec. I've read the drain request as as > > > "please stop decoding and complete all queue requests which are in the > > > virtqueue, even when you didn't process them yet". In which case > > > completing the last pending queue request would clearly indicate the > > > draining request is complete. Also completing the drain request should > > > only happen when the operation is finished. > > > > > > I think the various states a buffer can have and how queuing & deciding > > > & draining changes these states should be clarified in the > > > specification. > > > > > > cheers, > > > > > > Gerd
On Thu, Dec 19, 2019 at 7:55 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi Tomasz, > > On Donnerstag, 19. Dezember 2019 10:59:02 CET Tomasz Figa wrote: > > On Thu, Dec 19, 2019 at 6:48 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> > wrote: > > > Hi, > > > > > > On Donnerstag, 19. Dezember 2019 08:46:39 CET Gerd Hoffmann wrote: > > > > On Wed, Dec 18, 2019 at 11:08:37PM +0900, Tomasz Figa wrote: > > > > > On Wed, Dec 18, 2019 at 10:40 PM Gerd Hoffmann <kraxel@redhat.com> > wrote: > > > > > > Hi, > > > > > > > > > > > > > +The device MUST mark the last buffer with the > > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > > > > > > +sequence. > > > > > > > > > > > > No, that would build a race condition into the protocol. The device > > > > > > could complete the last buffer after the driver has sent the drain > > > > > > command but before the device saw it. So the flag would not be > > > > > > reliable. > > > > > > > > > > > > I also can't see why the flag is needed in the first place. The > > > > > > driver > > > > > > should know which buffers are queued still and be able to figure > > > > > > whenever the drain is complete or not without depending on that > > > > > > flag. > > > > > > So I'd suggest to simply drop it. > > > > > > > > > > Unfortunately video decoders are not that simple. There are always > > > > > going to be some buffers on the decoder side used as reference frames. > > > > > Only the decoder knows when to release them, as it continues decoding > > > > > the stream. > > > > > > > > Not clearly defined in the spec: When is the decoder supposed to send > > > > the response for a queue request? When it finished decoding (i.e. frame > > > > is ready for playback), or when it doesn't need the buffer any more for > > > > decoding (i.e. buffer can be re-queued or pages can be released)? > > > > > > In my eyes the both statements mean almost the same and both are valid. I > > > think whatever underlying libraries are used for decoding on the device > > > side, they simply won't return us the buffer as long as the HW device > > > needs them to continue its normal operation. So your first sentence > > > applies to output buffers, the second - to input buffers. > > > > > > My understanding is as follows: we send the response for a queue request > > > as > > > soon as the HW device on the host side passes the buffer ownership back to > > > the client (like when VIDIOC_DQBUF has returned a buffer). > > > > That's how it's defined in V4L2 and what makes the most sense from the > > video decoding point of view, as one wants to display frames as soon > > as they are available. > > > > However that still doesn't let the driver know which buffers will be > > dequeued when. A simple example of this scenario is when the guest is > > done displaying a frame and requeues the buffer back to the decoder. > > Then the decoder will not choose it for decoding next frames into as > > long as the frame in that buffer is still used as a reference frame, > > even if one sends the drain request. > It might be that I'm getting your point wrong, but do you mean some hardware > can mark a buffer as ready to be displayed yet still using the underlying > memory to decode other frames? This means, if you occasionally/intentionally > write to the buffer you mess up the whole decoding pipeline. That would be > strange at least... That's correct. It depends on the hardware, but in principle we don't want to copy the frames decoded to temporary buffers for using them as reference frames, as that would waste bandwidth and increase latency. The contract between the kernel and the application is that it must not write to the frame buffers if it wants to get correct decoding results. But after all, I don't see a reason why the application would write to those buffers. Best regards, Tomasz
Hi, > > I also can't see why the flag is needed in the first place. The driver > > should know which buffers are queued still and be able to figure > > whenever the drain is complete or not without depending on that flag. > > So I'd suggest to simply drop it. > This flag is used not for drain only. In marks the completion of whatever > specific buffer sequence, like a full end-of-stream, resolution change, drain > etc. We also need this to handle nested sequences. For instance, a resolution > change event might happen while in drain. Ah, ok. That makes sense (please clarify this in the spec). cheers, Gerd
Hi, > > Not clearly defined in the spec: When is the decoder supposed to send > > the response for a queue request? When it finished decoding (i.e. frame > > is ready for playback), or when it doesn't need the buffer any more for > > decoding (i.e. buffer can be re-queued or pages can be released)? > In my eyes the both statements mean almost the same and both are valid. Well, no. When the device decoded a P-Frame it can notify the device, saying "here is your decoded frame". But the device might still need the buffer with the decoded frame to properly decode the following B/I Frames which reference the P-Frame. cheers, Gerd
Hi, > > However that still doesn't let the driver know which buffers will be > > dequeued when. A simple example of this scenario is when the guest is > > done displaying a frame and requeues the buffer back to the decoder. > > Then the decoder will not choose it for decoding next frames into as > > long as the frame in that buffer is still used as a reference frame, > > even if one sends the drain request. > It might be that I'm getting your point wrong, but do you mean some hardware > can mark a buffer as ready to be displayed yet still using the underlying > memory to decode other frames? Yes, this is how I understand Tomasz Figa. > This means, if you occasionally/intentionally > write to the buffer you mess up the whole decoding pipeline. And to avoid this the buffer handling aspect must be clarified in the specification. Is the device allowed to continue using the buffer after finishing decoding and completing the queue request? If so, how do we hand over buffer ownership back to the driver so it can free the pages? drain request? How do we handle re-using buffers? Can the driver simply re-queue them and expect the device figures by itself whenever it can use the buffer or whenever it is still needed as reference frame? cheers, Gerd
Hi Keiichi, Thank you for the update. Please see some comments below. Also, we need to bring the virtio_video_control back as it is in fact used by the driver to enumerate supported encoder controls. But yes, it still needs to be documemnted, it's true. On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > The virtio video encoder device and decoder device provide functionalities > to encode and decode video stream respectively. > Though video encoder and decoder are provided as different devices, they use > a same protocol. > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > --- > content.tex | 1 + > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 580 insertions(+) > create mode 100644 virtio-video.tex > > diff --git a/content.tex b/content.tex > index 556b373..9e56839 100644 > --- a/content.tex > +++ b/content.tex > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > Requirements}\label{sec:Device \input{virtio-vsock.tex} > \input{virtio-fs.tex} > \input{virtio-rpmb.tex} > +\input{virtio-video.tex} > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > diff --git a/virtio-video.tex b/virtio-video.tex > new file mode 100644 > index 0000000..30e728d > --- /dev/null > +++ b/virtio-video.tex > @@ -0,0 +1,579 @@ > +\section{Video Device}\label{sec:Device Types / Video Device} > + > +The virtio video encoder device and decoder device are virtual devices that > +supports encoding and decoding respectively. Though the encoder and the > decoder +are different devices, they use the same protocol. > + > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID} > + > +\begin{description} > +\item[30] encoder device > +\item[31] decoder device > +\end{description} > + > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues} > + > +\begin{description} > +\item[0] controlq - queue for sending control commands. > +\item[1] eventq - queue for sending events happened in the device. > +\end{description} > + > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature > bits} + > +\begin{description} > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for > video + buffers. > +\end{description} > + > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device > / Feature bits} + > +The device MUST offer at least one of feature bits. > + > +\subsection{Device configuration layout}\label{sec:Device Types / Video > Device / Device configuration layout} + > +Video device configuration uses the following layout structure: > + > +\begin{lstlisting} > +struct virtio_video_config { > + le32 max_cap_len; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > + MUST set this value. > +\end{description} > + > +\subsection{Device Initialization}\label{sec:Device Types / Video Device / > Device Initialization} + > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / > Video Device / Device Initialization} + > +The driver SHOULD query device capability by using the > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > +setup. > + > +\subsection{Device Operation}\label{sec:Device Types / Video Device / > Device Operation} + > +The driver allocates input and output buffers and queues the buffers > +to the device. The device performs operations on the buffers according > +to the function in question. > + > +\subsubsection{Device Operation: Create stream} > + > +To process buffers, the device needs to associate them with a certain > +video stream (essentially, a context). Streams are created by > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > +determined by the device. > + > +\subsubsection{Device Operation: Create buffers} > + > +Buffers are used to store the actual data as well as the relevant > +metadata. Scatter lists are supported, so the buffer doesn't need to > +be contiguous in guest physical memory. > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > + resource that is backed by a buffer allocated from the driver's > + memory. > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > + is no longer needed. > +\end{itemize*} > + > +\subsubsection{Device Operation: Stream parameter control} > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters > for + input and output streams from the device. > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the > + device. > +\item After setting stream parameters, the driver may issue > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can > be + changed implicitly by the device during the set operation. > +\end{itemize*} > + > +\subsubsection{Device Operation: Process buffers} > + > +\begin{itemize*} > +\item If the function and the buffer type require so, write data to > +the buffer memory. > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > +processing in the device. > +\item The request completes asynchronously when the device has > +finished with the buffer. > +\end{itemize*} > + > +\subsubsection{Device Operation: Buffer processing control} > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > + return all of the already queued buffers. > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > + already queued buffers from the input or the output queue. This also > + includes input or output buffers that can be currently owned by the > + device's processing pipeline. > +\end{itemize*} > + > +\subsubsection{Device Operation: Asynchronous events} > + > +While processing buffers, the device can send asynchronous event > +notifications to the driver. The behaviour depends on the exact > +stream. For example, the decoder device sends a resolution change > +event when it encounters new resolution metadata in the stream. > + > +\subsubsection{Device Operation: Request header} > + > +All requests and responses on the control virt queue have a fixed > +header using the following layout structure and definitions: > + > +\begin{lstlisting} > +enum virtio_video_ctrl_type { > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > + > + /* request */ > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > + VIRTIO_VIDEO_T_STREAM_CREATE, > + VIRTIO_VIDEO_T_STREAM_DESTROY, > + VIRTIO_VIDEO_T_STREAM_DRAIN, > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > + VIRTIO_VIDEO_T_SET_PARAMS, > + VIRTIO_VIDEO_T_GET_PARAMS, > + > + /* response */ > + VIRTIO_VIDEO_S_OK = 0x0200, > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > + > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > +}; > + > +struct virtio_video_ctrl_hdr { > + le32 type; > + le32 stream_id; > + le32 len; /* Length of the structure in bytes. */ > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{type}] is the type of the driver request or the device > +response. > +\item[\field{stream_id}] specifies a target stream. > +\item[\field{len}] is the length of data in bytes, which includes > +length of the header. > +\end{description} > + > +\subsubsection{Device Operation: controlq} > + > +\begin{description} > + > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > +supported formats. > + > +The driver uses \field{struct virtio_video_get_capability} to send a > +query request. > + > +\begin{lstlisting} > +enum virtio_video_buf_type { > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > +}; I personally didn't like the previous term: pin_type. But, to be honest, I don't like the buf_type neither. Consider the GET/SET_PARAMS request: buf_type there looks a bit unnatural. We are trying to get stream parameters there, not some parameters of whatever buffer. Also I don't see any strict reason to mimic v4l2 naming scheme. I'd better rename it to PORT_TYPE or QUEUE_TYPE. > + > +struct virtio_video_get_capability { > + struct virtio_video_ctrl_hdr hdr; > + enum virtio_video_buf_type buf_type; > +}; > +\end{lstlisting} > +\begin{description} > +\item[\field{buf_type}] is the buffer type that the driver asks > +information about. The driver MUST set either > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > + > +The device responds a capability by using \field{struct > +virtio_video_get_capability_resp}. > +\begin{lstlisting} > +enum virtio_video_format { > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > + /* Raw formats */ > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > + VIRTIO_VIDEO_FORMAT_YUV420, > + VIRTIO_VIDEO_FORMAT_YVU420, Let's add some variants of RGB, like RGBA, ARGB. We need it for the encoder in particular . > + > + /* Compressed formats */ > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, Let's add H265, MPEG4, MPEG2. We already support and use them. Regards, Dmitry. > +}; > + > +enum virtio_video_profile { > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > + > + /* H.264 */ > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > VIRTIO_VIDEO_PROFILE_H264_MAIN, > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > + VIRTIO_VIDEO_PROFILE_H264_MAX = > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > + /* VP8 */ > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > + > + /* VP9 */ > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > + VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > +}; > + > +struct virtio_video_format_range { > + le32 min; > + le32 max; > + le32 step; > + u8 paddings[4]; > +}; > + > +struct virtio_video_format_desc { > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > + le64 mask; > + struct virtio_video_format_range width; > + struct virtio_video_format_range height; > + le32 num_rates; > + u8 padding[4]; > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > +}; > + > +struct virtio_video_get_capability_resp { > + struct virtio_video_ctrl_hdr hdr; > + le32 num_descs; > + /* Followed by struct virtio_video_format_desc desc[] */ > +}; > +\end{lstlisting} > + > +The format description \field{struct virtio_video_format_desc} > +includes the following fields: > +\begin{description} > +\item[\field{format}] specifies an image format. The device MUST set one > + of \field{enum virtio_video_format}. > +\item[\field{profile}] specifies a profile of the compressed image format > + specified in \field{format}. The driver SHOULD ignore this value if > + \field{format} is a raw format. > +\item[\field{mask}] is a bitset that represents the supported > + combination of input and output format. If \textit{i}-th bit is set > + in \field{mask} of \textit{j}-th \field{struct > + virtio_video_format_desc} for input, the device supports encoding or > + decoding from the \textit{j}-th input format to \textit{i}-th output > + format. > +\item[\field{width, height}] represents a range of resolutions > + supported by the device. If its \field{step} is not applicable, its > + \field{min} is equal to its \field{max}. > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In > case of decoder, the driver SHOULD ignore this value. > +\item[\field{frame_rates}] is an array of supported frame rates. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > + within the device. > + > +\begin{lstlisting} > +enum virtio_video_mem_type { > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > +}; > + > +struct virtio_video_stream_create { > + struct virtio_video_ctrl_hdr hdr; > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > + char debug_name[64]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{in_mem_type}] is a type of buffer management for input > +buffers. The driver MUST set a value in \field{enum > +virtio_video_mem_type}. > +\item[\field{out_mem_type}] is a type of buffer management for output > +buffers. The driver MUST set a value in \field{enum > +virtio_video_mem_type}. > +\item[\field{debug_name}] is a text string for a debug purpose. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_stream_destroy { > + struct virtio_video_ctrl_hdr hdr; > +}; > +\end{lstlisting} > + > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > + queued buffers through the pipeline. > + > +\begin{lstlisting} > +struct virtio_video_stream_drain { > + struct virtio_video_ctrl_hdr hdr; > +}; > +\end{lstlisting} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_mem_entry { > + le64 addr; > + le32 length; > + u8 padding[4]; > +}; > + > +struct virtio_video_resource_create { > + struct virtio_video_ctrl_hdr hdr; > + le32 resource_id; > + le32 nr_entries; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{resource_id}] internal id of the resource. > +\item[\field{nr_entries}] number of \field{struct > + virtio_video_mem_entry} memory entries. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_resource_destroy { > + struct virtio_video_ctrl_hdr hdr; > + le32 resource_id; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{resource_id}] internal id of the resource. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > +queue. > + > +\begin{lstlisting} > +#define VIRTIO_VIDEO_MAX_PLANES 8 > + > +struct virtio_video_resource_queue { > + struct virtio_video_ctrl_hdr hdr; > + le32 buf_type; > + le32 resource_id; > + le64 timestamp; > + le32 nr_data_size; > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{buf_type}] buf_type of the . > +\item[\field{resource_id}] internal id of the resource. > +\item[\field{timestamp}] an abstract sequence counter that can be used > + for synchronisation. > +\item[\field{nr_data_size}] number of \field{data_size} entries. > +\item[\field{data_size}] number of data bytes within a plane. > +\end{description} > + > +\begin{lstlisting} > +enum virtio_video_buffer_flag { > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > + /* Encoder only */ > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > +}; > + > +struct virtio_video_resource_queue_resp { > + struct virtio_video_ctrl_hdr hdr; > + le64 timestamp; > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > + le32 size; /* Encoded size */ > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{timestamp}] an abstract sequence counter that can be used > + for synchronisation. > +\item[\field{flags}] mark specific buffers in the sequence. > +\item[\field{size}] data size in the buffer (encoder only). > +\end{description} > + > +The device sends a response to the queue request asynchronously when > +it has finished processing the buffer. > + > +The device SHOULD mark a buffer that triggered a processing error with > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > + > +The device MUST mark the last buffer with the > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > +sequence. > + > +In case of encoder, to denote a particular frame type the devie MUST > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > + > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > + buffers back from the input or the output queue of the device. The > + device SHOULD return all of the buffers from the respective queue as > + soon as possible without pushing the buffers through the processing > + pipeline. > + > +\begin{lstlisting} > +struct virtio_video_queue_clear { > + struct virtio_video_ctrl_hdr hdr; > + le32 buf_type; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{buf_type}] buffer type. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > + output of a stream. > + > +\begin{lstlisting} > +struct virtio_video_plane_format { > + le32 plane_size; > + le32 stride; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{plane_size}] size of the plane in bytes. > +\item[\field{stride}] stride used for the plane in bytes. > +\end{description} > + > +\begin{lstlisting} > +struct virtio_video_params { > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > + le32 frame_width; > + le32 frame_height; > + le32 min_buffers; > + le32 max_buffers; > + le32 frame_rate; > + struct virtio_video_crop { > + le32 left; > + le32 top; > + le32 width; > + le32 height; > + } crop; > + le32 num_planes; > + struct virtio_video_plane_format > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{frame_width}] the value to get/set. > +\item[\field{frame_height}] the value to get/set. > +\item[\field{pixel_format}] the value to get/set. > +\item[\field{min_buffers}] minimum buffers required to handle the > + format (r/o). > +\item[\field{max_buffers}] maximum buffers required to handle the > + format (r/o). > +\item[\field{frame_rate}] the value to get/set. > +\item[\field{crop}] cropping (composing) rectangle. > +\item[\field{num_planes}] number of planes used to store pixel data > +(r/o). > +\item[\field{plane_formats}] description of each plane. > +\end{description} > + > +\begin{lstlisting} > +struct virtio_video_get_params { > + struct virtio_video_ctrl_hdr hdr; > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > +}; > + > +struct virtio_video_get_params_resp { > + struct virtio_video_ctrl_hdr hdr; > + struct virtio_video_params params; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{buf_type}] buffer type. > +\item[\field{params}] parameter values. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > + > + > +\begin{lstlisting} > +struct virtio_video_set_params { > + struct virtio_video_ctrl_hdr hdr; > + struct virtio_video_params params; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{params}] parameters to set. > +\end{description} > + > +Setting stream parameters might have side effects within the device. > +For example, the device MAY perform alignment of width and height, > +change the number of planes it uses for the format, or do whatever > +changes that are required to continue normal operation using the > +updated parameters. It is up to the driver to check the parameter set > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > + > +\end{description} > + > +\subsubsection{Device Operation: eventq} > + > +The device can report events on the event queue. The driver initially > +populates the queue with device-writeable buffers. When the device > +needs to report an event, it fills a buffer and notifies the driver. > +The driver consumes the report and adds a new buffer to the virtqueue. > + > +\begin{lstlisting} > +enum virtio_video_event_type { > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > + /* For all functions */ > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > + /* For decoder only */ > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > +}; > + > +struct virtio_video_event { > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > + le32 stream_id; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{event_type}] type of the triggered event . > +\item[\field{stream_id}] id of the source stream. > +\end{description} > + > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > +whenever it encounters new resolution data in the stream. This > +includes the case of the initial device configuration after metadata > +has been parsed and the case of dynamic resolution change.
Hi Frediano, Thanks for reviewing! On Thu, Dec 19, 2019 at 2:29 AM Frediano Ziglio <fziglio@redhat.com> wrote: > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > The virtio video encoder device and decoder device provide functionalities to > > encode and decode video stream respectively. > > Though video encoder and decoder are provided as different devices, they use > > a > > same protocol. > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > --- > > content.tex | 1 + > > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > > 2 files changed, 580 insertions(+) > > create mode 100644 virtio-video.tex > > > > diff --git a/content.tex b/content.tex > > index 556b373..9e56839 100644 > > --- a/content.tex > > +++ b/content.tex > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > Requirements}\label{sec:Device > > \input{virtio-vsock.tex} > > \input{virtio-fs.tex} > > \input{virtio-rpmb.tex} > > +\input{virtio-video.tex} > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > new file mode 100644 > > index 0000000..30e728d > > --- /dev/null > > +++ b/virtio-video.tex > > @@ -0,0 +1,579 @@ > > +\section{Video Device}\label{sec:Device Types / Video Device} > > + > > +The virtio video encoder device and decoder device are virtual devices that > > +supports encoding and decoding respectively. Though the encoder and the > > decoder > > +are different devices, they use the same protocol. > > + > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID} > > + > > +\begin{description} > > +\item[30] encoder device > > +\item[31] decoder device > > +\end{description} > > + > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues} > > + > > +\begin{description} > > +\item[0] controlq - queue for sending control commands. > > +\item[1] eventq - queue for sending events happened in the device. > > +\end{description} > > + > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature > > bits} > > + > > +\begin{description} > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for > > video > > + buffers. > > +\end{description} > > + > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / > > Feature bits} > > + > > +The device MUST offer at least one of feature bits. > > + > > +\subsection{Device configuration layout}\label{sec:Device Types / Video > > Device / Device configuration layout} > > + > > +Video device configuration uses the following layout structure: > > + > > +\begin{lstlisting} > > +struct virtio_video_config { > > + le32 max_cap_len; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > > + MUST set this value. > > +\end{description} > > + > > +\subsection{Device Initialization}\label{sec:Device Types / Video Device / > > Device Initialization} > > + > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / Video > > Device / Device Initialization} > > + > > +The driver SHOULD query device capability by using the > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > > +setup. > > + > > +\subsection{Device Operation}\label{sec:Device Types / Video Device / Device > > Operation} > > + > > +The driver allocates input and output buffers and queues the buffers > > +to the device. The device performs operations on the buffers according > > +to the function in question. > > + > > +\subsubsection{Device Operation: Create stream} > > + > > +To process buffers, the device needs to associate them with a certain > > +video stream (essentially, a context). Streams are created by > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > +determined by the device. > > + > > +\subsubsection{Device Operation: Create buffers} > > + > > +Buffers are used to store the actual data as well as the relevant > > +metadata. Scatter lists are supported, so the buffer doesn't need to > > +be contiguous in guest physical memory. > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > > + resource that is backed by a buffer allocated from the driver's > > + memory. > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > > + is no longer needed. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Stream parameter control} > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters for > > + input and output streams from the device. > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the > > + device. > > +\item After setting stream parameters, the driver may issue > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can > > be > > + changed implicitly by the device during the set operation. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Process buffers} > > + > > +\begin{itemize*} > > +\item If the function and the buffer type require so, write data to > > +the buffer memory. > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > > +processing in the device. > > +\item The request completes asynchronously when the device has > > +finished with the buffer. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Buffer processing control} > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > > + return all of the already queued buffers. > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > > + already queued buffers from the input or the output queue. This also > > + includes input or output buffers that can be currently owned by the > > + device's processing pipeline. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Asynchronous events} > > + > > +While processing buffers, the device can send asynchronous event > > +notifications to the driver. The behaviour depends on the exact > > +stream. For example, the decoder device sends a resolution change > > +event when it encounters new resolution metadata in the stream. > > + > > +\subsubsection{Device Operation: Request header} > > + > > +All requests and responses on the control virt queue have a fixed > > +header using the following layout structure and definitions: > > + > > +\begin{lstlisting} > > +enum virtio_video_ctrl_type { > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > + > > + /* request */ > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > + VIRTIO_VIDEO_T_SET_PARAMS, > > + VIRTIO_VIDEO_T_GET_PARAMS, > > + > > + /* response */ > > + VIRTIO_VIDEO_S_OK = 0x0200, > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > + > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > +}; > > + > > +struct virtio_video_ctrl_hdr { > > + le32 type; > > + le32 stream_id; > > + le32 len; /* Length of the structure in bytes. */ > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{type}] is the type of the driver request or the device > > +response. > > +\item[\field{stream_id}] specifies a target stream. > > +\item[\field{len}] is the length of data in bytes, which includes > > +length of the header. > > I suppose is implicit that it's the size of the container structure. > You're right. All struct except virtio_video_get_capability_resp have fixed size and virtio_video_get_capability_resp has num_descs. So, no need to have len. Let me remove this in the next iteration. > > +\end{description} > > + > > +\subsubsection{Device Operation: controlq} > > + > > +\begin{description} > > + > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > > +supported formats. > > + > > +The driver uses \field{struct virtio_video_get_capability} to send a > > +query request. > > + > > +\begin{lstlisting} > > +enum virtio_video_buf_type { > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > +}; > > + > > +struct virtio_video_get_capability { > > + struct virtio_video_ctrl_hdr hdr; > > + enum virtio_video_buf_type buf_type; > > Here you used an enumeration but later you are using just le32 type > with a comment. Why this difference? No difference. I will change this to use le32 since virtio-gpu's spec does so. > > > +}; > > +\end{lstlisting} > > +\begin{description} > > +\item[\field{buf_type}] is the buffer type that the driver asks > > +information about. The driver MUST set either > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. > > +\end{description} > > + > > +The device responds a capability by using \field{struct > > +virtio_video_get_capability_resp}. > > +\begin{lstlisting} > > +enum virtio_video_format { > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > Really minor: maybe you want to insert an empty line to be coherent > with other enumerations? Yes. Thanks. > > > + /* Raw formats */ > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > + VIRTIO_VIDEO_FORMAT_YUV420, > > + VIRTIO_VIDEO_FORMAT_YVU420, > > + > > + /* Compressed formats */ > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > +}; > > + > > +enum virtio_video_profile { > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > + > > + /* H.264 */ > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, > > Maybe you want > > VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_MIN, > > like others? Thanks for catching this. I think we can find this kind of errors once we update the driver implementation. > > > + VIRTIO_VIDEO_PROFILE_H264_MAIN, > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > + > > + /* VP8 */ > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > > + > > + /* VP9 */ > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > +}; > > + > > +struct virtio_video_format_range { > > + le32 min; > > + le32 max; > > + le32 step; > > + u8 paddings[4]; > > Here you padded at 8-byte like many other structures but not for > virtio_video_ctrl_hdr. Is this expected? I padded here for 64-bits alignment. So, I should have padded for ctrl_hdr as you said. But, no need anymore as we will remove |len| from the struct as you suggested above:) > > > +}; > > + > > +struct virtio_video_format_desc { > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > > + le64 mask; > > This will limit the formats to 64 while "num_descs" below is a > 32 bit. It's my intention. I will explicitly write that "num_desc" must not exceed 64. I guess 64 is big enough as a number of supported input/output format and couldn't find a better idea to represent this bipartite graph. If you have another idea, please let me hear. > > > + struct virtio_video_format_range width; > > + struct virtio_video_format_range height; > > + le32 num_rates; > > + u8 padding[4]; > > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > > +}; > > + > > +struct virtio_video_get_capability_resp { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 num_descs; > > + /* Followed by struct virtio_video_format_desc desc[] */ > > +}; > > +\end{lstlisting} > > + > > +The format description \field{struct virtio_video_format_desc} > > +includes the following fields: > > +\begin{description} > > +\item[\field{format}] specifies an image format. The device MUST set one > > + of \field{enum virtio_video_format}. > > +\item[\field{profile}] specifies a profile of the compressed image format > > + specified in \field{format}. The driver SHOULD ignore this value if > > + \field{format} is a raw format. > > Why not documenting that MUST be VIRTIO_VIDEO_PROFILE_UNDEFINED for > raw formats so the field could be extended in the future? Sounds good. Will do. > > > +\item[\field{mask}] is a bitset that represents the supported > > + combination of input and output format. If \textit{i}-th bit is set > > + in \field{mask} of \textit{j}-th \field{struct > > + virtio_video_format_desc} for input, the device supports encoding or > > + decoding from the \textit{j}-th input format to \textit{i}-th output > > + format. > > +\item[\field{width, height}] represents a range of resolutions > > + supported by the device. If its \field{step} is not applicable, its > > + \field{min} is equal to its \field{max}. > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In > > case of decoder, the driver SHOULD ignore this value. > > +\item[\field{frame_rates}] is an array of supported frame rates. > > I suppose here we are talking about bitrates, right? Could be confused by > FPS. Nope. This is for frame rates or frame intervals. In V4L2 driver, the values will be used for VIDIOC_ENUM_FRAMEINTERVALS ioctl. (Dmitry, please correct me if I am wrong) > > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > > + within the device. > > + > > +\begin{lstlisting} > > +enum virtio_video_mem_type { > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > +}; > > + > > +struct virtio_video_stream_create { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > + char debug_name[64]; > > +}; > > This structure has a size not multiple of 8-bytes. Not an issue, but > the same apply to other structures that instead are padded to 8-byte size. It's my fault. I tried to add paddings to make all structs' size multiple of 8-bytes, I overlooked some. > > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{in_mem_type}] is a type of buffer management for input > > +buffers. The driver MUST set a value in \field{enum > > +virtio_video_mem_type}. > > +\item[\field{out_mem_type}] is a type of buffer management for output > > +buffers. The driver MUST set a value in \field{enum > > +virtio_video_mem_type}. > > +\item[\field{debug_name}] is a text string for a debug purpose. > > Must be NUL-terminated ? UTF-8 ? Yeah, I will add a description similar to one for "tag" in virtio_fs_config. > > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_stream_destroy { > > + struct virtio_video_ctrl_hdr hdr; > > +}; > > +\end{lstlisting} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > > + queued buffers through the pipeline. > > + > > +\begin{lstlisting} > > +struct virtio_video_stream_drain { > > + struct virtio_video_ctrl_hdr hdr; > > +}; > > +\end{lstlisting} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_mem_entry { > > + le64 addr; > > + le32 length; > > + u8 padding[4]; > > +}; > > + > > +struct virtio_video_resource_create { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 resource_id; > > + le32 nr_entries; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{resource_id}] internal id of the resource. > > Is it input or output? Will add a field for it. > > > +\item[\field{nr_entries}] number of \field{struct > > + virtio_video_mem_entry} memory entries. > > I suppose that the structure is followed by an array of > virtio_video_mem_entry after a padding of 4-bytes for > alignment. > > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_resource_destroy { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 resource_id; > > + u8 padding[4]; > > Is this padding for future extensions? Nope, it's mistake. When I was editing structs, I forgot to add/remove padding. I hope we will not have such mistakes after we have an updated driver implementation. > > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{resource_id}] internal id of the resource. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > > +queue. > > + > > +\begin{lstlisting} > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > + > > +struct virtio_video_resource_queue { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 buf_type; > > + le32 resource_id; > > + le64 timestamp; > > I suppose you would like this field aligned to avoid > hidden paddings. Thanks. Best regards, Keiichi > > > + le32 nr_data_size; > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{buf_type}] buf_type of the . > > +\item[\field{resource_id}] internal id of the resource. > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > + for synchronisation. > > +\item[\field{nr_data_size}] number of \field{data_size} entries. > > +\item[\field{data_size}] number of data bytes within a plane. > > +\end{description} > > + > ... omissis ... > > -- > > 2.24.1.735.g03f4e72817-goog > > > > > > Frediano >
Hi Dmitry, Thanks for the feedback. On Thu, Dec 19, 2019 at 10:28 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi Keiichi, > > Thank you for the update. Please see some comments below. > > Also, we need to bring the virtio_video_control back as it is in fact used by > the driver to enumerate supported encoder controls. But yes, it still needs to > be documemnted, it's true. Yeah, we need it eventually, but I'm wondering what is the difference between values set by SET_PARAMS and ones by controls. I saw your driver implementation has BITRATE, PROFILE and LEVEL as controls. What's the reason why you didn't add them in virtio_video_params? > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > The virtio video encoder device and decoder device provide functionalities > > to encode and decode video stream respectively. > > Though video encoder and decoder are provided as different devices, they use > > a same protocol. > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > --- > > content.tex | 1 + > > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > > 2 files changed, 580 insertions(+) > > create mode 100644 virtio-video.tex > > > > diff --git a/content.tex b/content.tex > > index 556b373..9e56839 100644 > > --- a/content.tex > > +++ b/content.tex > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > Requirements}\label{sec:Device \input{virtio-vsock.tex} > > \input{virtio-fs.tex} > > \input{virtio-rpmb.tex} > > +\input{virtio-video.tex} > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > new file mode 100644 > > index 0000000..30e728d > > --- /dev/null > > +++ b/virtio-video.tex > > @@ -0,0 +1,579 @@ > > +\section{Video Device}\label{sec:Device Types / Video Device} > > + > > +The virtio video encoder device and decoder device are virtual devices that > > +supports encoding and decoding respectively. Though the encoder and the > > decoder +are different devices, they use the same protocol. > > + > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID} > > + > > +\begin{description} > > +\item[30] encoder device > > +\item[31] decoder device > > +\end{description} > > + > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues} > > + > > +\begin{description} > > +\item[0] controlq - queue for sending control commands. > > +\item[1] eventq - queue for sending events happened in the device. > > +\end{description} > > + > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature > > bits} + > > +\begin{description} > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for > > video + buffers. > > +\end{description} > > + > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device > > / Feature bits} + > > +The device MUST offer at least one of feature bits. > > + > > +\subsection{Device configuration layout}\label{sec:Device Types / Video > > Device / Device configuration layout} + > > +Video device configuration uses the following layout structure: > > + > > +\begin{lstlisting} > > +struct virtio_video_config { > > + le32 max_cap_len; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > > + MUST set this value. > > +\end{description} > > + > > +\subsection{Device Initialization}\label{sec:Device Types / Video Device / > > Device Initialization} + > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / > > Video Device / Device Initialization} + > > +The driver SHOULD query device capability by using the > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > > +setup. > > + > > +\subsection{Device Operation}\label{sec:Device Types / Video Device / > > Device Operation} + > > +The driver allocates input and output buffers and queues the buffers > > +to the device. The device performs operations on the buffers according > > +to the function in question. > > + > > +\subsubsection{Device Operation: Create stream} > > + > > +To process buffers, the device needs to associate them with a certain > > +video stream (essentially, a context). Streams are created by > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > +determined by the device. > > + > > +\subsubsection{Device Operation: Create buffers} > > + > > +Buffers are used to store the actual data as well as the relevant > > +metadata. Scatter lists are supported, so the buffer doesn't need to > > +be contiguous in guest physical memory. > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > > + resource that is backed by a buffer allocated from the driver's > > + memory. > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > > + is no longer needed. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Stream parameter control} > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters > > for + input and output streams from the device. > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the > > + device. > > +\item After setting stream parameters, the driver may issue > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can > > be + changed implicitly by the device during the set operation. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Process buffers} > > + > > +\begin{itemize*} > > +\item If the function and the buffer type require so, write data to > > +the buffer memory. > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > > +processing in the device. > > +\item The request completes asynchronously when the device has > > +finished with the buffer. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Buffer processing control} > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > > + return all of the already queued buffers. > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > > + already queued buffers from the input or the output queue. This also > > + includes input or output buffers that can be currently owned by the > > + device's processing pipeline. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Asynchronous events} > > + > > +While processing buffers, the device can send asynchronous event > > +notifications to the driver. The behaviour depends on the exact > > +stream. For example, the decoder device sends a resolution change > > +event when it encounters new resolution metadata in the stream. > > + > > +\subsubsection{Device Operation: Request header} > > + > > +All requests and responses on the control virt queue have a fixed > > +header using the following layout structure and definitions: > > + > > +\begin{lstlisting} > > +enum virtio_video_ctrl_type { > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > + > > + /* request */ > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > + VIRTIO_VIDEO_T_SET_PARAMS, > > + VIRTIO_VIDEO_T_GET_PARAMS, > > + > > + /* response */ > > + VIRTIO_VIDEO_S_OK = 0x0200, > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > + > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > +}; > > + > > +struct virtio_video_ctrl_hdr { > > + le32 type; > > + le32 stream_id; > > + le32 len; /* Length of the structure in bytes. */ > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{type}] is the type of the driver request or the device > > +response. > > +\item[\field{stream_id}] specifies a target stream. > > +\item[\field{len}] is the length of data in bytes, which includes > > +length of the header. > > +\end{description} > > + > > +\subsubsection{Device Operation: controlq} > > + > > +\begin{description} > > + > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > > +supported formats. > > + > > +The driver uses \field{struct virtio_video_get_capability} to send a > > +query request. > > + > > +\begin{lstlisting} > > +enum virtio_video_buf_type { > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > +}; > I personally didn't like the previous term: pin_type. But, to be honest, I > don't like the buf_type neither. Consider the GET/SET_PARAMS request: buf_type > there looks a bit unnatural. We are trying to get stream parameters there, not > some parameters of whatever buffer. Also I don't see any strict reason to mimic > v4l2 naming scheme. > > I'd better rename it to PORT_TYPE or QUEUE_TYPE. To be honest, I also wasn't satisfied with the name "buf_type". QUEUE_TYPE sounds better. Will rename it. > > > + > > +struct virtio_video_get_capability { > > + struct virtio_video_ctrl_hdr hdr; > > + enum virtio_video_buf_type buf_type; > > +}; > > +\end{lstlisting} > > +\begin{description} > > +\item[\field{buf_type}] is the buffer type that the driver asks > > +information about. The driver MUST set either > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > + > > +The device responds a capability by using \field{struct > > +virtio_video_get_capability_resp}. > > +\begin{lstlisting} > > +enum virtio_video_format { > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > + /* Raw formats */ > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > + VIRTIO_VIDEO_FORMAT_YUV420, > > + VIRTIO_VIDEO_FORMAT_YVU420, > Let's add some variants of RGB, like RGBA, ARGB. We need it for the encoder in > particular . > Sounds good. BTW, which "ARGB8888" or "A8R8G8B8" is preferred? While the first one comes from DRM's FourCC, the second one comes from virtio_gpu_formats. I personally prefer the first one, then we can have a naming convention like: VIRTIO_VIDEO_FORMAT_<name from drm_fourcc.h> > > + > > + /* Compressed formats */ > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > Let's add H265, MPEG4, MPEG2. We already support and use them. Sure. Best regards, Keiichi > > Regards, > Dmitry. > > > +}; > > + > > +enum virtio_video_profile { > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > + > > + /* H.264 */ > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > > VIRTIO_VIDEO_PROFILE_H264_MAIN, > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > > + /* VP8 */ > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > > + > > + /* VP9 */ > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > +}; > > + > > +struct virtio_video_format_range { > > + le32 min; > > + le32 max; > > + le32 step; > > + u8 paddings[4]; > > +}; > > + > > +struct virtio_video_format_desc { > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > > + le64 mask; > > + struct virtio_video_format_range width; > > + struct virtio_video_format_range height; > > + le32 num_rates; > > + u8 padding[4]; > > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > > +}; > > + > > +struct virtio_video_get_capability_resp { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 num_descs; > > + /* Followed by struct virtio_video_format_desc desc[] */ > > +}; > > +\end{lstlisting} > > + > > +The format description \field{struct virtio_video_format_desc} > > +includes the following fields: > > +\begin{description} > > +\item[\field{format}] specifies an image format. The device MUST set one > > + of \field{enum virtio_video_format}. > > +\item[\field{profile}] specifies a profile of the compressed image format > > + specified in \field{format}. The driver SHOULD ignore this value if > > + \field{format} is a raw format. > > +\item[\field{mask}] is a bitset that represents the supported > > + combination of input and output format. If \textit{i}-th bit is set > > + in \field{mask} of \textit{j}-th \field{struct > > + virtio_video_format_desc} for input, the device supports encoding or > > + decoding from the \textit{j}-th input format to \textit{i}-th output > > + format. > > +\item[\field{width, height}] represents a range of resolutions > > + supported by the device. If its \field{step} is not applicable, its > > + \field{min} is equal to its \field{max}. > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In > > case of decoder, the driver SHOULD ignore this value. > > +\item[\field{frame_rates}] is an array of supported frame rates. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > > + within the device. > > + > > +\begin{lstlisting} > > +enum virtio_video_mem_type { > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > +}; > > + > > +struct virtio_video_stream_create { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > + char debug_name[64]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{in_mem_type}] is a type of buffer management for input > > +buffers. The driver MUST set a value in \field{enum > > +virtio_video_mem_type}. > > +\item[\field{out_mem_type}] is a type of buffer management for output > > +buffers. The driver MUST set a value in \field{enum > > +virtio_video_mem_type}. > > +\item[\field{debug_name}] is a text string for a debug purpose. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_stream_destroy { > > + struct virtio_video_ctrl_hdr hdr; > > +}; > > +\end{lstlisting} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > > + queued buffers through the pipeline. > > + > > +\begin{lstlisting} > > +struct virtio_video_stream_drain { > > + struct virtio_video_ctrl_hdr hdr; > > +}; > > +\end{lstlisting} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_mem_entry { > > + le64 addr; > > + le32 length; > > + u8 padding[4]; > > +}; > > + > > +struct virtio_video_resource_create { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 resource_id; > > + le32 nr_entries; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{resource_id}] internal id of the resource. > > +\item[\field{nr_entries}] number of \field{struct > > + virtio_video_mem_entry} memory entries. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_resource_destroy { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 resource_id; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{resource_id}] internal id of the resource. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > > +queue. > > + > > +\begin{lstlisting} > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > + > > +struct virtio_video_resource_queue { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 buf_type; > > + le32 resource_id; > > + le64 timestamp; > > + le32 nr_data_size; > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{buf_type}] buf_type of the . > > +\item[\field{resource_id}] internal id of the resource. > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > + for synchronisation. > > +\item[\field{nr_data_size}] number of \field{data_size} entries. > > +\item[\field{data_size}] number of data bytes within a plane. > > +\end{description} > > + > > +\begin{lstlisting} > > +enum virtio_video_buffer_flag { > > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > > + /* Encoder only */ > > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > > +}; > > + > > +struct virtio_video_resource_queue_resp { > > + struct virtio_video_ctrl_hdr hdr; > > + le64 timestamp; > > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > > + le32 size; /* Encoded size */ > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > + for synchronisation. > > +\item[\field{flags}] mark specific buffers in the sequence. > > +\item[\field{size}] data size in the buffer (encoder only). > > +\end{description} > > + > > +The device sends a response to the queue request asynchronously when > > +it has finished processing the buffer. > > + > > +The device SHOULD mark a buffer that triggered a processing error with > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > > + > > +The device MUST mark the last buffer with the > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > +sequence. > > + > > +In case of encoder, to denote a particular frame type the devie MUST > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > > + buffers back from the input or the output queue of the device. The > > + device SHOULD return all of the buffers from the respective queue as > > + soon as possible without pushing the buffers through the processing > > + pipeline. > > + > > +\begin{lstlisting} > > +struct virtio_video_queue_clear { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 buf_type; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{buf_type}] buffer type. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > > + output of a stream. > > + > > +\begin{lstlisting} > > +struct virtio_video_plane_format { > > + le32 plane_size; > > + le32 stride; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{plane_size}] size of the plane in bytes. > > +\item[\field{stride}] stride used for the plane in bytes. > > +\end{description} > > + > > +\begin{lstlisting} > > +struct virtio_video_params { > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > > + le32 frame_width; > > + le32 frame_height; > > + le32 min_buffers; > > + le32 max_buffers; > > + le32 frame_rate; > > + struct virtio_video_crop { > > + le32 left; > > + le32 top; > > + le32 width; > > + le32 height; > > + } crop; > > + le32 num_planes; > > + struct virtio_video_plane_format > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{frame_width}] the value to get/set. > > +\item[\field{frame_height}] the value to get/set. > > +\item[\field{pixel_format}] the value to get/set. > > +\item[\field{min_buffers}] minimum buffers required to handle the > > + format (r/o). > > +\item[\field{max_buffers}] maximum buffers required to handle the > > + format (r/o). > > +\item[\field{frame_rate}] the value to get/set. > > +\item[\field{crop}] cropping (composing) rectangle. > > +\item[\field{num_planes}] number of planes used to store pixel data > > +(r/o). > > +\item[\field{plane_formats}] description of each plane. > > +\end{description} > > + > > +\begin{lstlisting} > > +struct virtio_video_get_params { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > +}; > > + > > +struct virtio_video_get_params_resp { > > + struct virtio_video_ctrl_hdr hdr; > > + struct virtio_video_params params; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{buf_type}] buffer type. > > +\item[\field{params}] parameter values. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > > + > > + > > +\begin{lstlisting} > > +struct virtio_video_set_params { > > + struct virtio_video_ctrl_hdr hdr; > > + struct virtio_video_params params; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{params}] parameters to set. > > +\end{description} > > + > > +Setting stream parameters might have side effects within the device. > > +For example, the device MAY perform alignment of width and height, > > +change the number of planes it uses for the format, or do whatever > > +changes that are required to continue normal operation using the > > +updated parameters. It is up to the driver to check the parameter set > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > > + > > +\end{description} > > + > > +\subsubsection{Device Operation: eventq} > > + > > +The device can report events on the event queue. The driver initially > > +populates the queue with device-writeable buffers. When the device > > +needs to report an event, it fills a buffer and notifies the driver. > > +The driver consumes the report and adds a new buffer to the virtqueue. > > + > > +\begin{lstlisting} > > +enum virtio_video_event_type { > > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > > + /* For all functions */ > > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > > + /* For decoder only */ > > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > > +}; > > + > > +struct virtio_video_event { > > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > > + le32 stream_id; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{event_type}] type of the triggered event . > > +\item[\field{stream_id}] id of the source stream. > > +\end{description} > > + > > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > > +whenever it encounters new resolution data in the stream. This > > +includes the case of the initial device configuration after metadata > > +has been parsed and the case of dynamic resolution change. > >
Hi Keiichi, On Freitag, 20. Dezember 2019 15:05:02 CET Keiichi Watanabe wrote: > Hi Frediano, > Thanks for reviewing! > > On Thu, Dec 19, 2019 at 2:29 AM Frediano Ziglio <fziglio@redhat.com> wrote: > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > The virtio video encoder device and decoder device provide > > > functionalities to encode and decode video stream respectively. > > > Though video encoder and decoder are provided as different devices, they > > > use a > > > same protocol. > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > > --- > > > > > > content.tex | 1 + > > > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > > > 2 files changed, 580 insertions(+) > > > create mode 100644 virtio-video.tex > > > > > > diff --git a/content.tex b/content.tex > > > index 556b373..9e56839 100644 > > > --- a/content.tex > > > +++ b/content.tex > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > > Requirements}\label{sec:Device > > > > > > \input{virtio-vsock.tex} > > > \input{virtio-fs.tex} > > > \input{virtio-rpmb.tex} > > > > > > +\input{virtio-video.tex} > > > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > > new file mode 100644 > > > index 0000000..30e728d > > > --- /dev/null > > > +++ b/virtio-video.tex > > > @@ -0,0 +1,579 @@ > > > +\section{Video Device}\label{sec:Device Types / Video Device} > > > + > > > +The virtio video encoder device and decoder device are virtual devices > > > that +supports encoding and decoding respectively. Though the encoder > > > and the decoder > > > +are different devices, they use the same protocol. > > > + > > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device > > > ID} > > > + > > > +\begin{description} > > > +\item[30] encoder device > > > +\item[31] decoder device > > > +\end{description} > > > + > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / > > > Virtqueues} + > > > +\begin{description} > > > +\item[0] controlq - queue for sending control commands. > > > +\item[1] eventq - queue for sending events happened in the device. > > > +\end{description} > > > + > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / > > > Feature > > > bits} > > > + > > > +\begin{description} > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used > > > for > > > video > > > + buffers. > > > +\end{description} > > > + > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video > > > Device / Feature bits} > > > + > > > +The device MUST offer at least one of feature bits. > > > + > > > +\subsection{Device configuration layout}\label{sec:Device Types / Video > > > Device / Device configuration layout} > > > + > > > +Video device configuration uses the following layout structure: > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_config { > > > + le32 max_cap_len; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > > > + MUST set this value. > > > +\end{description} > > > + > > > +\subsection{Device Initialization}\label{sec:Device Types / Video > > > Device / > > > Device Initialization} > > > + > > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / > > > Video Device / Device Initialization} > > > + > > > +The driver SHOULD query device capability by using the > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > > > +setup. > > > + > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device / > > > Device Operation} > > > + > > > +The driver allocates input and output buffers and queues the buffers > > > +to the device. The device performs operations on the buffers according > > > +to the function in question. > > > + > > > +\subsubsection{Device Operation: Create stream} > > > + > > > +To process buffers, the device needs to associate them with a certain > > > +video stream (essentially, a context). Streams are created by > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > > +determined by the device. > > > + > > > +\subsubsection{Device Operation: Create buffers} > > > + > > > +Buffers are used to store the actual data as well as the relevant > > > +metadata. Scatter lists are supported, so the buffer doesn't need to > > > +be contiguous in guest physical memory. > > > + > > > +\begin{itemize*} > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > > > + resource that is backed by a buffer allocated from the driver's > > > + memory. > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > > > + is no longer needed. > > > +\end{itemize*} > > > + > > > +\subsubsection{Device Operation: Stream parameter control} > > > + > > > +\begin{itemize*} > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream > > > parameters for + input and output streams from the device. > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to > > > the + device. > > > +\item After setting stream parameters, the driver may issue > > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output > > > can be > > > + changed implicitly by the device during the set operation. > > > +\end{itemize*} > > > + > > > +\subsubsection{Device Operation: Process buffers} > > > + > > > +\begin{itemize*} > > > +\item If the function and the buffer type require so, write data to > > > +the buffer memory. > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > > > +processing in the device. > > > +\item The request completes asynchronously when the device has > > > +finished with the buffer. > > > +\end{itemize*} > > > + > > > +\subsubsection{Device Operation: Buffer processing control} > > > + > > > +\begin{itemize*} > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > > > + return all of the already queued buffers. > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > > > + already queued buffers from the input or the output queue. This also > > > + includes input or output buffers that can be currently owned by the > > > + device's processing pipeline. > > > +\end{itemize*} > > > + > > > +\subsubsection{Device Operation: Asynchronous events} > > > + > > > +While processing buffers, the device can send asynchronous event > > > +notifications to the driver. The behaviour depends on the exact > > > +stream. For example, the decoder device sends a resolution change > > > +event when it encounters new resolution metadata in the stream. > > > + > > > +\subsubsection{Device Operation: Request header} > > > + > > > +All requests and responses on the control virt queue have a fixed > > > +header using the following layout structure and definitions: > > > + > > > +\begin{lstlisting} > > > +enum virtio_video_ctrl_type { > > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > > + > > > + /* request */ > > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > > + VIRTIO_VIDEO_T_SET_PARAMS, > > > + VIRTIO_VIDEO_T_GET_PARAMS, > > > + > > > + /* response */ > > > + VIRTIO_VIDEO_S_OK = 0x0200, > > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > > + > > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > > +}; > > > + > > > +struct virtio_video_ctrl_hdr { > > > + le32 type; > > > + le32 stream_id; > > > + le32 len; /* Length of the structure in bytes. */ > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{type}] is the type of the driver request or the device > > > +response. > > > +\item[\field{stream_id}] specifies a target stream. > > > +\item[\field{len}] is the length of data in bytes, which includes > > > +length of the header. > > > > I suppose is implicit that it's the size of the container structure. > > You're right. All struct except virtio_video_get_capability_resp have > fixed size and virtio_video_get_capability_resp has num_descs. So, no > need to have len. > Let me remove this in the next iteration. > > > > +\end{description} > > > + > > > +\subsubsection{Device Operation: controlq} > > > + > > > +\begin{description} > > > + > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > > > +supported formats. > > > + > > > +The driver uses \field{struct virtio_video_get_capability} to send a > > > +query request. > > > + > > > +\begin{lstlisting} > > > +enum virtio_video_buf_type { > > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > > +}; > > > + > > > +struct virtio_video_get_capability { > > > + struct virtio_video_ctrl_hdr hdr; > > > + enum virtio_video_buf_type buf_type; > > > > Here you used an enumeration but later you are using just le32 type > > with a comment. Why this difference? > > No difference. I will change this to use le32 since virtio-gpu's spec does > so. > > > +}; > > > +\end{lstlisting} > > > +\begin{description} > > > +\item[\field{buf_type}] is the buffer type that the driver asks > > > +information about. The driver MUST set either > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > > + > > > +The device responds a capability by using \field{struct > > > +virtio_video_get_capability_resp}. > > > +\begin{lstlisting} > > > +enum virtio_video_format { > > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > > > Really minor: maybe you want to insert an empty line to be coherent > > with other enumerations? > > Yes. Thanks. > > > > + /* Raw formats */ > > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > > + VIRTIO_VIDEO_FORMAT_YUV420, > > > + VIRTIO_VIDEO_FORMAT_YVU420, > > > + > > > + /* Compressed formats */ > > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > > +}; > > > + > > > +enum virtio_video_profile { > > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > > + > > > + /* H.264 */ > > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, > > > > Maybe you want > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_MIN, > > > > like others? > > Thanks for catching this. I think we can find this kind of errors once > we update the driver implementation. > > > > + VIRTIO_VIDEO_PROFILE_H264_MAIN, > > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > > + > > > + /* VP8 */ > > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > > > + > > > + /* VP9 */ > > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = > > > VIRTIO_VIDEO_PROFILE_VP9_MIN, > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > +}; > > > + > > > +struct virtio_video_format_range { > > > + le32 min; > > > + le32 max; > > > + le32 step; > > > + u8 paddings[4]; > > > > Here you padded at 8-byte like many other structures but not for > > virtio_video_ctrl_hdr. Is this expected? > > I padded here for 64-bits alignment. So, I should have padded for > ctrl_hdr as you said. But, no need anymore as we will remove |len| > from the struct as you suggested above:) > > > > +}; > > > + > > > +struct virtio_video_format_desc { > > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > > > + le64 mask; > > > > This will limit the formats to 64 while "num_descs" below is a > > 32 bit. > > It's my intention. I will explicitly write that "num_desc" must not exceed > 64. I guess 64 is big enough as a number of supported input/output format > and couldn't find a better idea to represent this bipartite graph. If you > have another idea, please let me hear. > > > > + struct virtio_video_format_range width; > > > + struct virtio_video_format_range height; > > > + le32 num_rates; > > > + u8 padding[4]; > > > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > > > +}; > > > + > > > +struct virtio_video_get_capability_resp { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 num_descs; > > > + /* Followed by struct virtio_video_format_desc desc[] */ > > > +}; > > > +\end{lstlisting} > > > + > > > +The format description \field{struct virtio_video_format_desc} > > > +includes the following fields: > > > +\begin{description} > > > +\item[\field{format}] specifies an image format. The device MUST set > > > one > > > + of \field{enum virtio_video_format}. > > > +\item[\field{profile}] specifies a profile of the compressed image > > > format > > > + specified in \field{format}. The driver SHOULD ignore this value if > > > + \field{format} is a raw format. > > > > Why not documenting that MUST be VIRTIO_VIDEO_PROFILE_UNDEFINED for > > raw formats so the field could be extended in the future? > > Sounds good. Will do. > > > > +\item[\field{mask}] is a bitset that represents the supported > > > + combination of input and output format. If \textit{i}-th bit is set > > > + in \field{mask} of \textit{j}-th \field{struct > > > + virtio_video_format_desc} for input, the device supports encoding or > > > + decoding from the \textit{j}-th input format to \textit{i}-th output > > > + format. > > > +\item[\field{width, height}] represents a range of resolutions > > > + supported by the device. If its \field{step} is not applicable, its > > > + \field{min} is equal to its \field{max}. > > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. > > > In > > > case of decoder, the driver SHOULD ignore this value. > > > +\item[\field{frame_rates}] is an array of supported frame rates. > > > > I suppose here we are talking about bitrates, right? Could be confused by > > FPS. > > Nope. This is for frame rates or frame intervals. In V4L2 driver, the > values will be used for VIDIOC_ENUM_FRAMEINTERVALS ioctl. > (Dmitry, please correct me if I am wrong) Right, this is correct. Regards, Dmitry. > > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > > > + within the device. > > > + > > > +\begin{lstlisting} > > > +enum virtio_video_mem_type { > > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > > +}; > > > + > > > +struct virtio_video_stream_create { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > > + char debug_name[64]; > > > +}; > > > > This structure has a size not multiple of 8-bytes. Not an issue, but > > the same apply to other structures that instead are padded to 8-byte size. > > It's my fault. I tried to add paddings to make all structs' size > multiple of 8-bytes, I overlooked some. > > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{in_mem_type}] is a type of buffer management for input > > > +buffers. The driver MUST set a value in \field{enum > > > +virtio_video_mem_type}. > > > +\item[\field{out_mem_type}] is a type of buffer management for output > > > +buffers. The driver MUST set a value in \field{enum > > > +virtio_video_mem_type}. > > > +\item[\field{debug_name}] is a text string for a debug purpose. > > > > Must be NUL-terminated ? UTF-8 ? > > Yeah, I will add a description similar to one for "tag" in virtio_fs_config. > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > > > + within the device. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_stream_destroy { > > > + struct virtio_video_ctrl_hdr hdr; > > > +}; > > > +\end{lstlisting} > > > + > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > > > + queued buffers through the pipeline. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_stream_drain { > > > + struct virtio_video_ctrl_hdr hdr; > > > +}; > > > +\end{lstlisting} > > > + > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > > > + within the device. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_mem_entry { > > > + le64 addr; > > > + le32 length; > > > + u8 padding[4]; > > > +}; > > > + > > > +struct virtio_video_resource_create { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 resource_id; > > > + le32 nr_entries; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{resource_id}] internal id of the resource. > > > > Is it input or output? > > Will add a field for it. > > > > +\item[\field{nr_entries}] number of \field{struct > > > + virtio_video_mem_entry} memory entries. > > > > I suppose that the structure is followed by an array of > > virtio_video_mem_entry after a padding of 4-bytes for > > alignment. > > > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > > > + within the device. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_resource_destroy { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 resource_id; > > > + u8 padding[4]; > > > > Is this padding for future extensions? > > Nope, it's mistake. When I was editing structs, I forgot to add/remove > padding. I hope we will not have such mistakes after we have an > updated driver implementation. > > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{resource_id}] internal id of the resource. > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > > > +queue. > > > + > > > +\begin{lstlisting} > > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > > + > > > +struct virtio_video_resource_queue { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 buf_type; > > > + le32 resource_id; > > > + le64 timestamp; > > > > I suppose you would like this field aligned to avoid > > hidden paddings. > > Thanks. > > Best regards, > Keiichi > > > > + le32 nr_data_size; > > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{buf_type}] buf_type of the . > > > +\item[\field{resource_id}] internal id of the resource. > > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > > + for synchronisation. > > > +\item[\field{nr_data_size}] number of \field{data_size} entries. > > > +\item[\field{data_size}] number of data bytes within a plane. > > > +\end{description} > > > + > > > > ... omissis ... > > > > > -- > > > 2.24.1.735.g03f4e72817-goog > > > > Frediano
On Sat, Dec 21, 2019 at 12:46 AM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > On Freitag, 20. Dezember 2019 16:26:50 CET Keiichi Watanabe wrote: > > On Thu, Dec 19, 2019 at 10:28 PM Dmitry Sepp > > <dmitry.sepp@opensynergy.com> wrote: > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: [snip] > > > > +enum virtio_video_format { > > > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > > > + /* Raw formats */ > > > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > > > + VIRTIO_VIDEO_FORMAT_YUV420, > > > > + VIRTIO_VIDEO_FORMAT_YVU420, > > > > > > Let's add some variants of RGB, like RGBA, ARGB. We need it for the > > > encoder in particular . > > > > Sounds good. > > BTW, which "ARGB8888" or "A8R8G8B8" is preferred? While the first one > > comes from DRM's FourCC, the second one comes from virtio_gpu_formats. > > I personally prefer the first one, then we can have a naming convention > > like: VIRTIO_VIDEO_FORMAT_<name from drm_fourcc.h> > > > > I'd go with ARGB8888 and BGRA8888 (might be with X variants). Just to make sure we're talking about the same formats. DRM naming convention is based on the little endian convention, which for 32-bit formats means that you interpret the whole pixel as a packed 32-bit word on a little endian system. For ARGB8888 that would mean (bit 31) ARGB (bit 0) in the 32-bit word and (byte 0) B, G, R, A (byte 3) when looking at separate bytes in memory. Does that correspond to your expected format? We also have to be specific about the A and X formats, as the A format should be supported only if the hardware (host) doesn't ignore the alpha channel. I haven't seen any hardware capable of encoding alpha channel yet, but apparently for WebM the standard is to just encode the alpha channel into another stream as Y, together with dummy U and V values. [1] That sounds like something that would be handled by two separate encoding streams and not just one that accepts RGBA on the input. [1] http://wiki.webmproject.org/alpha-channel Best regards, Tomasz
On Donnerstag, 19. Dezember 2019 14:28:23 CET Dmitry Sepp wrote: > Hi Keiichi, > > Thank you for the update. Please see some comments below. > > Also, we need to bring the virtio_video_control back as it is in fact used > by the driver to enumerate supported encoder controls. But yes, it still > needs to be documemnted, it's true. > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > The virtio video encoder device and decoder device provide functionalities > > to encode and decode video stream respectively. > > Though video encoder and decoder are provided as different devices, they > > use a same protocol. > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > --- > > > > content.tex | 1 + > > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > > 2 files changed, 580 insertions(+) > > create mode 100644 virtio-video.tex > > > > diff --git a/content.tex b/content.tex > > index 556b373..9e56839 100644 > > --- a/content.tex > > +++ b/content.tex > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > Requirements}\label{sec:Device \input{virtio-vsock.tex} > > > > \input{virtio-fs.tex} > > \input{virtio-rpmb.tex} > > > > +\input{virtio-video.tex} > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > new file mode 100644 > > index 0000000..30e728d > > --- /dev/null > > +++ b/virtio-video.tex > > @@ -0,0 +1,579 @@ > > +\section{Video Device}\label{sec:Device Types / Video Device} > > + > > +The virtio video encoder device and decoder device are virtual devices > > that +supports encoding and decoding respectively. Though the encoder and > > the decoder +are different devices, they use the same protocol. > > + > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID} > > + > > +\begin{description} > > +\item[30] encoder device > > +\item[31] decoder device > > +\end{description} > > + > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / > > Virtqueues} + > > +\begin{description} > > +\item[0] controlq - queue for sending control commands. > > +\item[1] eventq - queue for sending events happened in the device. > > +\end{description} > > + > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature > > bits} + > > +\begin{description} > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used > > for > > video + buffers. > > +\end{description} > > + > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video > > Device > > / Feature bits} + > > +The device MUST offer at least one of feature bits. > > + > > +\subsection{Device configuration layout}\label{sec:Device Types / Video > > Device / Device configuration layout} + > > +Video device configuration uses the following layout structure: > > + > > +\begin{lstlisting} > > +struct virtio_video_config { > > + le32 max_cap_len; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > > + MUST set this value. > > +\end{description} > > + > > +\subsection{Device Initialization}\label{sec:Device Types / Video Device > > / > > Device Initialization} + > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / > > Video Device / Device Initialization} + > > +The driver SHOULD query device capability by using the > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > > +setup. > > + > > +\subsection{Device Operation}\label{sec:Device Types / Video Device / > > Device Operation} + > > +The driver allocates input and output buffers and queues the buffers > > +to the device. The device performs operations on the buffers according > > +to the function in question. > > + > > +\subsubsection{Device Operation: Create stream} > > + > > +To process buffers, the device needs to associate them with a certain > > +video stream (essentially, a context). Streams are created by > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > +determined by the device. > > + > > +\subsubsection{Device Operation: Create buffers} > > + > > +Buffers are used to store the actual data as well as the relevant > > +metadata. Scatter lists are supported, so the buffer doesn't need to > > +be contiguous in guest physical memory. > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > > + resource that is backed by a buffer allocated from the driver's > > + memory. > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > > + is no longer needed. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Stream parameter control} > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters > > for + input and output streams from the device. > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to > > the + device. > > +\item After setting stream parameters, the driver may issue > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output > > can be + changed implicitly by the device during the set operation. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Process buffers} > > + > > +\begin{itemize*} > > +\item If the function and the buffer type require so, write data to > > +the buffer memory. > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > > +processing in the device. > > +\item The request completes asynchronously when the device has > > +finished with the buffer. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Buffer processing control} > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > > + return all of the already queued buffers. > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > > + already queued buffers from the input or the output queue. This also > > + includes input or output buffers that can be currently owned by the > > + device's processing pipeline. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Asynchronous events} > > + > > +While processing buffers, the device can send asynchronous event > > +notifications to the driver. The behaviour depends on the exact > > +stream. For example, the decoder device sends a resolution change > > +event when it encounters new resolution metadata in the stream. > > + > > +\subsubsection{Device Operation: Request header} > > + > > +All requests and responses on the control virt queue have a fixed > > +header using the following layout structure and definitions: > > + > > +\begin{lstlisting} > > +enum virtio_video_ctrl_type { > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > + > > + /* request */ > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > + VIRTIO_VIDEO_T_SET_PARAMS, > > + VIRTIO_VIDEO_T_GET_PARAMS, > > + > > + /* response */ > > + VIRTIO_VIDEO_S_OK = 0x0200, > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > + > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > +}; > > + > > +struct virtio_video_ctrl_hdr { > > + le32 type; > > + le32 stream_id; > > + le32 len; /* Length of the structure in bytes. */ > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{type}] is the type of the driver request or the device > > +response. > > +\item[\field{stream_id}] specifies a target stream. > > +\item[\field{len}] is the length of data in bytes, which includes > > +length of the header. > > +\end{description} > > + > > +\subsubsection{Device Operation: controlq} > > + > > +\begin{description} > > + > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > > +supported formats. > > + > > +The driver uses \field{struct virtio_video_get_capability} to send a > > +query request. > > + > > +\begin{lstlisting} > > +enum virtio_video_buf_type { > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > +}; > > I personally didn't like the previous term: pin_type. But, to be honest, I > don't like the buf_type neither. Consider the GET/SET_PARAMS request: > buf_type there looks a bit unnatural. We are trying to get stream > parameters there, not some parameters of whatever buffer. Also I don't see > any strict reason to mimic v4l2 naming scheme. > > I'd better rename it to PORT_TYPE or QUEUE_TYPE. > > > + > > +struct virtio_video_get_capability { > > + struct virtio_video_ctrl_hdr hdr; > > + enum virtio_video_buf_type buf_type; > > +}; > > +\end{lstlisting} > > +\begin{description} > > +\item[\field{buf_type}] is the buffer type that the driver asks > > +information about. The driver MUST set either > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > + > > +The device responds a capability by using \field{struct > > +virtio_video_get_capability_resp}. > > +\begin{lstlisting} > > +enum virtio_video_format { > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > + /* Raw formats */ > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > + VIRTIO_VIDEO_FORMAT_YUV420, > > + VIRTIO_VIDEO_FORMAT_YVU420, > > Let's add some variants of RGB, like RGBA, ARGB. We need it for the encoder > in particular . > > > + > > + /* Compressed formats */ > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > Let's add H265, MPEG4, MPEG2. We already support and use them. > > Regards, > Dmitry. > > > +}; > > + > > +enum virtio_video_profile { > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > + > > + /* H.264 */ > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > > VIRTIO_VIDEO_PROFILE_H264_MAIN, > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > > + /* VP8 */ > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > > + > > + /* VP9 */ > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > +}; > > + > > +struct virtio_video_format_range { > > + le32 min; > > + le32 max; > > + le32 step; > > + u8 paddings[4]; > > +}; > > + > > +struct virtio_video_format_desc { > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > > + le64 mask; > > + struct virtio_video_format_range width; > > + struct virtio_video_format_range height; > > + le32 num_rates; > > + u8 padding[4]; > > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > > +}; > > + > > +struct virtio_video_get_capability_resp { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 num_descs; > > + /* Followed by struct virtio_video_format_desc desc[] */ > > +}; > > +\end{lstlisting} > > + We also see the need to add a max_streams value to this structure so as to explicitly provide a limit on the number of streams the guest can create. Regards, Dmitry. > > +The format description \field{struct virtio_video_format_desc} > > +includes the following fields: > > +\begin{description} > > +\item[\field{format}] specifies an image format. The device MUST set one > > + of \field{enum virtio_video_format}. > > +\item[\field{profile}] specifies a profile of the compressed image format > > + specified in \field{format}. The driver SHOULD ignore this value if > > + \field{format} is a raw format. > > +\item[\field{mask}] is a bitset that represents the supported > > + combination of input and output format. If \textit{i}-th bit is set > > + in \field{mask} of \textit{j}-th \field{struct > > + virtio_video_format_desc} for input, the device supports encoding or > > + decoding from the \textit{j}-th input format to \textit{i}-th output > > + format. > > +\item[\field{width, height}] represents a range of resolutions > > + supported by the device. If its \field{step} is not applicable, its > > + \field{min} is equal to its \field{max}. > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. > > In > > case of decoder, the driver SHOULD ignore this value. > > +\item[\field{frame_rates}] is an array of supported frame rates. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > > + within the device. > > + > > +\begin{lstlisting} > > +enum virtio_video_mem_type { > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > +}; > > + > > +struct virtio_video_stream_create { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > + char debug_name[64]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{in_mem_type}] is a type of buffer management for input > > +buffers. The driver MUST set a value in \field{enum > > +virtio_video_mem_type}. > > +\item[\field{out_mem_type}] is a type of buffer management for output > > +buffers. The driver MUST set a value in \field{enum > > +virtio_video_mem_type}. > > +\item[\field{debug_name}] is a text string for a debug purpose. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_stream_destroy { > > + struct virtio_video_ctrl_hdr hdr; > > +}; > > +\end{lstlisting} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > > + queued buffers through the pipeline. > > + > > +\begin{lstlisting} > > +struct virtio_video_stream_drain { > > + struct virtio_video_ctrl_hdr hdr; > > +}; > > +\end{lstlisting} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_mem_entry { > > + le64 addr; > > + le32 length; > > + u8 padding[4]; > > +}; > > + > > +struct virtio_video_resource_create { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 resource_id; > > + le32 nr_entries; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{resource_id}] internal id of the resource. > > +\item[\field{nr_entries}] number of \field{struct > > + virtio_video_mem_entry} memory entries. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_resource_destroy { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 resource_id; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{resource_id}] internal id of the resource. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > > +queue. > > + > > +\begin{lstlisting} > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > + > > +struct virtio_video_resource_queue { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 buf_type; > > + le32 resource_id; > > + le64 timestamp; > > + le32 nr_data_size; > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{buf_type}] buf_type of the . > > +\item[\field{resource_id}] internal id of the resource. > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > + for synchronisation. > > +\item[\field{nr_data_size}] number of \field{data_size} entries. > > +\item[\field{data_size}] number of data bytes within a plane. > > +\end{description} > > + > > +\begin{lstlisting} > > +enum virtio_video_buffer_flag { > > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > > + /* Encoder only */ > > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > > +}; > > + > > +struct virtio_video_resource_queue_resp { > > + struct virtio_video_ctrl_hdr hdr; > > + le64 timestamp; > > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > > + le32 size; /* Encoded size */ > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > + for synchronisation. > > +\item[\field{flags}] mark specific buffers in the sequence. > > +\item[\field{size}] data size in the buffer (encoder only). > > +\end{description} > > + > > +The device sends a response to the queue request asynchronously when > > +it has finished processing the buffer. > > + > > +The device SHOULD mark a buffer that triggered a processing error with > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > > + > > +The device MUST mark the last buffer with the > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > +sequence. > > + > > +In case of encoder, to denote a particular frame type the devie MUST > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > > + buffers back from the input or the output queue of the device. The > > + device SHOULD return all of the buffers from the respective queue as > > + soon as possible without pushing the buffers through the processing > > + pipeline. > > + > > +\begin{lstlisting} > > +struct virtio_video_queue_clear { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 buf_type; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{buf_type}] buffer type. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > > + output of a stream. > > + > > +\begin{lstlisting} > > +struct virtio_video_plane_format { > > + le32 plane_size; > > + le32 stride; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{plane_size}] size of the plane in bytes. > > +\item[\field{stride}] stride used for the plane in bytes. > > +\end{description} > > + > > +\begin{lstlisting} > > +struct virtio_video_params { > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > > + le32 frame_width; > > + le32 frame_height; > > + le32 min_buffers; > > + le32 max_buffers; > > + le32 frame_rate; > > + struct virtio_video_crop { > > + le32 left; > > + le32 top; > > + le32 width; > > + le32 height; > > + } crop; > > + le32 num_planes; > > + struct virtio_video_plane_format > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{frame_width}] the value to get/set. > > +\item[\field{frame_height}] the value to get/set. > > +\item[\field{pixel_format}] the value to get/set. > > +\item[\field{min_buffers}] minimum buffers required to handle the > > + format (r/o). > > +\item[\field{max_buffers}] maximum buffers required to handle the > > + format (r/o). > > +\item[\field{frame_rate}] the value to get/set. > > +\item[\field{crop}] cropping (composing) rectangle. > > +\item[\field{num_planes}] number of planes used to store pixel data > > +(r/o). > > +\item[\field{plane_formats}] description of each plane. > > +\end{description} > > + > > +\begin{lstlisting} > > +struct virtio_video_get_params { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > +}; > > + > > +struct virtio_video_get_params_resp { > > + struct virtio_video_ctrl_hdr hdr; > > + struct virtio_video_params params; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{buf_type}] buffer type. > > +\item[\field{params}] parameter values. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > > + > > + > > +\begin{lstlisting} > > +struct virtio_video_set_params { > > + struct virtio_video_ctrl_hdr hdr; > > + struct virtio_video_params params; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{params}] parameters to set. > > +\end{description} > > + > > +Setting stream parameters might have side effects within the device. > > +For example, the device MAY perform alignment of width and height, > > +change the number of planes it uses for the format, or do whatever > > +changes that are required to continue normal operation using the > > +updated parameters. It is up to the driver to check the parameter set > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > > + > > +\end{description} > > + > > +\subsubsection{Device Operation: eventq} > > + > > +The device can report events on the event queue. The driver initially > > +populates the queue with device-writeable buffers. When the device > > +needs to report an event, it fills a buffer and notifies the driver. > > +The driver consumes the report and adds a new buffer to the virtqueue. > > + > > +\begin{lstlisting} > > +enum virtio_video_event_type { > > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > > + /* For all functions */ > > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > > + /* For decoder only */ > > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > > +}; > > + > > +struct virtio_video_event { > > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > > + le32 stream_id; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{event_type}] type of the triggered event . > > +\item[\field{stream_id}] id of the source stream. > > +\end{description} > > + > > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > > +whenever it encounters new resolution data in the stream. This > > +includes the case of the initial device configuration after metadata > > +has been parsed and the case of dynamic resolution change. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
Hi Keiichi, On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > The virtio video encoder device and decoder device provide functionalities > to encode and decode video stream respectively. > Though video encoder and decoder are provided as different devices, they use > a same protocol. > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > --- > content.tex | 1 + > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 580 insertions(+) > create mode 100644 virtio-video.tex > > diff --git a/content.tex b/content.tex > index 556b373..9e56839 100644 > --- a/content.tex > +++ b/content.tex > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > Requirements}\label{sec:Device \input{virtio-vsock.tex} > \input{virtio-fs.tex} > \input{virtio-rpmb.tex} > +\input{virtio-video.tex} > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > diff --git a/virtio-video.tex b/virtio-video.tex > new file mode 100644 > index 0000000..30e728d > --- /dev/null > +++ b/virtio-video.tex > @@ -0,0 +1,579 @@ > +\section{Video Device}\label{sec:Device Types / Video Device} > + > +The virtio video encoder device and decoder device are virtual devices that > +supports encoding and decoding respectively. Though the encoder and the > decoder +are different devices, they use the same protocol. > + > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID} > + > +\begin{description} > +\item[30] encoder device > +\item[31] decoder device > +\end{description} > + > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues} > + > +\begin{description} > +\item[0] controlq - queue for sending control commands. > +\item[1] eventq - queue for sending events happened in the device. > +\end{description} > + > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature > bits} + > +\begin{description} > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for > video + buffers. > +\end{description} > + > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device > / Feature bits} + > +The device MUST offer at least one of feature bits. > + > +\subsection{Device configuration layout}\label{sec:Device Types / Video > Device / Device configuration layout} + > +Video device configuration uses the following layout structure: > + > +\begin{lstlisting} > +struct virtio_video_config { > + le32 max_cap_len; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > + MUST set this value. > +\end{description} > + > +\subsection{Device Initialization}\label{sec:Device Types / Video Device / > Device Initialization} + > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / > Video Device / Device Initialization} + > +The driver SHOULD query device capability by using the > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > +setup. > + > +\subsection{Device Operation}\label{sec:Device Types / Video Device / > Device Operation} + > +The driver allocates input and output buffers and queues the buffers > +to the device. The device performs operations on the buffers according > +to the function in question. > + > +\subsubsection{Device Operation: Create stream} > + > +To process buffers, the device needs to associate them with a certain > +video stream (essentially, a context). Streams are created by > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > +determined by the device. > + > +\subsubsection{Device Operation: Create buffers} > + > +Buffers are used to store the actual data as well as the relevant > +metadata. Scatter lists are supported, so the buffer doesn't need to > +be contiguous in guest physical memory. > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > + resource that is backed by a buffer allocated from the driver's > + memory. > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > + is no longer needed. > +\end{itemize*} > + > +\subsubsection{Device Operation: Stream parameter control} > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters > for + input and output streams from the device. > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the > + device. > +\item After setting stream parameters, the driver may issue > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can > be + changed implicitly by the device during the set operation. > +\end{itemize*} > + > +\subsubsection{Device Operation: Process buffers} > + > +\begin{itemize*} > +\item If the function and the buffer type require so, write data to > +the buffer memory. > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > +processing in the device. > +\item The request completes asynchronously when the device has > +finished with the buffer. > +\end{itemize*} > + > +\subsubsection{Device Operation: Buffer processing control} > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > + return all of the already queued buffers. > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > + already queued buffers from the input or the output queue. This also > + includes input or output buffers that can be currently owned by the > + device's processing pipeline. > +\end{itemize*} > + > +\subsubsection{Device Operation: Asynchronous events} > + > +While processing buffers, the device can send asynchronous event > +notifications to the driver. The behaviour depends on the exact > +stream. For example, the decoder device sends a resolution change > +event when it encounters new resolution metadata in the stream. > + > +\subsubsection{Device Operation: Request header} > + > +All requests and responses on the control virt queue have a fixed > +header using the following layout structure and definitions: > + > +\begin{lstlisting} > +enum virtio_video_ctrl_type { > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > + > + /* request */ > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > + VIRTIO_VIDEO_T_STREAM_CREATE, > + VIRTIO_VIDEO_T_STREAM_DESTROY, > + VIRTIO_VIDEO_T_STREAM_DRAIN, > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > + VIRTIO_VIDEO_T_SET_PARAMS, > + VIRTIO_VIDEO_T_GET_PARAMS, > + > + /* response */ > + VIRTIO_VIDEO_S_OK = 0x0200, > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > + > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > +}; > + > +struct virtio_video_ctrl_hdr { > + le32 type; > + le32 stream_id; > + le32 len; /* Length of the structure in bytes. */ > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{type}] is the type of the driver request or the device > +response. > +\item[\field{stream_id}] specifies a target stream. > +\item[\field{len}] is the length of data in bytes, which includes > +length of the header. > +\end{description} > + > +\subsubsection{Device Operation: controlq} > + > +\begin{description} > + > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > +supported formats. > + > +The driver uses \field{struct virtio_video_get_capability} to send a > +query request. > + > +\begin{lstlisting} > +enum virtio_video_buf_type { > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > +}; > + > +struct virtio_video_get_capability { > + struct virtio_video_ctrl_hdr hdr; > + enum virtio_video_buf_type buf_type; > +}; > +\end{lstlisting} > +\begin{description} > +\item[\field{buf_type}] is the buffer type that the driver asks > +information about. The driver MUST set either > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > + > +The device responds a capability by using \field{struct > +virtio_video_get_capability_resp}. > +\begin{lstlisting} > +enum virtio_video_format { > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > + /* Raw formats */ > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > + VIRTIO_VIDEO_FORMAT_YUV420, > + VIRTIO_VIDEO_FORMAT_YVU420, > + > + /* Compressed formats */ > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > +}; > + > +enum virtio_video_profile { > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > + > + /* H.264 */ > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > VIRTIO_VIDEO_PROFILE_H264_MAIN, > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > + VIRTIO_VIDEO_PROFILE_H264_MAX = > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > + /* VP8 */ > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > + > + /* VP9 */ > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > + VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > +}; > + > +struct virtio_video_format_range { > + le32 min; > + le32 max; > + le32 step; > + u8 paddings[4]; > +}; > + > +struct virtio_video_format_desc { > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > + le64 mask; > + struct virtio_video_format_range width; > + struct virtio_video_format_range height; > + le32 num_rates; > + u8 padding[4]; > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > +}; > + > +struct virtio_video_get_capability_resp { > + struct virtio_video_ctrl_hdr hdr; > + le32 num_descs; > + /* Followed by struct virtio_video_format_desc desc[] */ > +}; > +\end{lstlisting} > + > +The format description \field{struct virtio_video_format_desc} > +includes the following fields: > +\begin{description} > +\item[\field{format}] specifies an image format. The device MUST set one > + of \field{enum virtio_video_format}. > +\item[\field{profile}] specifies a profile of the compressed image format > + specified in \field{format}. The driver SHOULD ignore this value if > + \field{format} is a raw format. > +\item[\field{mask}] is a bitset that represents the supported > + combination of input and output format. If \textit{i}-th bit is set > + in \field{mask} of \textit{j}-th \field{struct > + virtio_video_format_desc} for input, the device supports encoding or > + decoding from the \textit{j}-th input format to \textit{i}-th output > + format. > +\item[\field{width, height}] represents a range of resolutions > + supported by the device. If its \field{step} is not applicable, its > + \field{min} is equal to its \field{max}. > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In > case of decoder, the driver SHOULD ignore this value. > +\item[\field{frame_rates}] is an array of supported frame rates. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > + within the device. > + > +\begin{lstlisting} > +enum virtio_video_mem_type { > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > +}; > + > +struct virtio_video_stream_create { > + struct virtio_video_ctrl_hdr hdr; > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > + char debug_name[64]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{in_mem_type}] is a type of buffer management for input > +buffers. The driver MUST set a value in \field{enum > +virtio_video_mem_type}. > +\item[\field{out_mem_type}] is a type of buffer management for output > +buffers. The driver MUST set a value in \field{enum > +virtio_video_mem_type}. > +\item[\field{debug_name}] is a text string for a debug purpose. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_stream_destroy { > + struct virtio_video_ctrl_hdr hdr; > +}; > +\end{lstlisting} > + > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > + queued buffers through the pipeline. > + > +\begin{lstlisting} > +struct virtio_video_stream_drain { > + struct virtio_video_ctrl_hdr hdr; > +}; > +\end{lstlisting} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_mem_entry { > + le64 addr; > + le32 length; > + u8 padding[4]; > +}; > + > +struct virtio_video_resource_create { > + struct virtio_video_ctrl_hdr hdr; > + le32 resource_id; > + le32 nr_entries; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{resource_id}] internal id of the resource. > +\item[\field{nr_entries}] number of \field{struct > + virtio_video_mem_entry} memory entries. > +\end{description} > + How should one deal with multiplanar formats? Do we create one resource per plane? Otherwise we need a way to send mem entries for each plane in one request. Currently the v1 driver attaches backing to the same resource multiple times when a buffer has multiple planes. Not the nicest way to be honest. Regards, Dmitry. > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_resource_destroy { > + struct virtio_video_ctrl_hdr hdr; > + le32 resource_id; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{resource_id}] internal id of the resource. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > +queue. > + > +\begin{lstlisting} > +#define VIRTIO_VIDEO_MAX_PLANES 8 > + > +struct virtio_video_resource_queue { > + struct virtio_video_ctrl_hdr hdr; > + le32 buf_type; > + le32 resource_id; > + le64 timestamp; > + le32 nr_data_size; > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{buf_type}] buf_type of the . > +\item[\field{resource_id}] internal id of the resource. > +\item[\field{timestamp}] an abstract sequence counter that can be used > + for synchronisation. > +\item[\field{nr_data_size}] number of \field{data_size} entries. > +\item[\field{data_size}] number of data bytes within a plane. > +\end{description} > + > +\begin{lstlisting} > +enum virtio_video_buffer_flag { > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > + /* Encoder only */ > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > +}; > + > +struct virtio_video_resource_queue_resp { > + struct virtio_video_ctrl_hdr hdr; > + le64 timestamp; > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > + le32 size; /* Encoded size */ > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{timestamp}] an abstract sequence counter that can be used > + for synchronisation. > +\item[\field{flags}] mark specific buffers in the sequence. > +\item[\field{size}] data size in the buffer (encoder only). > +\end{description} > + > +The device sends a response to the queue request asynchronously when > +it has finished processing the buffer. > + > +The device SHOULD mark a buffer that triggered a processing error with > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > + > +The device MUST mark the last buffer with the > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > +sequence. > + > +In case of encoder, to denote a particular frame type the devie MUST > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > + > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > + buffers back from the input or the output queue of the device. The > + device SHOULD return all of the buffers from the respective queue as > + soon as possible without pushing the buffers through the processing > + pipeline. > + > +\begin{lstlisting} > +struct virtio_video_queue_clear { > + struct virtio_video_ctrl_hdr hdr; > + le32 buf_type; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{buf_type}] buffer type. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > + output of a stream. > + > +\begin{lstlisting} > +struct virtio_video_plane_format { > + le32 plane_size; > + le32 stride; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{plane_size}] size of the plane in bytes. > +\item[\field{stride}] stride used for the plane in bytes. > +\end{description} > + > +\begin{lstlisting} > +struct virtio_video_params { > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > + le32 frame_width; > + le32 frame_height; > + le32 min_buffers; > + le32 max_buffers; > + le32 frame_rate; > + struct virtio_video_crop { > + le32 left; > + le32 top; > + le32 width; > + le32 height; > + } crop; > + le32 num_planes; > + struct virtio_video_plane_format > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{frame_width}] the value to get/set. > +\item[\field{frame_height}] the value to get/set. > +\item[\field{pixel_format}] the value to get/set. > +\item[\field{min_buffers}] minimum buffers required to handle the > + format (r/o). > +\item[\field{max_buffers}] maximum buffers required to handle the > + format (r/o). > +\item[\field{frame_rate}] the value to get/set. > +\item[\field{crop}] cropping (composing) rectangle. > +\item[\field{num_planes}] number of planes used to store pixel data > +(r/o). > +\item[\field{plane_formats}] description of each plane. > +\end{description} > + > +\begin{lstlisting} > +struct virtio_video_get_params { > + struct virtio_video_ctrl_hdr hdr; > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > +}; > + > +struct virtio_video_get_params_resp { > + struct virtio_video_ctrl_hdr hdr; > + struct virtio_video_params params; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{buf_type}] buffer type. > +\item[\field{params}] parameter values. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > + > + > +\begin{lstlisting} > +struct virtio_video_set_params { > + struct virtio_video_ctrl_hdr hdr; > + struct virtio_video_params params; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{params}] parameters to set. > +\end{description} > + > +Setting stream parameters might have side effects within the device. > +For example, the device MAY perform alignment of width and height, > +change the number of planes it uses for the format, or do whatever > +changes that are required to continue normal operation using the > +updated parameters. It is up to the driver to check the parameter set > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > + > +\end{description} > + > +\subsubsection{Device Operation: eventq} > + > +The device can report events on the event queue. The driver initially > +populates the queue with device-writeable buffers. When the device > +needs to report an event, it fills a buffer and notifies the driver. > +The driver consumes the report and adds a new buffer to the virtqueue. > + > +\begin{lstlisting} > +enum virtio_video_event_type { > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > + /* For all functions */ > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > + /* For decoder only */ > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > +}; > + > +struct virtio_video_event { > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > + le32 stream_id; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{event_type}] type of the triggered event . > +\item[\field{stream_id}] id of the source stream. > +\end{description} > + > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > +whenever it encounters new resolution data in the stream. This > +includes the case of the initial device configuration after metadata > +has been parsed and the case of dynamic resolution change.
On Mon, Dec 30, 2019 at 9:16 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > > On Donnerstag, 19. Dezember 2019 14:28:23 CET Dmitry Sepp wrote: > > Hi Keiichi, > > > > Thank you for the update. Please see some comments below. > > > > Also, we need to bring the virtio_video_control back as it is in fact used > > by the driver to enumerate supported encoder controls. But yes, it still > > needs to be documemnted, it's true. > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > The virtio video encoder device and decoder device provide functionalities > > > to encode and decode video stream respectively. > > > Though video encoder and decoder are provided as different devices, they > > > use a same protocol. > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > > --- > > > > > > content.tex | 1 + > > > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > > > 2 files changed, 580 insertions(+) > > > create mode 100644 virtio-video.tex > > > > > > diff --git a/content.tex b/content.tex > > > index 556b373..9e56839 100644 > > > --- a/content.tex > > > +++ b/content.tex > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > > Requirements}\label{sec:Device \input{virtio-vsock.tex} > > > > > > \input{virtio-fs.tex} > > > \input{virtio-rpmb.tex} > > > > > > +\input{virtio-video.tex} > > > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > > new file mode 100644 > > > index 0000000..30e728d > > > --- /dev/null > > > +++ b/virtio-video.tex > > > @@ -0,0 +1,579 @@ > > > +\section{Video Device}\label{sec:Device Types / Video Device} > > > + > > > +The virtio video encoder device and decoder device are virtual devices > > > that +supports encoding and decoding respectively. Though the encoder and > > > the decoder +are different devices, they use the same protocol. > > > + > > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID} > > > + > > > +\begin{description} > > > +\item[30] encoder device > > > +\item[31] decoder device > > > +\end{description} > > > + > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / > > > Virtqueues} + > > > +\begin{description} > > > +\item[0] controlq - queue for sending control commands. > > > +\item[1] eventq - queue for sending events happened in the device. > > > +\end{description} > > > + > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature > > > bits} + > > > +\begin{description} > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used > > > for > > > video + buffers. > > > +\end{description} > > > + > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video > > > Device > > > / Feature bits} + > > > +The device MUST offer at least one of feature bits. > > > + > > > +\subsection{Device configuration layout}\label{sec:Device Types / Video > > > Device / Device configuration layout} + > > > +Video device configuration uses the following layout structure: > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_config { > > > + le32 max_cap_len; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > > > + MUST set this value. > > > +\end{description} > > > + > > > +\subsection{Device Initialization}\label{sec:Device Types / Video Device > > > / > > > Device Initialization} + > > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / > > > Video Device / Device Initialization} + > > > +The driver SHOULD query device capability by using the > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > > > +setup. > > > + > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device / > > > Device Operation} + > > > +The driver allocates input and output buffers and queues the buffers > > > +to the device. The device performs operations on the buffers according > > > +to the function in question. > > > + > > > +\subsubsection{Device Operation: Create stream} > > > + > > > +To process buffers, the device needs to associate them with a certain > > > +video stream (essentially, a context). Streams are created by > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > > +determined by the device. > > > + > > > +\subsubsection{Device Operation: Create buffers} > > > + > > > +Buffers are used to store the actual data as well as the relevant > > > +metadata. Scatter lists are supported, so the buffer doesn't need to > > > +be contiguous in guest physical memory. > > > + > > > +\begin{itemize*} > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > > > + resource that is backed by a buffer allocated from the driver's > > > + memory. > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > > > + is no longer needed. > > > +\end{itemize*} > > > + > > > +\subsubsection{Device Operation: Stream parameter control} > > > + > > > +\begin{itemize*} > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters > > > for + input and output streams from the device. > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to > > > the + device. > > > +\item After setting stream parameters, the driver may issue > > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output > > > can be + changed implicitly by the device during the set operation. > > > +\end{itemize*} > > > + > > > +\subsubsection{Device Operation: Process buffers} > > > + > > > +\begin{itemize*} > > > +\item If the function and the buffer type require so, write data to > > > +the buffer memory. > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > > > +processing in the device. > > > +\item The request completes asynchronously when the device has > > > +finished with the buffer. > > > +\end{itemize*} > > > + > > > +\subsubsection{Device Operation: Buffer processing control} > > > + > > > +\begin{itemize*} > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > > > + return all of the already queued buffers. > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > > > + already queued buffers from the input or the output queue. This also > > > + includes input or output buffers that can be currently owned by the > > > + device's processing pipeline. > > > +\end{itemize*} > > > + > > > +\subsubsection{Device Operation: Asynchronous events} > > > + > > > +While processing buffers, the device can send asynchronous event > > > +notifications to the driver. The behaviour depends on the exact > > > +stream. For example, the decoder device sends a resolution change > > > +event when it encounters new resolution metadata in the stream. > > > + > > > +\subsubsection{Device Operation: Request header} > > > + > > > +All requests and responses on the control virt queue have a fixed > > > +header using the following layout structure and definitions: > > > + > > > +\begin{lstlisting} > > > +enum virtio_video_ctrl_type { > > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > > + > > > + /* request */ > > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > > + VIRTIO_VIDEO_T_SET_PARAMS, > > > + VIRTIO_VIDEO_T_GET_PARAMS, > > > + > > > + /* response */ > > > + VIRTIO_VIDEO_S_OK = 0x0200, > > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > > + > > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > > +}; > > > + > > > +struct virtio_video_ctrl_hdr { > > > + le32 type; > > > + le32 stream_id; > > > + le32 len; /* Length of the structure in bytes. */ > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{type}] is the type of the driver request or the device > > > +response. > > > +\item[\field{stream_id}] specifies a target stream. > > > +\item[\field{len}] is the length of data in bytes, which includes > > > +length of the header. > > > +\end{description} > > > + > > > +\subsubsection{Device Operation: controlq} > > > + > > > +\begin{description} > > > + > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > > > +supported formats. > > > + > > > +The driver uses \field{struct virtio_video_get_capability} to send a > > > +query request. > > > + > > > +\begin{lstlisting} > > > +enum virtio_video_buf_type { > > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > > +}; > > > > I personally didn't like the previous term: pin_type. But, to be honest, I > > don't like the buf_type neither. Consider the GET/SET_PARAMS request: > > buf_type there looks a bit unnatural. We are trying to get stream > > parameters there, not some parameters of whatever buffer. Also I don't see > > any strict reason to mimic v4l2 naming scheme. > > > > I'd better rename it to PORT_TYPE or QUEUE_TYPE. > > > > > + > > > +struct virtio_video_get_capability { > > > + struct virtio_video_ctrl_hdr hdr; > > > + enum virtio_video_buf_type buf_type; > > > +}; > > > +\end{lstlisting} > > > +\begin{description} > > > +\item[\field{buf_type}] is the buffer type that the driver asks > > > +information about. The driver MUST set either > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > > + > > > +The device responds a capability by using \field{struct > > > +virtio_video_get_capability_resp}. > > > +\begin{lstlisting} > > > +enum virtio_video_format { > > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > > + /* Raw formats */ > > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > > + VIRTIO_VIDEO_FORMAT_YUV420, > > > + VIRTIO_VIDEO_FORMAT_YVU420, > > > > Let's add some variants of RGB, like RGBA, ARGB. We need it for the encoder > > in particular . > > > > > + > > > + /* Compressed formats */ > > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > > > Let's add H265, MPEG4, MPEG2. We already support and use them. > > > > Regards, > > Dmitry. > > > > > +}; > > > + > > > +enum virtio_video_profile { > > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > > + > > > + /* H.264 */ > > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > > > VIRTIO_VIDEO_PROFILE_H264_MAIN, > > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > > > + /* VP8 */ > > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > > > + > > > + /* VP9 */ > > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN, > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > +}; > > > + > > > +struct virtio_video_format_range { > > > + le32 min; > > > + le32 max; > > > + le32 step; > > > + u8 paddings[4]; > > > +}; > > > + > > > +struct virtio_video_format_desc { > > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > > > + le64 mask; > > > + struct virtio_video_format_range width; > > > + struct virtio_video_format_range height; > > > + le32 num_rates; > > > + u8 padding[4]; > > > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > > > +}; > > > + > > > +struct virtio_video_get_capability_resp { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 num_descs; > > > + /* Followed by struct virtio_video_format_desc desc[] */ > > > +}; > > > +\end{lstlisting} > > > + > We also see the need to add a max_streams value to this structure so as to > explicitly provide a limit on the number of streams the guest can create. What would be the advantage over just trying to create one and failing? The maximum number would be only meaningful for the special case when the streams are always only created by one user space process. Otherwise, if several processes do that, they are not aware of each other and the number could be higher than they can actually create, because other processes could have some streams created already. Best regards, Tomasz > > Regards, > Dmitry. > > > > +The format description \field{struct virtio_video_format_desc} > > > +includes the following fields: > > > +\begin{description} > > > +\item[\field{format}] specifies an image format. The device MUST set one > > > + of \field{enum virtio_video_format}. > > > +\item[\field{profile}] specifies a profile of the compressed image format > > > + specified in \field{format}. The driver SHOULD ignore this value if > > > + \field{format} is a raw format. > > > +\item[\field{mask}] is a bitset that represents the supported > > > + combination of input and output format. If \textit{i}-th bit is set > > > + in \field{mask} of \textit{j}-th \field{struct > > > + virtio_video_format_desc} for input, the device supports encoding or > > > + decoding from the \textit{j}-th input format to \textit{i}-th output > > > + format. > > > +\item[\field{width, height}] represents a range of resolutions > > > + supported by the device. If its \field{step} is not applicable, its > > > + \field{min} is equal to its \field{max}. > > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. > > > In > > > case of decoder, the driver SHOULD ignore this value. > > > +\item[\field{frame_rates}] is an array of supported frame rates. > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > > > + within the device. > > > + > > > +\begin{lstlisting} > > > +enum virtio_video_mem_type { > > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > > +}; > > > + > > > +struct virtio_video_stream_create { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > > + char debug_name[64]; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{in_mem_type}] is a type of buffer management for input > > > +buffers. The driver MUST set a value in \field{enum > > > +virtio_video_mem_type}. > > > +\item[\field{out_mem_type}] is a type of buffer management for output > > > +buffers. The driver MUST set a value in \field{enum > > > +virtio_video_mem_type}. > > > +\item[\field{debug_name}] is a text string for a debug purpose. > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > > > + within the device. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_stream_destroy { > > > + struct virtio_video_ctrl_hdr hdr; > > > +}; > > > +\end{lstlisting} > > > + > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > > > + queued buffers through the pipeline. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_stream_drain { > > > + struct virtio_video_ctrl_hdr hdr; > > > +}; > > > +\end{lstlisting} > > > + > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > > > + within the device. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_mem_entry { > > > + le64 addr; > > > + le32 length; > > > + u8 padding[4]; > > > +}; > > > + > > > +struct virtio_video_resource_create { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 resource_id; > > > + le32 nr_entries; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{resource_id}] internal id of the resource. > > > +\item[\field{nr_entries}] number of \field{struct > > > + virtio_video_mem_entry} memory entries. > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > > > + within the device. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_resource_destroy { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 resource_id; > > > + u8 padding[4]; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{resource_id}] internal id of the resource. > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > > > +queue. > > > + > > > +\begin{lstlisting} > > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > > + > > > +struct virtio_video_resource_queue { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 buf_type; > > > + le32 resource_id; > > > + le64 timestamp; > > > + le32 nr_data_size; > > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{buf_type}] buf_type of the . > > > +\item[\field{resource_id}] internal id of the resource. > > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > > + for synchronisation. > > > +\item[\field{nr_data_size}] number of \field{data_size} entries. > > > +\item[\field{data_size}] number of data bytes within a plane. > > > +\end{description} > > > + > > > +\begin{lstlisting} > > > +enum virtio_video_buffer_flag { > > > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > > > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > > > + /* Encoder only */ > > > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > > > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > > > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > > > +}; > > > + > > > +struct virtio_video_resource_queue_resp { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le64 timestamp; > > > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > > > + le32 size; /* Encoded size */ > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > > + for synchronisation. > > > +\item[\field{flags}] mark specific buffers in the sequence. > > > +\item[\field{size}] data size in the buffer (encoder only). > > > +\end{description} > > > + > > > +The device sends a response to the queue request asynchronously when > > > +it has finished processing the buffer. > > > + > > > +The device SHOULD mark a buffer that triggered a processing error with > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > > > + > > > +The device MUST mark the last buffer with the > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > > +sequence. > > > + > > > +In case of encoder, to denote a particular frame type the devie MUST > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > > > + > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > > > + buffers back from the input or the output queue of the device. The > > > + device SHOULD return all of the buffers from the respective queue as > > > + soon as possible without pushing the buffers through the processing > > > + pipeline. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_queue_clear { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 buf_type; > > > + u8 padding[4]; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{buf_type}] buffer type. > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > > > + output of a stream. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_plane_format { > > > + le32 plane_size; > > > + le32 stride; > > > + u8 padding[4]; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{plane_size}] size of the plane in bytes. > > > +\item[\field{stride}] stride used for the plane in bytes. > > > +\end{description} > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_params { > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > > > + le32 frame_width; > > > + le32 frame_height; > > > + le32 min_buffers; > > > + le32 max_buffers; > > > + le32 frame_rate; > > > + struct virtio_video_crop { > > > + le32 left; > > > + le32 top; > > > + le32 width; > > > + le32 height; > > > + } crop; > > > + le32 num_planes; > > > + struct virtio_video_plane_format > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{frame_width}] the value to get/set. > > > +\item[\field{frame_height}] the value to get/set. > > > +\item[\field{pixel_format}] the value to get/set. > > > +\item[\field{min_buffers}] minimum buffers required to handle the > > > + format (r/o). > > > +\item[\field{max_buffers}] maximum buffers required to handle the > > > + format (r/o). > > > +\item[\field{frame_rate}] the value to get/set. > > > +\item[\field{crop}] cropping (composing) rectangle. > > > +\item[\field{num_planes}] number of planes used to store pixel data > > > +(r/o). > > > +\item[\field{plane_formats}] description of each plane. > > > +\end{description} > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_get_params { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > +}; > > > + > > > +struct virtio_video_get_params_resp { > > > + struct virtio_video_ctrl_hdr hdr; > > > + struct virtio_video_params params; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{buf_type}] buffer type. > > > +\item[\field{params}] parameter values. > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > > > + > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_set_params { > > > + struct virtio_video_ctrl_hdr hdr; > > > + struct virtio_video_params params; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{params}] parameters to set. > > > +\end{description} > > > + > > > +Setting stream parameters might have side effects within the device. > > > +For example, the device MAY perform alignment of width and height, > > > +change the number of planes it uses for the format, or do whatever > > > +changes that are required to continue normal operation using the > > > +updated parameters. It is up to the driver to check the parameter set > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > > > + > > > +\end{description} > > > + > > > +\subsubsection{Device Operation: eventq} > > > + > > > +The device can report events on the event queue. The driver initially > > > +populates the queue with device-writeable buffers. When the device > > > +needs to report an event, it fills a buffer and notifies the driver. > > > +The driver consumes the report and adds a new buffer to the virtqueue. > > > + > > > +\begin{lstlisting} > > > +enum virtio_video_event_type { > > > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > > > + /* For all functions */ > > > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > > > + /* For decoder only */ > > > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > > > +}; > > > + > > > +struct virtio_video_event { > > > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > > > + le32 stream_id; > > > + u8 padding[4]; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{event_type}] type of the triggered event . > > > +\item[\field{stream_id}] id of the source stream. > > > +\end{description} > > > + > > > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > > > +whenever it encounters new resolution data in the stream. This > > > +includes the case of the initial device configuration after metadata > > > +has been parsed and the case of dynamic resolution change. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org > > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org > >
Hi, > > We also see the need to add a max_streams value to this structure so as to > > explicitly provide a limit on the number of streams the guest can create. > > What would be the advantage over just trying to create one and > failing? The maximum number would be only meaningful for the special > case when the streams are always only created by one user space > process. Otherwise, if several processes do that, they are not aware > of each other and the number could be higher than they can actually > create, because other processes could have some streams created > already. Also the number of streams might not be fixed but depend on stream parameters, i.e. hardware can decode one hd or two sd streams ... cheers, Gerd
Hi, > How should one deal with multiplanar formats? Do we create one resource per > plane? Otherwise we need a way to send mem entries for each plane in one > request. DRM uses arrays of handles and offsets (see struct drm_framebuffer). A handle references a gem object (roughly the same as a resource), and the offset specifies the start of the plane within the gem object. That allows both a single gem object with planes stored at different offsets and one gem object per plane. virtio-video could do the same, or pick one of the two approaches and support only that. cheers, Gerd
Hi, On Montag, 6. Januar 2020 09:33:35 CET Gerd Hoffmann wrote: > Hi, > > > > We also see the need to add a max_streams value to this structure so as > > > to > > > explicitly provide a limit on the number of streams the guest can > > > create. > > > > What would be the advantage over just trying to create one and > > failing? The maximum number would be only meaningful for the special > > case when the streams are always only created by one user space > > process. Otherwise, if several processes do that, they are not aware > > of each other and the number could be higher than they can actually > > create, because other processes could have some streams created > > already. > > Also the number of streams might not be fixed but depend on stream > parameters, i.e. hardware can decode one hd or two sd streams ... > Ok, you are right. We'd better return an error from the device side. Regards, Dmitry. > cheers, > Gerd
Hi Dmitry, On Sat, Jan 4, 2020 at 12:47 AM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi Keiichi, > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > The virtio video encoder device and decoder device provide functionalities > > to encode and decode video stream respectively. > > Though video encoder and decoder are provided as different devices, they use > > a same protocol. > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > --- > > content.tex | 1 + > > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > > 2 files changed, 580 insertions(+) > > create mode 100644 virtio-video.tex > > > > diff --git a/content.tex b/content.tex > > index 556b373..9e56839 100644 > > --- a/content.tex > > +++ b/content.tex > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > Requirements}\label{sec:Device \input{virtio-vsock.tex} > > \input{virtio-fs.tex} > > \input{virtio-rpmb.tex} > > +\input{virtio-video.tex} > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > new file mode 100644 > > index 0000000..30e728d > > --- /dev/null > > +++ b/virtio-video.tex > > @@ -0,0 +1,579 @@ > > +\section{Video Device}\label{sec:Device Types / Video Device} > > + > > +The virtio video encoder device and decoder device are virtual devices that > > +supports encoding and decoding respectively. Though the encoder and the > > decoder +are different devices, they use the same protocol. > > + > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID} > > + > > +\begin{description} > > +\item[30] encoder device > > +\item[31] decoder device > > +\end{description} > > + > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues} > > + > > +\begin{description} > > +\item[0] controlq - queue for sending control commands. > > +\item[1] eventq - queue for sending events happened in the device. > > +\end{description} > > + > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature > > bits} + > > +\begin{description} > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for > > video + buffers. > > +\end{description} > > + > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device > > / Feature bits} + > > +The device MUST offer at least one of feature bits. > > + > > +\subsection{Device configuration layout}\label{sec:Device Types / Video > > Device / Device configuration layout} + > > +Video device configuration uses the following layout structure: > > + > > +\begin{lstlisting} > > +struct virtio_video_config { > > + le32 max_cap_len; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > > + MUST set this value. > > +\end{description} > > + > > +\subsection{Device Initialization}\label{sec:Device Types / Video Device / > > Device Initialization} + > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / > > Video Device / Device Initialization} + > > +The driver SHOULD query device capability by using the > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > > +setup. > > + > > +\subsection{Device Operation}\label{sec:Device Types / Video Device / > > Device Operation} + > > +The driver allocates input and output buffers and queues the buffers > > +to the device. The device performs operations on the buffers according > > +to the function in question. > > + > > +\subsubsection{Device Operation: Create stream} > > + > > +To process buffers, the device needs to associate them with a certain > > +video stream (essentially, a context). Streams are created by > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > +determined by the device. > > + > > +\subsubsection{Device Operation: Create buffers} > > + > > +Buffers are used to store the actual data as well as the relevant > > +metadata. Scatter lists are supported, so the buffer doesn't need to > > +be contiguous in guest physical memory. > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > > + resource that is backed by a buffer allocated from the driver's > > + memory. > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > > + is no longer needed. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Stream parameter control} > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters > > for + input and output streams from the device. > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the > > + device. > > +\item After setting stream parameters, the driver may issue > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can > > be + changed implicitly by the device during the set operation. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Process buffers} > > + > > +\begin{itemize*} > > +\item If the function and the buffer type require so, write data to > > +the buffer memory. > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > > +processing in the device. > > +\item The request completes asynchronously when the device has > > +finished with the buffer. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Buffer processing control} > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > > + return all of the already queued buffers. > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > > + already queued buffers from the input or the output queue. This also > > + includes input or output buffers that can be currently owned by the > > + device's processing pipeline. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Asynchronous events} > > + > > +While processing buffers, the device can send asynchronous event > > +notifications to the driver. The behaviour depends on the exact > > +stream. For example, the decoder device sends a resolution change > > +event when it encounters new resolution metadata in the stream. > > + > > +\subsubsection{Device Operation: Request header} > > + > > +All requests and responses on the control virt queue have a fixed > > +header using the following layout structure and definitions: > > + > > +\begin{lstlisting} > > +enum virtio_video_ctrl_type { > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > + > > + /* request */ > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > + VIRTIO_VIDEO_T_SET_PARAMS, > > + VIRTIO_VIDEO_T_GET_PARAMS, > > + > > + /* response */ > > + VIRTIO_VIDEO_S_OK = 0x0200, > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > + > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > +}; > > + > > +struct virtio_video_ctrl_hdr { > > + le32 type; > > + le32 stream_id; > > + le32 len; /* Length of the structure in bytes. */ > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{type}] is the type of the driver request or the device > > +response. > > +\item[\field{stream_id}] specifies a target stream. > > +\item[\field{len}] is the length of data in bytes, which includes > > +length of the header. > > +\end{description} > > + > > +\subsubsection{Device Operation: controlq} > > + > > +\begin{description} > > + > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > > +supported formats. > > + > > +The driver uses \field{struct virtio_video_get_capability} to send a > > +query request. > > + > > +\begin{lstlisting} > > +enum virtio_video_buf_type { > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > +}; > > + > > +struct virtio_video_get_capability { > > + struct virtio_video_ctrl_hdr hdr; > > + enum virtio_video_buf_type buf_type; > > +}; > > +\end{lstlisting} > > +\begin{description} > > +\item[\field{buf_type}] is the buffer type that the driver asks > > +information about. The driver MUST set either > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > + > > +The device responds a capability by using \field{struct > > +virtio_video_get_capability_resp}. > > +\begin{lstlisting} > > +enum virtio_video_format { > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > + /* Raw formats */ > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > + VIRTIO_VIDEO_FORMAT_YUV420, > > + VIRTIO_VIDEO_FORMAT_YVU420, > > + > > + /* Compressed formats */ > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > +}; > > + > > +enum virtio_video_profile { > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > + > > + /* H.264 */ > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > > VIRTIO_VIDEO_PROFILE_H264_MAIN, > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > > + /* VP8 */ > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > > + > > + /* VP9 */ > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > +}; > > + > > +struct virtio_video_format_range { > > + le32 min; > > + le32 max; > > + le32 step; > > + u8 paddings[4]; > > +}; > > + > > +struct virtio_video_format_desc { > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > > + le64 mask; > > + struct virtio_video_format_range width; > > + struct virtio_video_format_range height; > > + le32 num_rates; > > + u8 padding[4]; > > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > > +}; > > + > > +struct virtio_video_get_capability_resp { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 num_descs; > > + /* Followed by struct virtio_video_format_desc desc[] */ > > +}; > > +\end{lstlisting} > > + > > +The format description \field{struct virtio_video_format_desc} > > +includes the following fields: > > +\begin{description} > > +\item[\field{format}] specifies an image format. The device MUST set one > > + of \field{enum virtio_video_format}. > > +\item[\field{profile}] specifies a profile of the compressed image format > > + specified in \field{format}. The driver SHOULD ignore this value if > > + \field{format} is a raw format. > > +\item[\field{mask}] is a bitset that represents the supported > > + combination of input and output format. If \textit{i}-th bit is set > > + in \field{mask} of \textit{j}-th \field{struct > > + virtio_video_format_desc} for input, the device supports encoding or > > + decoding from the \textit{j}-th input format to \textit{i}-th output > > + format. > > +\item[\field{width, height}] represents a range of resolutions > > + supported by the device. If its \field{step} is not applicable, its > > + \field{min} is equal to its \field{max}. > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In > > case of decoder, the driver SHOULD ignore this value. > > +\item[\field{frame_rates}] is an array of supported frame rates. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > > + within the device. > > + > > +\begin{lstlisting} > > +enum virtio_video_mem_type { > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > +}; > > + > > +struct virtio_video_stream_create { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > + char debug_name[64]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{in_mem_type}] is a type of buffer management for input > > +buffers. The driver MUST set a value in \field{enum > > +virtio_video_mem_type}. > > +\item[\field{out_mem_type}] is a type of buffer management for output > > +buffers. The driver MUST set a value in \field{enum > > +virtio_video_mem_type}. > > +\item[\field{debug_name}] is a text string for a debug purpose. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_stream_destroy { > > + struct virtio_video_ctrl_hdr hdr; > > +}; > > +\end{lstlisting} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > > + queued buffers through the pipeline. > > + > > +\begin{lstlisting} > > +struct virtio_video_stream_drain { > > + struct virtio_video_ctrl_hdr hdr; > > +}; > > +\end{lstlisting} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_mem_entry { > > + le64 addr; > > + le32 length; > > + u8 padding[4]; > > +}; > > + > > +struct virtio_video_resource_create { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 resource_id; > > + le32 nr_entries; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{resource_id}] internal id of the resource. > > +\item[\field{nr_entries}] number of \field{struct > > + virtio_video_mem_entry} memory entries. > > +\end{description} > > + > > How should one deal with multiplanar formats? Do we create one resource per > plane? Otherwise we need a way to send mem entries for each plane in one > request. That's a good point. We should be able to create a resource that corresponds multiple planes. How about changing the struct like this: struct virtio_video_resource_create { struct virtio_video_ctrl_hdr hdr; le32 resource_id; le32 nums_entries[VIRTIO_VIDEO_MAX_PLANES]; u8 padding[4]; /* Followed by struct virtio_video_mem_entry entries[]. |length of entries[]| == |sum of nums_entries[i] for each i| */ }; Best regards, Keiichi > > Currently the v1 driver attaches backing to the same resource multiple times > when a buffer has multiple planes. Not the nicest way to be honest. > > Regards, > Dmitry. > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_resource_destroy { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 resource_id; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{resource_id}] internal id of the resource. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > > +queue. > > + > > +\begin{lstlisting} > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > + > > +struct virtio_video_resource_queue { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 buf_type; > > + le32 resource_id; > > + le64 timestamp; > > + le32 nr_data_size; > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{buf_type}] buf_type of the . > > +\item[\field{resource_id}] internal id of the resource. > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > + for synchronisation. > > +\item[\field{nr_data_size}] number of \field{data_size} entries. > > +\item[\field{data_size}] number of data bytes within a plane. > > +\end{description} > > + > > +\begin{lstlisting} > > +enum virtio_video_buffer_flag { > > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > > + /* Encoder only */ > > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > > +}; > > + > > +struct virtio_video_resource_queue_resp { > > + struct virtio_video_ctrl_hdr hdr; > > + le64 timestamp; > > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > > + le32 size; /* Encoded size */ > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > + for synchronisation. > > +\item[\field{flags}] mark specific buffers in the sequence. > > +\item[\field{size}] data size in the buffer (encoder only). > > +\end{description} > > + > > +The device sends a response to the queue request asynchronously when > > +it has finished processing the buffer. > > + > > +The device SHOULD mark a buffer that triggered a processing error with > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > > + > > +The device MUST mark the last buffer with the > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > +sequence. > > + > > +In case of encoder, to denote a particular frame type the devie MUST > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > > + buffers back from the input or the output queue of the device. The > > + device SHOULD return all of the buffers from the respective queue as > > + soon as possible without pushing the buffers through the processing > > + pipeline. > > + > > +\begin{lstlisting} > > +struct virtio_video_queue_clear { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 buf_type; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{buf_type}] buffer type. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > > + output of a stream. > > + > > +\begin{lstlisting} > > +struct virtio_video_plane_format { > > + le32 plane_size; > > + le32 stride; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{plane_size}] size of the plane in bytes. > > +\item[\field{stride}] stride used for the plane in bytes. > > +\end{description} > > + > > +\begin{lstlisting} > > +struct virtio_video_params { > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > > + le32 frame_width; > > + le32 frame_height; > > + le32 min_buffers; > > + le32 max_buffers; > > + le32 frame_rate; > > + struct virtio_video_crop { > > + le32 left; > > + le32 top; > > + le32 width; > > + le32 height; > > + } crop; > > + le32 num_planes; > > + struct virtio_video_plane_format > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{frame_width}] the value to get/set. > > +\item[\field{frame_height}] the value to get/set. > > +\item[\field{pixel_format}] the value to get/set. > > +\item[\field{min_buffers}] minimum buffers required to handle the > > + format (r/o). > > +\item[\field{max_buffers}] maximum buffers required to handle the > > + format (r/o). > > +\item[\field{frame_rate}] the value to get/set. > > +\item[\field{crop}] cropping (composing) rectangle. > > +\item[\field{num_planes}] number of planes used to store pixel data > > +(r/o). > > +\item[\field{plane_formats}] description of each plane. > > +\end{description} > > + > > +\begin{lstlisting} > > +struct virtio_video_get_params { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > +}; > > + > > +struct virtio_video_get_params_resp { > > + struct virtio_video_ctrl_hdr hdr; > > + struct virtio_video_params params; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{buf_type}] buffer type. > > +\item[\field{params}] parameter values. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > > + > > + > > +\begin{lstlisting} > > +struct virtio_video_set_params { > > + struct virtio_video_ctrl_hdr hdr; > > + struct virtio_video_params params; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{params}] parameters to set. > > +\end{description} > > + > > +Setting stream parameters might have side effects within the device. > > +For example, the device MAY perform alignment of width and height, > > +change the number of planes it uses for the format, or do whatever > > +changes that are required to continue normal operation using the > > +updated parameters. It is up to the driver to check the parameter set > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > > + > > +\end{description} > > + > > +\subsubsection{Device Operation: eventq} > > + > > +The device can report events on the event queue. The driver initially > > +populates the queue with device-writeable buffers. When the device > > +needs to report an event, it fills a buffer and notifies the driver. > > +The driver consumes the report and adds a new buffer to the virtqueue. > > + > > +\begin{lstlisting} > > +enum virtio_video_event_type { > > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > > + /* For all functions */ > > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > > + /* For decoder only */ > > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > > +}; > > + > > +struct virtio_video_event { > > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > > + le32 stream_id; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{event_type}] type of the triggered event . > > +\item[\field{stream_id}] id of the source stream. > > +\end{description} > > + > > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > > +whenever it encounters new resolution data in the stream. This > > +includes the case of the initial device configuration after metadata > > +has been parsed and the case of dynamic resolution change. > >
Hi, a couple of new comments: On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > The virtio video encoder device and decoder device provide functionalities > to encode and decode video stream respectively. > Though video encoder and decoder are provided as different devices, they use > a same protocol. > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > --- > content.tex | 1 + > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 580 insertions(+) > create mode 100644 virtio-video.tex > > diff --git a/content.tex b/content.tex > index 556b373..9e56839 100644 > --- a/content.tex > +++ b/content.tex > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > Requirements}\label{sec:Device \input{virtio-vsock.tex} > \input{virtio-fs.tex} > \input{virtio-rpmb.tex} > +\input{virtio-video.tex} > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > diff --git a/virtio-video.tex b/virtio-video.tex > new file mode 100644 > index 0000000..30e728d > --- /dev/null > +++ b/virtio-video.tex > @@ -0,0 +1,579 @@ > +\section{Video Device}\label{sec:Device Types / Video Device} > + > +The virtio video encoder device and decoder device are virtual devices that > +supports encoding and decoding respectively. Though the encoder and the > decoder +are different devices, they use the same protocol. > + > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID} > + > +\begin{description} > +\item[30] encoder device > +\item[31] decoder device > +\end{description} > + > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues} > + > +\begin{description} > +\item[0] controlq - queue for sending control commands. > +\item[1] eventq - queue for sending events happened in the device. > +\end{description} > + > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature > bits} + > +\begin{description} > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for > video + buffers. > +\end{description} > + > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device > / Feature bits} + > +The device MUST offer at least one of feature bits. > + > +\subsection{Device configuration layout}\label{sec:Device Types / Video > Device / Device configuration layout} + > +Video device configuration uses the following layout structure: > + > +\begin{lstlisting} > +struct virtio_video_config { > + le32 max_cap_len; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > + MUST set this value. > +\end{description} > + > +\subsection{Device Initialization}\label{sec:Device Types / Video Device / > Device Initialization} + > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / > Video Device / Device Initialization} + > +The driver SHOULD query device capability by using the > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > +setup. > + > +\subsection{Device Operation}\label{sec:Device Types / Video Device / > Device Operation} + > +The driver allocates input and output buffers and queues the buffers > +to the device. The device performs operations on the buffers according > +to the function in question. > + > +\subsubsection{Device Operation: Create stream} > + > +To process buffers, the device needs to associate them with a certain > +video stream (essentially, a context). Streams are created by > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > +determined by the device. > + > +\subsubsection{Device Operation: Create buffers} > + > +Buffers are used to store the actual data as well as the relevant > +metadata. Scatter lists are supported, so the buffer doesn't need to > +be contiguous in guest physical memory. > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > + resource that is backed by a buffer allocated from the driver's > + memory. > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > + is no longer needed. > +\end{itemize*} > + > +\subsubsection{Device Operation: Stream parameter control} > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters > for + input and output streams from the device. > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the > + device. > +\item After setting stream parameters, the driver may issue > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can > be + changed implicitly by the device during the set operation. > +\end{itemize*} > + > +\subsubsection{Device Operation: Process buffers} > + > +\begin{itemize*} > +\item If the function and the buffer type require so, write data to > +the buffer memory. > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > +processing in the device. > +\item The request completes asynchronously when the device has > +finished with the buffer. > +\end{itemize*} > + > +\subsubsection{Device Operation: Buffer processing control} > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > + return all of the already queued buffers. > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > + already queued buffers from the input or the output queue. This also > + includes input or output buffers that can be currently owned by the > + device's processing pipeline. > +\end{itemize*} > + > +\subsubsection{Device Operation: Asynchronous events} > + > +While processing buffers, the device can send asynchronous event > +notifications to the driver. The behaviour depends on the exact > +stream. For example, the decoder device sends a resolution change > +event when it encounters new resolution metadata in the stream. > + > +\subsubsection{Device Operation: Request header} > + > +All requests and responses on the control virt queue have a fixed > +header using the following layout structure and definitions: > + > +\begin{lstlisting} > +enum virtio_video_ctrl_type { > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > + > + /* request */ > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > + VIRTIO_VIDEO_T_STREAM_CREATE, > + VIRTIO_VIDEO_T_STREAM_DESTROY, > + VIRTIO_VIDEO_T_STREAM_DRAIN, > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > + VIRTIO_VIDEO_T_SET_PARAMS, > + VIRTIO_VIDEO_T_GET_PARAMS, > + > + /* response */ > + VIRTIO_VIDEO_S_OK = 0x0200, > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > + > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > +}; > + > +struct virtio_video_ctrl_hdr { > + le32 type; > + le32 stream_id; > + le32 len; /* Length of the structure in bytes. */ > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{type}] is the type of the driver request or the device > +response. > +\item[\field{stream_id}] specifies a target stream. > +\item[\field{len}] is the length of data in bytes, which includes > +length of the header. > +\end{description} > + > +\subsubsection{Device Operation: controlq} > + > +\begin{description} > + > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > +supported formats. > + > +The driver uses \field{struct virtio_video_get_capability} to send a > +query request. > + > +\begin{lstlisting} > +enum virtio_video_buf_type { > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > +}; > + > +struct virtio_video_get_capability { > + struct virtio_video_ctrl_hdr hdr; > + enum virtio_video_buf_type buf_type; > +}; > +\end{lstlisting} > +\begin{description} > +\item[\field{buf_type}] is the buffer type that the driver asks > +information about. The driver MUST set either > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > + > +The device responds a capability by using \field{struct > +virtio_video_get_capability_resp}. > +\begin{lstlisting} > +enum virtio_video_format { > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > + /* Raw formats */ > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > + VIRTIO_VIDEO_FORMAT_YUV420, > + VIRTIO_VIDEO_FORMAT_YVU420, > + > + /* Compressed formats */ > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > +}; > + > +enum virtio_video_profile { > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > + > + /* H.264 */ > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > VIRTIO_VIDEO_PROFILE_H264_MAIN, > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > + VIRTIO_VIDEO_PROFILE_H264_MAX = > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > + /* VP8 */ > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > + > + /* VP9 */ > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > + VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > +}; > + > +struct virtio_video_format_range { > + le32 min; > + le32 max; > + le32 step; > + u8 paddings[4]; > +}; > + > +struct virtio_video_format_desc { > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > + le64 mask; > + struct virtio_video_format_range width; > + struct virtio_video_format_range height; > + le32 num_rates; > + u8 padding[4]; > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > +}; > + > +struct virtio_video_get_capability_resp { > + struct virtio_video_ctrl_hdr hdr; > + le32 num_descs; > + /* Followed by struct virtio_video_format_desc desc[] */ > +}; > +\end{lstlisting} > + > +The format description \field{struct virtio_video_format_desc} > +includes the following fields: > +\begin{description} > +\item[\field{format}] specifies an image format. The device MUST set one > + of \field{enum virtio_video_format}. > +\item[\field{profile}] specifies a profile of the compressed image format > + specified in \field{format}. The driver SHOULD ignore this value if > + \field{format} is a raw format. So how should this be used? The spec does not define any way to set profile for the device. It is very important for encoder. Also, shouldn't the profile come together with level? Would make sense for encoders. > +\item[\field{mask}] is a bitset that represents the supported > + combination of input and output format. If \textit{i}-th bit is set > + in \field{mask} of \textit{j}-th \field{struct > + virtio_video_format_desc} for input, the device supports encoding or > + decoding from the \textit{j}-th input format to \textit{i}-th output > + format. > +\item[\field{width, height}] represents a range of resolutions > + supported by the device. If its \field{step} is not applicable, its > + \field{min} is equal to its \field{max}. > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In > case of decoder, the driver SHOULD ignore this value. > +\item[\field{frame_rates}] is an array of supported frame rates. > +\end{description} > + I'd guess frame rates depend on the resolution as well. This dependency was clear in the v1 spec, but in the v2 there is no dependency anymore. I think we need to update this. Best regards, Dmitry. > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > + within the device. > + > +\begin{lstlisting} > +enum virtio_video_mem_type { > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > +}; > + > +struct virtio_video_stream_create { > + struct virtio_video_ctrl_hdr hdr; > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > + char debug_name[64]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{in_mem_type}] is a type of buffer management for input > +buffers. The driver MUST set a value in \field{enum > +virtio_video_mem_type}. > +\item[\field{out_mem_type}] is a type of buffer management for output > +buffers. The driver MUST set a value in \field{enum > +virtio_video_mem_type}. > +\item[\field{debug_name}] is a text string for a debug purpose. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_stream_destroy { > + struct virtio_video_ctrl_hdr hdr; > +}; > +\end{lstlisting} > + > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > + queued buffers through the pipeline. > + > +\begin{lstlisting} > +struct virtio_video_stream_drain { > + struct virtio_video_ctrl_hdr hdr; > +}; > +\end{lstlisting} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_mem_entry { > + le64 addr; > + le32 length; > + u8 padding[4]; > +}; > + > +struct virtio_video_resource_create { > + struct virtio_video_ctrl_hdr hdr; > + le32 resource_id; > + le32 nr_entries; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{resource_id}] internal id of the resource. > +\item[\field{nr_entries}] number of \field{struct > + virtio_video_mem_entry} memory entries. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_resource_destroy { > + struct virtio_video_ctrl_hdr hdr; > + le32 resource_id; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{resource_id}] internal id of the resource. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > +queue. > + > +\begin{lstlisting} > +#define VIRTIO_VIDEO_MAX_PLANES 8 > + > +struct virtio_video_resource_queue { > + struct virtio_video_ctrl_hdr hdr; > + le32 buf_type; > + le32 resource_id; > + le64 timestamp; > + le32 nr_data_size; > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{buf_type}] buf_type of the . > +\item[\field{resource_id}] internal id of the resource. > +\item[\field{timestamp}] an abstract sequence counter that can be used > + for synchronisation. > +\item[\field{nr_data_size}] number of \field{data_size} entries. > +\item[\field{data_size}] number of data bytes within a plane. > +\end{description} > + > +\begin{lstlisting} > +enum virtio_video_buffer_flag { > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > + /* Encoder only */ > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > +}; > + > +struct virtio_video_resource_queue_resp { > + struct virtio_video_ctrl_hdr hdr; > + le64 timestamp; > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > + le32 size; /* Encoded size */ > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{timestamp}] an abstract sequence counter that can be used > + for synchronisation. > +\item[\field{flags}] mark specific buffers in the sequence. > +\item[\field{size}] data size in the buffer (encoder only). > +\end{description} > + > +The device sends a response to the queue request asynchronously when > +it has finished processing the buffer. > + > +The device SHOULD mark a buffer that triggered a processing error with > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > + > +The device MUST mark the last buffer with the > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > +sequence. > + > +In case of encoder, to denote a particular frame type the devie MUST > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > + > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > + buffers back from the input or the output queue of the device. The > + device SHOULD return all of the buffers from the respective queue as > + soon as possible without pushing the buffers through the processing > + pipeline. > + > +\begin{lstlisting} > +struct virtio_video_queue_clear { > + struct virtio_video_ctrl_hdr hdr; > + le32 buf_type; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{buf_type}] buffer type. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > + output of a stream. > + > +\begin{lstlisting} > +struct virtio_video_plane_format { > + le32 plane_size; > + le32 stride; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{plane_size}] size of the plane in bytes. > +\item[\field{stride}] stride used for the plane in bytes. > +\end{description} > + > +\begin{lstlisting} > +struct virtio_video_params { > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > + le32 frame_width; > + le32 frame_height; > + le32 min_buffers; > + le32 max_buffers; > + le32 frame_rate; > + struct virtio_video_crop { > + le32 left; > + le32 top; > + le32 width; > + le32 height; > + } crop; > + le32 num_planes; > + struct virtio_video_plane_format > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{frame_width}] the value to get/set. > +\item[\field{frame_height}] the value to get/set. > +\item[\field{pixel_format}] the value to get/set. > +\item[\field{min_buffers}] minimum buffers required to handle the > + format (r/o). > +\item[\field{max_buffers}] maximum buffers required to handle the > + format (r/o). > +\item[\field{frame_rate}] the value to get/set. > +\item[\field{crop}] cropping (composing) rectangle. > +\item[\field{num_planes}] number of planes used to store pixel data > +(r/o). > +\item[\field{plane_formats}] description of each plane. > +\end{description} > + > +\begin{lstlisting} > +struct virtio_video_get_params { > + struct virtio_video_ctrl_hdr hdr; > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > +}; > + > +struct virtio_video_get_params_resp { > + struct virtio_video_ctrl_hdr hdr; > + struct virtio_video_params params; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{buf_type}] buffer type. > +\item[\field{params}] parameter values. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > + > + > +\begin{lstlisting} > +struct virtio_video_set_params { > + struct virtio_video_ctrl_hdr hdr; > + struct virtio_video_params params; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{params}] parameters to set. > +\end{description} > + > +Setting stream parameters might have side effects within the device. > +For example, the device MAY perform alignment of width and height, > +change the number of planes it uses for the format, or do whatever > +changes that are required to continue normal operation using the > +updated parameters. It is up to the driver to check the parameter set > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > + > +\end{description} > + > +\subsubsection{Device Operation: eventq} > + > +The device can report events on the event queue. The driver initially > +populates the queue with device-writeable buffers. When the device > +needs to report an event, it fills a buffer and notifies the driver. > +The driver consumes the report and adds a new buffer to the virtqueue. > + > +\begin{lstlisting} > +enum virtio_video_event_type { > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > + /* For all functions */ > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > + /* For decoder only */ > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > +}; > + > +struct virtio_video_event { > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > + le32 stream_id; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{event_type}] type of the triggered event . > +\item[\field{stream_id}] id of the source stream. > +\end{description} > + > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > +whenever it encounters new resolution data in the stream. This > +includes the case of the initial device configuration after metadata > +has been parsed and the case of dynamic resolution change.
Hi Dmitry, On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi, > > a couple of new comments: > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > The virtio video encoder device and decoder device provide functionalities > > to encode and decode video stream respectively. > > Though video encoder and decoder are provided as different devices, they use > > a same protocol. > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > --- > > content.tex | 1 + > > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > > 2 files changed, 580 insertions(+) > > create mode 100644 virtio-video.tex > > > > diff --git a/content.tex b/content.tex > > index 556b373..9e56839 100644 > > --- a/content.tex > > +++ b/content.tex > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > Requirements}\label{sec:Device \input{virtio-vsock.tex} > > \input{virtio-fs.tex} > > \input{virtio-rpmb.tex} > > +\input{virtio-video.tex} > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > new file mode 100644 > > index 0000000..30e728d > > --- /dev/null > > +++ b/virtio-video.tex > > @@ -0,0 +1,579 @@ > > +\section{Video Device}\label{sec:Device Types / Video Device} > > + > > +The virtio video encoder device and decoder device are virtual devices that > > +supports encoding and decoding respectively. Though the encoder and the > > decoder +are different devices, they use the same protocol. > > + > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID} > > + > > +\begin{description} > > +\item[30] encoder device > > +\item[31] decoder device > > +\end{description} > > + > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues} > > + > > +\begin{description} > > +\item[0] controlq - queue for sending control commands. > > +\item[1] eventq - queue for sending events happened in the device. > > +\end{description} > > + > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature > > bits} + > > +\begin{description} > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for > > video + buffers. > > +\end{description} > > + > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device > > / Feature bits} + > > +The device MUST offer at least one of feature bits. > > + > > +\subsection{Device configuration layout}\label{sec:Device Types / Video > > Device / Device configuration layout} + > > +Video device configuration uses the following layout structure: > > + > > +\begin{lstlisting} > > +struct virtio_video_config { > > + le32 max_cap_len; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > > + MUST set this value. > > +\end{description} > > + > > +\subsection{Device Initialization}\label{sec:Device Types / Video Device / > > Device Initialization} + > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / > > Video Device / Device Initialization} + > > +The driver SHOULD query device capability by using the > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > > +setup. > > + > > +\subsection{Device Operation}\label{sec:Device Types / Video Device / > > Device Operation} + > > +The driver allocates input and output buffers and queues the buffers > > +to the device. The device performs operations on the buffers according > > +to the function in question. > > + > > +\subsubsection{Device Operation: Create stream} > > + > > +To process buffers, the device needs to associate them with a certain > > +video stream (essentially, a context). Streams are created by > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > +determined by the device. > > + > > +\subsubsection{Device Operation: Create buffers} > > + > > +Buffers are used to store the actual data as well as the relevant > > +metadata. Scatter lists are supported, so the buffer doesn't need to > > +be contiguous in guest physical memory. > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > > + resource that is backed by a buffer allocated from the driver's > > + memory. > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > > + is no longer needed. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Stream parameter control} > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters > > for + input and output streams from the device. > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the > > + device. > > +\item After setting stream parameters, the driver may issue > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can > > be + changed implicitly by the device during the set operation. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Process buffers} > > + > > +\begin{itemize*} > > +\item If the function and the buffer type require so, write data to > > +the buffer memory. > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > > +processing in the device. > > +\item The request completes asynchronously when the device has > > +finished with the buffer. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Buffer processing control} > > + > > +\begin{itemize*} > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > > + return all of the already queued buffers. > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > > + already queued buffers from the input or the output queue. This also > > + includes input or output buffers that can be currently owned by the > > + device's processing pipeline. > > +\end{itemize*} > > + > > +\subsubsection{Device Operation: Asynchronous events} > > + > > +While processing buffers, the device can send asynchronous event > > +notifications to the driver. The behaviour depends on the exact > > +stream. For example, the decoder device sends a resolution change > > +event when it encounters new resolution metadata in the stream. > > + > > +\subsubsection{Device Operation: Request header} > > + > > +All requests and responses on the control virt queue have a fixed > > +header using the following layout structure and definitions: > > + > > +\begin{lstlisting} > > +enum virtio_video_ctrl_type { > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > + > > + /* request */ > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > + VIRTIO_VIDEO_T_SET_PARAMS, > > + VIRTIO_VIDEO_T_GET_PARAMS, > > + > > + /* response */ > > + VIRTIO_VIDEO_S_OK = 0x0200, > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > + > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > +}; > > + > > +struct virtio_video_ctrl_hdr { > > + le32 type; > > + le32 stream_id; > > + le32 len; /* Length of the structure in bytes. */ > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{type}] is the type of the driver request or the device > > +response. > > +\item[\field{stream_id}] specifies a target stream. > > +\item[\field{len}] is the length of data in bytes, which includes > > +length of the header. > > +\end{description} > > + > > +\subsubsection{Device Operation: controlq} > > + > > +\begin{description} > > + > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > > +supported formats. > > + > > +The driver uses \field{struct virtio_video_get_capability} to send a > > +query request. > > + > > +\begin{lstlisting} > > +enum virtio_video_buf_type { > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > +}; > > + > > +struct virtio_video_get_capability { > > + struct virtio_video_ctrl_hdr hdr; > > + enum virtio_video_buf_type buf_type; > > +}; > > +\end{lstlisting} > > +\begin{description} > > +\item[\field{buf_type}] is the buffer type that the driver asks > > +information about. The driver MUST set either > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > + > > +The device responds a capability by using \field{struct > > +virtio_video_get_capability_resp}. > > +\begin{lstlisting} > > +enum virtio_video_format { > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > + /* Raw formats */ > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > + VIRTIO_VIDEO_FORMAT_YUV420, > > + VIRTIO_VIDEO_FORMAT_YVU420, > > + > > + /* Compressed formats */ > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > +}; > > + > > +enum virtio_video_profile { > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > + > > + /* H.264 */ > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > > VIRTIO_VIDEO_PROFILE_H264_MAIN, > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > > + /* VP8 */ > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > > + > > + /* VP9 */ > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > +}; > > + > > +struct virtio_video_format_range { > > + le32 min; > > + le32 max; > > + le32 step; > > + u8 paddings[4]; > > +}; > > + > > +struct virtio_video_format_desc { > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > > + le64 mask; > > + struct virtio_video_format_range width; > > + struct virtio_video_format_range height; > > + le32 num_rates; > > + u8 padding[4]; > > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > > +}; > > + > > +struct virtio_video_get_capability_resp { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 num_descs; > > + /* Followed by struct virtio_video_format_desc desc[] */ > > +}; > > +\end{lstlisting} > > + > > +The format description \field{struct virtio_video_format_desc} > > +includes the following fields: > > +\begin{description} > > +\item[\field{format}] specifies an image format. The device MUST set one > > + of \field{enum virtio_video_format}. > > +\item[\field{profile}] specifies a profile of the compressed image format > > + specified in \field{format}. The driver SHOULD ignore this value if > > + \field{format} is a raw format. > > So how should this be used? The spec does not define any way to set profile for > the device. It is very important for encoder. > Thank you for pointing this. These points are overlooked, as I didn't care about encoder usage enough. After thinking it again, I think it's not a very good idea to include supported profiles and levels in a struct for capability. This is because these values are available only for limited number of formats. Also, it's true that we need to have a way to set these values as Dmitry pointed. Instead, it would make more sense to have additional three types of controls for profiles, levels, and bitrates: (1) QUERY_CONTROL: Query values supported by the device (2) GET_CONTROL: Read a value that is set in the device (3) SET_CONTROL: Set a value in the device These operations are similar to V4L2 controls. (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and G_CTRL in V4L2, respectively. Also, (3) would be similar to enum virtio_video_control_type in the virtio-video v1 driver implementation in https://markmail.org/message/dwghwdqsbl3gsjxu . For QUERY_CONTROL, my idea is like this: enum virtio_video_control_type { VIRTIO_VIDEO_CONTROL_UNDEFINED = 0, VIRTIO_VIDEO_CONTROL_BITRATE = 0x100, VIRTIO_VIDEO_CONTROL_PROFILE, VIRTIO_VIDEO_CONTROL_LEVEL, }; struct virtio_video_query_control { struct virtio_video_ctrl_hdr hdr; le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */ le32 length; /* Followed by additional data. * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE, * the device must pass a codec format like H264 or VP9. * The requred data must be defined in the specification. */ }; struct virtio_video_query_control_resp { struct virtio_video_ctrl_hdr hdr; le32 length; u8 padding[4]; /* Followed by data corresponds to the specified control. * The type of data must be defined in the spec. * For example, if the driver queries profiles, this part should be * an array of supported profiles of a given format. */ }; WDYT? > Also, shouldn't the profile come together with level? Would make sense for > encoders. Yeah. So, in the above idea of QUERY_CONTROL, profile should be required when querying supported levels. > > > +\item[\field{mask}] is a bitset that represents the supported > > + combination of input and output format. If \textit{i}-th bit is set > > + in \field{mask} of \textit{j}-th \field{struct > > + virtio_video_format_desc} for input, the device supports encoding or > > + decoding from the \textit{j}-th input format to \textit{i}-th output > > + format. > > +\item[\field{width, height}] represents a range of resolutions > > + supported by the device. If its \field{step} is not applicable, its > > + \field{min} is equal to its \field{max}. > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In > > case of decoder, the driver SHOULD ignore this value. > > +\item[\field{frame_rates}] is an array of supported frame rates. > > +\end{description} > > + > > I'd guess frame rates depend on the resolution as well. This dependency was > clear in the v1 spec, but in the v2 there is no dependency anymore. I think we > need to update this. That's a good point. I missed that dependency when updating the structures. So, let me update the structs like the following: struct virtio_video_format_frame { struct virtio_video_format_range width; struct virtio_video_format_range height; le32 num_rates; u8 padding[4]; /* Followed by struct virtio_video_format_range frame_rates[] */ }; struct virtio_video_format_desc { le64 mask; le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ le32 planes_layout; /* See the thread [v2 0/1] */ le32 num_frames; u8 padding[4]; /* Followed by struct virtio_video_format_frame frames[] */ }; Best regards, Keiichi. > > Best regards, > Dmitry. > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > > + within the device. > > + > > +\begin{lstlisting} > > +enum virtio_video_mem_type { > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > +}; > > + > > +struct virtio_video_stream_create { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > + char debug_name[64]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{in_mem_type}] is a type of buffer management for input > > +buffers. The driver MUST set a value in \field{enum > > +virtio_video_mem_type}. > > +\item[\field{out_mem_type}] is a type of buffer management for output > > +buffers. The driver MUST set a value in \field{enum > > +virtio_video_mem_type}. > > +\item[\field{debug_name}] is a text string for a debug purpose. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_stream_destroy { > > + struct virtio_video_ctrl_hdr hdr; > > +}; > > +\end{lstlisting} > > + > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > > + queued buffers through the pipeline. > > + > > +\begin{lstlisting} > > +struct virtio_video_stream_drain { > > + struct virtio_video_ctrl_hdr hdr; > > +}; > > +\end{lstlisting} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_mem_entry { > > + le64 addr; > > + le32 length; > > + u8 padding[4]; > > +}; > > + > > +struct virtio_video_resource_create { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 resource_id; > > + le32 nr_entries; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{resource_id}] internal id of the resource. > > +\item[\field{nr_entries}] number of \field{struct > > + virtio_video_mem_entry} memory entries. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > > + within the device. > > + > > +\begin{lstlisting} > > +struct virtio_video_resource_destroy { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 resource_id; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{resource_id}] internal id of the resource. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > > +queue. > > + > > +\begin{lstlisting} > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > + > > +struct virtio_video_resource_queue { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 buf_type; > > + le32 resource_id; > > + le64 timestamp; > > + le32 nr_data_size; > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{buf_type}] buf_type of the . > > +\item[\field{resource_id}] internal id of the resource. > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > + for synchronisation. > > +\item[\field{nr_data_size}] number of \field{data_size} entries. > > +\item[\field{data_size}] number of data bytes within a plane. > > +\end{description} > > + > > +\begin{lstlisting} > > +enum virtio_video_buffer_flag { > > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > > + /* Encoder only */ > > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > > +}; > > + > > +struct virtio_video_resource_queue_resp { > > + struct virtio_video_ctrl_hdr hdr; > > + le64 timestamp; > > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > > + le32 size; /* Encoded size */ > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > + for synchronisation. > > +\item[\field{flags}] mark specific buffers in the sequence. > > +\item[\field{size}] data size in the buffer (encoder only). > > +\end{description} > > + > > +The device sends a response to the queue request asynchronously when > > +it has finished processing the buffer. > > + > > +The device SHOULD mark a buffer that triggered a processing error with > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > > + > > +The device MUST mark the last buffer with the > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > +sequence. > > + > > +In case of encoder, to denote a particular frame type the devie MUST > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > > + > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > > + buffers back from the input or the output queue of the device. The > > + device SHOULD return all of the buffers from the respective queue as > > + soon as possible without pushing the buffers through the processing > > + pipeline. > > + > > +\begin{lstlisting} > > +struct virtio_video_queue_clear { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 buf_type; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{buf_type}] buffer type. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > > + output of a stream. > > + > > +\begin{lstlisting} > > +struct virtio_video_plane_format { > > + le32 plane_size; > > + le32 stride; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{plane_size}] size of the plane in bytes. > > +\item[\field{stride}] stride used for the plane in bytes. > > +\end{description} > > + > > +\begin{lstlisting} > > +struct virtio_video_params { > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > > + le32 frame_width; > > + le32 frame_height; > > + le32 min_buffers; > > + le32 max_buffers; > > + le32 frame_rate; > > + struct virtio_video_crop { > > + le32 left; > > + le32 top; > > + le32 width; > > + le32 height; > > + } crop; > > + le32 num_planes; > > + struct virtio_video_plane_format > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{frame_width}] the value to get/set. > > +\item[\field{frame_height}] the value to get/set. > > +\item[\field{pixel_format}] the value to get/set. > > +\item[\field{min_buffers}] minimum buffers required to handle the > > + format (r/o). > > +\item[\field{max_buffers}] maximum buffers required to handle the > > + format (r/o). > > +\item[\field{frame_rate}] the value to get/set. > > +\item[\field{crop}] cropping (composing) rectangle. > > +\item[\field{num_planes}] number of planes used to store pixel data > > +(r/o). > > +\item[\field{plane_formats}] description of each plane. > > +\end{description} > > + > > +\begin{lstlisting} > > +struct virtio_video_get_params { > > + struct virtio_video_ctrl_hdr hdr; > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > +}; > > + > > +struct virtio_video_get_params_resp { > > + struct virtio_video_ctrl_hdr hdr; > > + struct virtio_video_params params; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{buf_type}] buffer type. > > +\item[\field{params}] parameter values. > > +\end{description} > > + > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > > + > > + > > +\begin{lstlisting} > > +struct virtio_video_set_params { > > + struct virtio_video_ctrl_hdr hdr; > > + struct virtio_video_params params; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{params}] parameters to set. > > +\end{description} > > + > > +Setting stream parameters might have side effects within the device. > > +For example, the device MAY perform alignment of width and height, > > +change the number of planes it uses for the format, or do whatever > > +changes that are required to continue normal operation using the > > +updated parameters. It is up to the driver to check the parameter set > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > > + > > +\end{description} > > + > > +\subsubsection{Device Operation: eventq} > > + > > +The device can report events on the event queue. The driver initially > > +populates the queue with device-writeable buffers. When the device > > +needs to report an event, it fills a buffer and notifies the driver. > > +The driver consumes the report and adds a new buffer to the virtqueue. > > + > > +\begin{lstlisting} > > +enum virtio_video_event_type { > > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > > + /* For all functions */ > > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > > + /* For decoder only */ > > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > > +}; > > + > > +struct virtio_video_event { > > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > > + le32 stream_id; > > + u8 padding[4]; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > +\item[\field{event_type}] type of the triggered event . > > +\item[\field{stream_id}] id of the source stream. > > +\end{description} > > + > > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > > +whenever it encounters new resolution data in the stream. This > > +includes the case of the initial device configuration after metadata > > +has been parsed and the case of dynamic resolution change. > >
Hi Keiichi, thanks for the updates, please see my comments below. On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote: > Hi Dmitry, > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi, > > > > a couple of new comments: > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > The virtio video encoder device and decoder device provide > > > functionalities > > > to encode and decode video stream respectively. > > > Though video encoder and decoder are provided as different devices, they > > > use a same protocol. > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > > --- > > > > > > content.tex | 1 + > > > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > > > 2 files changed, 580 insertions(+) > > > create mode 100644 virtio-video.tex > > > > > > diff --git a/content.tex b/content.tex > > > index 556b373..9e56839 100644 > > > --- a/content.tex > > > +++ b/content.tex > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > > Requirements}\label{sec:Device \input{virtio-vsock.tex} > > > > > > \input{virtio-fs.tex} > > > \input{virtio-rpmb.tex} > > > > > > +\input{virtio-video.tex} > > > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > > new file mode 100644 > > > index 0000000..30e728d > > > --- /dev/null > > > +++ b/virtio-video.tex > > > @@ -0,0 +1,579 @@ > > > +\section{Video Device}\label{sec:Device Types / Video Device} > > > + > > > +The virtio video encoder device and decoder device are virtual devices > > > that +supports encoding and decoding respectively. Though the encoder > > > and the decoder +are different devices, they use the same protocol. > > > + > > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device > > > ID} > > > + > > > +\begin{description} > > > +\item[30] encoder device > > > +\item[31] decoder device > > > +\end{description} > > > + > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / > > > Virtqueues} + > > > +\begin{description} > > > +\item[0] controlq - queue for sending control commands. > > > +\item[1] eventq - queue for sending events happened in the device. > > > +\end{description} > > > + > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / > > > Feature > > > bits} + > > > +\begin{description} > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used > > > for > > > video + buffers. > > > +\end{description} > > > + > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video > > > Device > > > / Feature bits} + > > > +The device MUST offer at least one of feature bits. > > > + > > > +\subsection{Device configuration layout}\label{sec:Device Types / Video > > > Device / Device configuration layout} + > > > +Video device configuration uses the following layout structure: > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_config { > > > + le32 max_cap_len; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > > > + MUST set this value. > > > +\end{description} > > > + > > > +\subsection{Device Initialization}\label{sec:Device Types / Video > > > Device / > > > Device Initialization} + > > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / > > > Video Device / Device Initialization} + > > > +The driver SHOULD query device capability by using the > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > > > +setup. > > > + > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device / > > > Device Operation} + > > > +The driver allocates input and output buffers and queues the buffers > > > +to the device. The device performs operations on the buffers according > > > +to the function in question. > > > + > > > +\subsubsection{Device Operation: Create stream} > > > + > > > +To process buffers, the device needs to associate them with a certain > > > +video stream (essentially, a context). Streams are created by > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > > +determined by the device. > > > + > > > +\subsubsection{Device Operation: Create buffers} > > > + > > > +Buffers are used to store the actual data as well as the relevant > > > +metadata. Scatter lists are supported, so the buffer doesn't need to > > > +be contiguous in guest physical memory. > > > + > > > +\begin{itemize*} > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > > > + resource that is backed by a buffer allocated from the driver's > > > + memory. > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > > > + is no longer needed. > > > +\end{itemize*} > > > + > > > +\subsubsection{Device Operation: Stream parameter control} > > > + > > > +\begin{itemize*} > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream > > > parameters > > > for + input and output streams from the device. > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to > > > the + device. > > > +\item After setting stream parameters, the driver may issue > > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output > > > can be + changed implicitly by the device during the set operation. > > > +\end{itemize*} > > > + > > > +\subsubsection{Device Operation: Process buffers} > > > + > > > +\begin{itemize*} > > > +\item If the function and the buffer type require so, write data to > > > +the buffer memory. > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > > > +processing in the device. > > > +\item The request completes asynchronously when the device has > > > +finished with the buffer. > > > +\end{itemize*} > > > + > > > +\subsubsection{Device Operation: Buffer processing control} > > > + > > > +\begin{itemize*} > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > > > + return all of the already queued buffers. > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > > > + already queued buffers from the input or the output queue. This also > > > + includes input or output buffers that can be currently owned by the > > > + device's processing pipeline. > > > +\end{itemize*} > > > + > > > +\subsubsection{Device Operation: Asynchronous events} > > > + > > > +While processing buffers, the device can send asynchronous event > > > +notifications to the driver. The behaviour depends on the exact > > > +stream. For example, the decoder device sends a resolution change > > > +event when it encounters new resolution metadata in the stream. > > > + > > > +\subsubsection{Device Operation: Request header} > > > + > > > +All requests and responses on the control virt queue have a fixed > > > +header using the following layout structure and definitions: > > > + > > > +\begin{lstlisting} > > > +enum virtio_video_ctrl_type { > > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > > + > > > + /* request */ > > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > > + VIRTIO_VIDEO_T_SET_PARAMS, > > > + VIRTIO_VIDEO_T_GET_PARAMS, > > > + > > > + /* response */ > > > + VIRTIO_VIDEO_S_OK = 0x0200, > > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > > + > > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > > +}; > > > + > > > +struct virtio_video_ctrl_hdr { > > > + le32 type; > > > + le32 stream_id; > > > + le32 len; /* Length of the structure in bytes. */ > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{type}] is the type of the driver request or the device > > > +response. > > > +\item[\field{stream_id}] specifies a target stream. > > > +\item[\field{len}] is the length of data in bytes, which includes > > > +length of the header. > > > +\end{description} > > > + > > > +\subsubsection{Device Operation: controlq} > > > + > > > +\begin{description} > > > + > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > > > +supported formats. > > > + > > > +The driver uses \field{struct virtio_video_get_capability} to send a > > > +query request. > > > + > > > +\begin{lstlisting} > > > +enum virtio_video_buf_type { > > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > > +}; > > > + > > > +struct virtio_video_get_capability { > > > + struct virtio_video_ctrl_hdr hdr; > > > + enum virtio_video_buf_type buf_type; > > > +}; > > > +\end{lstlisting} > > > +\begin{description} > > > +\item[\field{buf_type}] is the buffer type that the driver asks > > > +information about. The driver MUST set either > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > > + > > > +The device responds a capability by using \field{struct > > > +virtio_video_get_capability_resp}. > > > +\begin{lstlisting} > > > +enum virtio_video_format { > > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > > + /* Raw formats */ > > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > > + VIRTIO_VIDEO_FORMAT_YUV420, > > > + VIRTIO_VIDEO_FORMAT_YVU420, > > > + > > > + /* Compressed formats */ > > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > > +}; > > > + > > > +enum virtio_video_profile { > > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > > + > > > + /* H.264 */ > > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > > > VIRTIO_VIDEO_PROFILE_H264_MAIN, > > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > > > + /* VP8 */ > > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > > > + > > > + /* VP9 */ > > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = > > > VIRTIO_VIDEO_PROFILE_VP9_MIN, > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > +}; > > > + > > > +struct virtio_video_format_range { > > > + le32 min; > > > + le32 max; > > > + le32 step; > > > + u8 paddings[4]; > > > +}; > > > + > > > +struct virtio_video_format_desc { > > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > > > + le64 mask; > > > + struct virtio_video_format_range width; > > > + struct virtio_video_format_range height; > > > + le32 num_rates; > > > + u8 padding[4]; > > > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > > > +}; > > > + > > > +struct virtio_video_get_capability_resp { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 num_descs; > > > + /* Followed by struct virtio_video_format_desc desc[] */ > > > +}; > > > +\end{lstlisting} > > > + > > > +The format description \field{struct virtio_video_format_desc} > > > +includes the following fields: > > > +\begin{description} > > > +\item[\field{format}] specifies an image format. The device MUST set > > > one > > > + of \field{enum virtio_video_format}. > > > +\item[\field{profile}] specifies a profile of the compressed image > > > format > > > + specified in \field{format}. The driver SHOULD ignore this value if > > > + \field{format} is a raw format. > > > > So how should this be used? The spec does not define any way to set > > profile for the device. It is very important for encoder. > > Thank you for pointing this. > These points are overlooked, as I didn't care about encoder usage enough. > > After thinking it again, I think it's not a very good idea to include > supported profiles and levels in a struct for capability. > This is because these values are available only for limited number of > formats. Also, it's true that we need to have a way to set these values as > Dmitry pointed. > Yes, you are right. In fact, the approach of the v1 spec to keep controls separately was not correct. > Instead, it would make more sense to have additional three types of > controls for profiles, levels, and bitrates: > (1) QUERY_CONTROL: Query values supported by the device > (2) GET_CONTROL: Read a value that is set in the device > (3) SET_CONTROL: Set a value in the device > > These operations are similar to V4L2 controls. > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and > G_CTRL in V4L2, respectively. > Also, (3) would be similar to enum virtio_video_control_type in the > virtio-video v1 driver implementation in > https://markmail.org/message/dwghwdqsbl3gsjxu . > > For QUERY_CONTROL, my idea is like this: > > enum virtio_video_control_type { > VIRTIO_VIDEO_CONTROL_UNDEFINED = 0, > > VIRTIO_VIDEO_CONTROL_BITRATE = 0x100, > VIRTIO_VIDEO_CONTROL_PROFILE, > VIRTIO_VIDEO_CONTROL_LEVEL, > }; > > struct virtio_video_query_control { > struct virtio_video_ctrl_hdr hdr; > le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */ > le32 length; > /* Followed by additional data. > * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE, > * the device must pass a codec format like H264 or VP9. > * The requred data must be defined in the specification. > */ > }; > > struct virtio_video_query_control_resp { > struct virtio_video_ctrl_hdr hdr; > le32 length; > u8 padding[4]; > /* Followed by data corresponds to the specified control. > * The type of data must be defined in the spec. > * For example, if the driver queries profiles, this part should be > * an array of supported profiles of a given format. > */ > }; > > WDYT? > I think virtio_video_control_type should make sense. But I would disagree with the need to have new QUERY_CONTROL and GET_CONTROL. I assume the set of supported controls is fixed for some particular format on a given IP. So we'd propose to include controls into format descriptors, so we don't need to QUERY_CONTROL. This way (with 'virtio_video_format_list') we can define not-contiguos ranges, e.g. for profiles. struct virtio_video_format_frame { /* As proposed in Keiichi's prev email */ }; virtio_video_format_list { le32 num_entries; u8 padding[4]; /* Followed by le64 entries[] */ }; struct virtio_video_format_control { le32 type; u8 padding[4]; struct virtio_video_format_list values; }; struct virtio_video_format_desc { le64 mask; le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ le32 planes_layout; /* See the thread [v2 0/1] */ le32 num_frames; le32 num_controls; /* Followed by struct virtio_video_format_frame frames[] */ /* Followed by struct virtio_video_format_control controls[] */ }; SET_CONTROL seems to be mandatory. If it succeeds, we can store the current value locally, so there is no need to have GET_CONTROL. The only exception is the initial (default) control value in the device. But with the removal of 'function' and with addition of 'caps' instead, the way to provide defaults is gone. So I suppose for formats we'll be just using GET on driver start to get the 'defaults'. But the thing is that for formats there are other uses for GET, but for controls GET apparently does not make to much sense at runtime. > > Also, shouldn't the profile come together with level? Would make sense for > > encoders. > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be > required when querying supported levels. > So probably should be enumerated together, not queried, as per the comment above. > > > +\item[\field{mask}] is a bitset that represents the supported > > > + combination of input and output format. If \textit{i}-th bit is set > > > + in \field{mask} of \textit{j}-th \field{struct > > > + virtio_video_format_desc} for input, the device supports encoding or > > > + decoding from the \textit{j}-th input format to \textit{i}-th output > > > + format. > > > +\item[\field{width, height}] represents a range of resolutions > > > + supported by the device. If its \field{step} is not applicable, its > > > + \field{min} is equal to its \field{max}. > > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. > > > In > > > case of decoder, the driver SHOULD ignore this value. > > > +\item[\field{frame_rates}] is an array of supported frame rates. > > > +\end{description} > > > + > > > > I'd guess frame rates depend on the resolution as well. This dependency > > was > > clear in the v1 spec, but in the v2 there is no dependency anymore. I > > think we need to update this. > > That's a good point. I missed that dependency when updating the structures. > So, let me update the structs like the following: > > struct virtio_video_format_frame { > struct virtio_video_format_range width; > struct virtio_video_format_range height; > le32 num_rates; > u8 padding[4]; > /* Followed by struct virtio_video_format_range frame_rates[] */ > }; > > struct virtio_video_format_desc { > le64 mask; > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > le32 planes_layout; /* See the thread [v2 0/1] */ > le32 num_frames; > u8 padding[4]; > /* Followed by struct virtio_video_format_frame frames[] */ > }; > Yes, I do agree with this approach. Best regards, Dmitry. > Best regards, > Keiichi. > > > Best regards, > > Dmitry. > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > > > + within the device. > > > + > > > +\begin{lstlisting} > > > +enum virtio_video_mem_type { > > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > > +}; > > > + > > > +struct virtio_video_stream_create { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > > + char debug_name[64]; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{in_mem_type}] is a type of buffer management for input > > > +buffers. The driver MUST set a value in \field{enum > > > +virtio_video_mem_type}. > > > +\item[\field{out_mem_type}] is a type of buffer management for output > > > +buffers. The driver MUST set a value in \field{enum > > > +virtio_video_mem_type}. > > > +\item[\field{debug_name}] is a text string for a debug purpose. > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > > > + within the device. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_stream_destroy { > > > + struct virtio_video_ctrl_hdr hdr; > > > +}; > > > +\end{lstlisting} > > > + > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > > > + queued buffers through the pipeline. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_stream_drain { > > > + struct virtio_video_ctrl_hdr hdr; > > > +}; > > > +\end{lstlisting} > > > + > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > > > + within the device. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_mem_entry { > > > + le64 addr; > > > + le32 length; > > > + u8 padding[4]; > > > +}; > > > + > > > +struct virtio_video_resource_create { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 resource_id; > > > + le32 nr_entries; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{resource_id}] internal id of the resource. > > > +\item[\field{nr_entries}] number of \field{struct > > > + virtio_video_mem_entry} memory entries. > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > > > + within the device. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_resource_destroy { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 resource_id; > > > + u8 padding[4]; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{resource_id}] internal id of the resource. > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > > > +queue. > > > + > > > +\begin{lstlisting} > > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > > + > > > +struct virtio_video_resource_queue { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 buf_type; > > > + le32 resource_id; > > > + le64 timestamp; > > > + le32 nr_data_size; > > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{buf_type}] buf_type of the . > > > +\item[\field{resource_id}] internal id of the resource. > > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > > + for synchronisation. > > > +\item[\field{nr_data_size}] number of \field{data_size} entries. > > > +\item[\field{data_size}] number of data bytes within a plane. > > > +\end{description} > > > + > > > +\begin{lstlisting} > > > +enum virtio_video_buffer_flag { > > > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > > > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > > > + /* Encoder only */ > > > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > > > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > > > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > > > +}; > > > + > > > +struct virtio_video_resource_queue_resp { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le64 timestamp; > > > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > > > + le32 size; /* Encoded size */ > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > > + for synchronisation. > > > +\item[\field{flags}] mark specific buffers in the sequence. > > > +\item[\field{size}] data size in the buffer (encoder only). > > > +\end{description} > > > + > > > +The device sends a response to the queue request asynchronously when > > > +it has finished processing the buffer. > > > + > > > +The device SHOULD mark a buffer that triggered a processing error with > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > > > + > > > +The device MUST mark the last buffer with the > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > > +sequence. > > > + > > > +In case of encoder, to denote a particular frame type the devie MUST > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > > > + > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > > > + buffers back from the input or the output queue of the device. The > > > + device SHOULD return all of the buffers from the respective queue as > > > + soon as possible without pushing the buffers through the processing > > > + pipeline. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_queue_clear { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 buf_type; > > > + u8 padding[4]; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{buf_type}] buffer type. > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > > > + output of a stream. > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_plane_format { > > > + le32 plane_size; > > > + le32 stride; > > > + u8 padding[4]; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{plane_size}] size of the plane in bytes. > > > +\item[\field{stride}] stride used for the plane in bytes. > > > +\end{description} > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_params { > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > > > + le32 frame_width; > > > + le32 frame_height; > > > + le32 min_buffers; > > > + le32 max_buffers; > > > + le32 frame_rate; > > > + struct virtio_video_crop { > > > + le32 left; > > > + le32 top; > > > + le32 width; > > > + le32 height; > > > + } crop; > > > + le32 num_planes; > > > + struct virtio_video_plane_format > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{frame_width}] the value to get/set. > > > +\item[\field{frame_height}] the value to get/set. > > > +\item[\field{pixel_format}] the value to get/set. > > > +\item[\field{min_buffers}] minimum buffers required to handle the > > > + format (r/o). > > > +\item[\field{max_buffers}] maximum buffers required to handle the > > > + format (r/o). > > > +\item[\field{frame_rate}] the value to get/set. > > > +\item[\field{crop}] cropping (composing) rectangle. > > > +\item[\field{num_planes}] number of planes used to store pixel data > > > +(r/o). > > > +\item[\field{plane_formats}] description of each plane. > > > +\end{description} > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_get_params { > > > + struct virtio_video_ctrl_hdr hdr; > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > +}; > > > + > > > +struct virtio_video_get_params_resp { > > > + struct virtio_video_ctrl_hdr hdr; > > > + struct virtio_video_params params; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{buf_type}] buffer type. > > > +\item[\field{params}] parameter values. > > > +\end{description} > > > + > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > > > + > > > + > > > +\begin{lstlisting} > > > +struct virtio_video_set_params { > > > + struct virtio_video_ctrl_hdr hdr; > > > + struct virtio_video_params params; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{params}] parameters to set. > > > +\end{description} > > > + > > > +Setting stream parameters might have side effects within the device. > > > +For example, the device MAY perform alignment of width and height, > > > +change the number of planes it uses for the format, or do whatever > > > +changes that are required to continue normal operation using the > > > +updated parameters. It is up to the driver to check the parameter set > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > > > + > > > +\end{description} > > > + > > > +\subsubsection{Device Operation: eventq} > > > + > > > +The device can report events on the event queue. The driver initially > > > +populates the queue with device-writeable buffers. When the device > > > +needs to report an event, it fills a buffer and notifies the driver. > > > +The driver consumes the report and adds a new buffer to the virtqueue. > > > + > > > +\begin{lstlisting} > > > +enum virtio_video_event_type { > > > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > > > + /* For all functions */ > > > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > > > + /* For decoder only */ > > > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > > > +}; > > > + > > > +struct virtio_video_event { > > > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > > > + le32 stream_id; > > > + u8 padding[4]; > > > +}; > > > +\end{lstlisting} > > > + > > > +\begin{description} > > > +\item[\field{event_type}] type of the triggered event . > > > +\item[\field{stream_id}] id of the source stream. > > > +\end{description} > > > + > > > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > > > +whenever it encounters new resolution data in the stream. This > > > +includes the case of the initial device configuration after metadata > > > +has been parsed and the case of dynamic resolution change.
Hi Dmitry, On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi Keiichi, > > thanks for the updates, please see my comments below. > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote: > > Hi Dmitry, > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> > wrote: > > > Hi, > > > > > > a couple of new comments: > > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > > > The virtio video encoder device and decoder device provide > > > > functionalities > > > > to encode and decode video stream respectively. > > > > Though video encoder and decoder are provided as different devices, they > > > > use a same protocol. > > > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > > > --- > > > > > > > > content.tex | 1 + > > > > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > > > > 2 files changed, 580 insertions(+) > > > > create mode 100644 virtio-video.tex > > > > > > > > diff --git a/content.tex b/content.tex > > > > index 556b373..9e56839 100644 > > > > --- a/content.tex > > > > +++ b/content.tex > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex} > > > > > > > > \input{virtio-fs.tex} > > > > \input{virtio-rpmb.tex} > > > > > > > > +\input{virtio-video.tex} > > > > > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > > > new file mode 100644 > > > > index 0000000..30e728d > > > > --- /dev/null > > > > +++ b/virtio-video.tex > > > > @@ -0,0 +1,579 @@ > > > > +\section{Video Device}\label{sec:Device Types / Video Device} > > > > + > > > > +The virtio video encoder device and decoder device are virtual devices > > > > that +supports encoding and decoding respectively. Though the encoder > > > > and the decoder +are different devices, they use the same protocol. > > > > + > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device > > > > ID} > > > > + > > > > +\begin{description} > > > > +\item[30] encoder device > > > > +\item[31] decoder device > > > > +\end{description} > > > > + > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / > > > > Virtqueues} + > > > > +\begin{description} > > > > +\item[0] controlq - queue for sending control commands. > > > > +\item[1] eventq - queue for sending events happened in the device. > > > > +\end{description} > > > > + > > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / > > > > Feature > > > > bits} + > > > > +\begin{description} > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used > > > > for > > > > video + buffers. > > > > +\end{description} > > > > + > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video > > > > Device > > > > / Feature bits} + > > > > +The device MUST offer at least one of feature bits. > > > > + > > > > +\subsection{Device configuration layout}\label{sec:Device Types / Video > > > > Device / Device configuration layout} + > > > > +Video device configuration uses the following layout structure: > > > > + > > > > +\begin{lstlisting} > > > > +struct virtio_video_config { > > > > + le32 max_cap_len; > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\begin{description} > > > > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > > > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > > > > + MUST set this value. > > > > +\end{description} > > > > + > > > > +\subsection{Device Initialization}\label{sec:Device Types / Video > > > > Device / > > > > Device Initialization} + > > > > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / > > > > Video Device / Device Initialization} + > > > > +The driver SHOULD query device capability by using the > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > > > > +setup. > > > > + > > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device / > > > > Device Operation} + > > > > +The driver allocates input and output buffers and queues the buffers > > > > +to the device. The device performs operations on the buffers according > > > > +to the function in question. > > > > + > > > > +\subsubsection{Device Operation: Create stream} > > > > + > > > > +To process buffers, the device needs to associate them with a certain > > > > +video stream (essentially, a context). Streams are created by > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > > > +determined by the device. > > > > + > > > > +\subsubsection{Device Operation: Create buffers} > > > > + > > > > +Buffers are used to store the actual data as well as the relevant > > > > +metadata. Scatter lists are supported, so the buffer doesn't need to > > > > +be contiguous in guest physical memory. > > > > + > > > > +\begin{itemize*} > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > > > > + resource that is backed by a buffer allocated from the driver's > > > > + memory. > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > > > > + is no longer needed. > > > > +\end{itemize*} > > > > + > > > > +\subsubsection{Device Operation: Stream parameter control} > > > > + > > > > +\begin{itemize*} > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream > > > > parameters > > > > for + input and output streams from the device. > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to > > > > the + device. > > > > +\item After setting stream parameters, the driver may issue > > > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output > > > > can be + changed implicitly by the device during the set operation. > > > > +\end{itemize*} > > > > + > > > > +\subsubsection{Device Operation: Process buffers} > > > > + > > > > +\begin{itemize*} > > > > +\item If the function and the buffer type require so, write data to > > > > +the buffer memory. > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > > > > +processing in the device. > > > > +\item The request completes asynchronously when the device has > > > > +finished with the buffer. > > > > +\end{itemize*} > > > > + > > > > +\subsubsection{Device Operation: Buffer processing control} > > > > + > > > > +\begin{itemize*} > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > > > > + return all of the already queued buffers. > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > > > > + already queued buffers from the input or the output queue. This also > > > > + includes input or output buffers that can be currently owned by the > > > > + device's processing pipeline. > > > > +\end{itemize*} > > > > + > > > > +\subsubsection{Device Operation: Asynchronous events} > > > > + > > > > +While processing buffers, the device can send asynchronous event > > > > +notifications to the driver. The behaviour depends on the exact > > > > +stream. For example, the decoder device sends a resolution change > > > > +event when it encounters new resolution metadata in the stream. > > > > + > > > > +\subsubsection{Device Operation: Request header} > > > > + > > > > +All requests and responses on the control virt queue have a fixed > > > > +header using the following layout structure and definitions: > > > > + > > > > +\begin{lstlisting} > > > > +enum virtio_video_ctrl_type { > > > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > > > + > > > > + /* request */ > > > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > > > + VIRTIO_VIDEO_T_SET_PARAMS, > > > > + VIRTIO_VIDEO_T_GET_PARAMS, > > > > + > > > > + /* response */ > > > > + VIRTIO_VIDEO_S_OK = 0x0200, > > > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > > > + > > > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > > > +}; > > > > + > > > > +struct virtio_video_ctrl_hdr { > > > > + le32 type; > > > > + le32 stream_id; > > > > + le32 len; /* Length of the structure in bytes. */ > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\begin{description} > > > > +\item[\field{type}] is the type of the driver request or the device > > > > +response. > > > > +\item[\field{stream_id}] specifies a target stream. > > > > +\item[\field{len}] is the length of data in bytes, which includes > > > > +length of the header. > > > > +\end{description} > > > > + > > > > +\subsubsection{Device Operation: controlq} > > > > + > > > > +\begin{description} > > > > + > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > > > > +supported formats. > > > > + > > > > +The driver uses \field{struct virtio_video_get_capability} to send a > > > > +query request. > > > > + > > > > +\begin{lstlisting} > > > > +enum virtio_video_buf_type { > > > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > > > +}; > > > > + > > > > +struct virtio_video_get_capability { > > > > + struct virtio_video_ctrl_hdr hdr; > > > > + enum virtio_video_buf_type buf_type; > > > > +}; > > > > +\end{lstlisting} > > > > +\begin{description} > > > > +\item[\field{buf_type}] is the buffer type that the driver asks > > > > +information about. The driver MUST set either > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > > > + > > > > +The device responds a capability by using \field{struct > > > > +virtio_video_get_capability_resp}. > > > > +\begin{lstlisting} > > > > +enum virtio_video_format { > > > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > > > + /* Raw formats */ > > > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > > > + VIRTIO_VIDEO_FORMAT_YUV420, > > > > + VIRTIO_VIDEO_FORMAT_YVU420, > > > > + > > > > + /* Compressed formats */ > > > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > > > +}; > > > > + > > > > +enum virtio_video_profile { > > > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > > > + > > > > + /* H.264 */ > > > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN, > > > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > > > > + /* VP8 */ > > > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > > > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > > > > + > > > > + /* VP9 */ > > > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN, > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > +}; > > > > + > > > > +struct virtio_video_format_range { > > > > + le32 min; > > > > + le32 max; > > > > + le32 step; > > > > + u8 paddings[4]; > > > > +}; > > > > + > > > > +struct virtio_video_format_desc { > > > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > > > > + le64 mask; > > > > + struct virtio_video_format_range width; > > > > + struct virtio_video_format_range height; > > > > + le32 num_rates; > > > > + u8 padding[4]; > > > > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > > > > +}; > > > > + > > > > +struct virtio_video_get_capability_resp { > > > > + struct virtio_video_ctrl_hdr hdr; > > > > + le32 num_descs; > > > > + /* Followed by struct virtio_video_format_desc desc[] */ > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +The format description \field{struct virtio_video_format_desc} > > > > +includes the following fields: > > > > +\begin{description} > > > > +\item[\field{format}] specifies an image format. The device MUST set > > > > one > > > > + of \field{enum virtio_video_format}. > > > > +\item[\field{profile}] specifies a profile of the compressed image > > > > format > > > > + specified in \field{format}. The driver SHOULD ignore this value if > > > > + \field{format} is a raw format. > > > > > > So how should this be used? The spec does not define any way to set > > > profile for the device. It is very important for encoder. > > > > Thank you for pointing this. > > These points are overlooked, as I didn't care about encoder usage enough. > > > > After thinking it again, I think it's not a very good idea to include > > supported profiles and levels in a struct for capability. > > This is because these values are available only for limited number of > > formats. Also, it's true that we need to have a way to set these values as > > Dmitry pointed. > > > > Yes, you are right. In fact, the approach of the v1 spec to keep controls > separately was not correct. > > > Instead, it would make more sense to have additional three types of > > controls for profiles, levels, and bitrates: > > (1) QUERY_CONTROL: Query values supported by the device > > (2) GET_CONTROL: Read a value that is set in the device > > (3) SET_CONTROL: Set a value in the device > > > > These operations are similar to V4L2 controls. > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and > > G_CTRL in V4L2, respectively. > > Also, (3) would be similar to enum virtio_video_control_type in the > > virtio-video v1 driver implementation in > > https://markmail.org/message/dwghwdqsbl3gsjxu . > > > > For QUERY_CONTROL, my idea is like this: > > > > enum virtio_video_control_type { > > VIRTIO_VIDEO_CONTROL_UNDEFINED = 0, > > > > VIRTIO_VIDEO_CONTROL_BITRATE = 0x100, > > VIRTIO_VIDEO_CONTROL_PROFILE, > > VIRTIO_VIDEO_CONTROL_LEVEL, > > }; > > > > struct virtio_video_query_control { > > struct virtio_video_ctrl_hdr hdr; > > le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */ > > le32 length; > > /* Followed by additional data. > > * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE, > > * the device must pass a codec format like H264 or VP9. > > * The requred data must be defined in the specification. > > */ > > }; > > > > struct virtio_video_query_control_resp { > > struct virtio_video_ctrl_hdr hdr; > > le32 length; > > u8 padding[4]; > > /* Followed by data corresponds to the specified control. > > * The type of data must be defined in the spec. > > * For example, if the driver queries profiles, this part should be > > * an array of supported profiles of a given format. > > */ > > }; > > > > WDYT? > > > > I think virtio_video_control_type should make sense. But I would disagree with > the need to have new QUERY_CONTROL and GET_CONTROL. > > I assume the set of supported controls is fixed for some particular format on a > given IP. So we'd propose to include controls into format descriptors, so we > don't need to QUERY_CONTROL. This way (with 'virtio_video_format_list') we can > define not-contiguos ranges, e.g. for profiles. > > struct virtio_video_format_frame { > /* As proposed in Keiichi's prev email */ > }; > > virtio_video_format_list { > le32 num_entries; > u8 padding[4]; > /* Followed by le64 entries[] */ > }; > > struct virtio_video_format_control { > le32 type; > u8 padding[4]; > struct virtio_video_format_list values; > }; > > struct virtio_video_format_desc { > le64 mask; > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > le32 planes_layout; /* See the thread [v2 0/1] */ > le32 num_frames; > le32 num_controls; > /* Followed by struct virtio_video_format_frame frames[] */ > /* Followed by struct virtio_video_format_control controls[] */ > }; In my understanding, a set of supported levels depends on a profile. H.264 spec says that levels are specified within each profile. cf. "0.5 Profiles and levels" in https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU doesn't seem to provide a way to query levels for each profile properly, though) So, if we want to have supported profiles and levels in format_desc as your idea, each supported profile should have a list of supported levels. I suppose it makes the structure of video_format_desc too complicated. I'd like to avoid this complexity caused by some specific formats. Instead, I'd like to keep virtio_video_format_desc minimal and have QUERY_CONTROL to query format-specific values like profiles and levels. This design would make it easy to extend controls when we want to support new types of formats. We will just need to define new VIRTIO_VIDEO_CONTROL_*. What do you think? Best regards, Keiichi > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the current > value locally, so there is no need to have GET_CONTROL. > > The only exception is the initial (default) control value in the device. But > with the removal of 'function' and with addition of 'caps' instead, the way to > provide defaults is gone. So I suppose for formats we'll be just using GET on > driver start to get the 'defaults'. But the thing is that for formats there > are other uses for GET, but for controls GET apparently does not make to much > sense at runtime. > > > > Also, shouldn't the profile come together with level? Would make sense for > > > encoders. > > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be > > required when querying supported levels. > > > > So probably should be enumerated together, not queried, as per the comment > above. > > > > > +\item[\field{mask}] is a bitset that represents the supported > > > > + combination of input and output format. If \textit{i}-th bit is set > > > > + in \field{mask} of \textit{j}-th \field{struct > > > > + virtio_video_format_desc} for input, the device supports encoding or > > > > + decoding from the \textit{j}-th input format to \textit{i}-th output > > > > + format. > > > > +\item[\field{width, height}] represents a range of resolutions > > > > + supported by the device. If its \field{step} is not applicable, its > > > > + \field{min} is equal to its \field{max}. > > > > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. > > > > In > > > > case of decoder, the driver SHOULD ignore this value. > > > > +\item[\field{frame_rates}] is an array of supported frame rates. > > > > +\end{description} > > > > + > > > > > > I'd guess frame rates depend on the resolution as well. This dependency > > > was > > > clear in the v1 spec, but in the v2 there is no dependency anymore. I > > > think we need to update this. > > > > That's a good point. I missed that dependency when updating the structures. > > So, let me update the structs like the following: > > > > struct virtio_video_format_frame { > > struct virtio_video_format_range width; > > struct virtio_video_format_range height; > > le32 num_rates; > > u8 padding[4]; > > /* Followed by struct virtio_video_format_range frame_rates[] */ > > }; > > > > struct virtio_video_format_desc { > > le64 mask; > > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > le32 planes_layout; /* See the thread [v2 0/1] */ > > le32 num_frames; > > u8 padding[4]; > > /* Followed by struct virtio_video_format_frame frames[] */ > > }; > > > > Yes, I do agree with this approach. > > Best regards, > Dmitry. > > > Best regards, > > Keiichi. > > > > > Best regards, > > > Dmitry. > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > > > > + within the device. > > > > + > > > > +\begin{lstlisting} > > > > +enum virtio_video_mem_type { > > > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > > > +}; > > > > + > > > > +struct virtio_video_stream_create { > > > > + struct virtio_video_ctrl_hdr hdr; > > > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > > > > + char debug_name[64]; > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\begin{description} > > > > +\item[\field{in_mem_type}] is a type of buffer management for input > > > > +buffers. The driver MUST set a value in \field{enum > > > > +virtio_video_mem_type}. > > > > +\item[\field{out_mem_type}] is a type of buffer management for output > > > > +buffers. The driver MUST set a value in \field{enum > > > > +virtio_video_mem_type}. > > > > +\item[\field{debug_name}] is a text string for a debug purpose. > > > > +\end{description} > > > > + > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > > > > + within the device. > > > > + > > > > +\begin{lstlisting} > > > > +struct virtio_video_stream_destroy { > > > > + struct virtio_video_ctrl_hdr hdr; > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > > > > + queued buffers through the pipeline. > > > > + > > > > +\begin{lstlisting} > > > > +struct virtio_video_stream_drain { > > > > + struct virtio_video_ctrl_hdr hdr; > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > > > > + within the device. > > > > + > > > > +\begin{lstlisting} > > > > +struct virtio_video_mem_entry { > > > > + le64 addr; > > > > + le32 length; > > > > + u8 padding[4]; > > > > +}; > > > > + > > > > +struct virtio_video_resource_create { > > > > + struct virtio_video_ctrl_hdr hdr; > > > > + le32 resource_id; > > > > + le32 nr_entries; > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\begin{description} > > > > +\item[\field{resource_id}] internal id of the resource. > > > > +\item[\field{nr_entries}] number of \field{struct > > > > + virtio_video_mem_entry} memory entries. > > > > +\end{description} > > > > + > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > > > > + within the device. > > > > + > > > > +\begin{lstlisting} > > > > +struct virtio_video_resource_destroy { > > > > + struct virtio_video_ctrl_hdr hdr; > > > > + le32 resource_id; > > > > + u8 padding[4]; > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\begin{description} > > > > +\item[\field{resource_id}] internal id of the resource. > > > > +\end{description} > > > > + > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > > > > +queue. > > > > + > > > > +\begin{lstlisting} > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > > > + > > > > +struct virtio_video_resource_queue { > > > > + struct virtio_video_ctrl_hdr hdr; > > > > + le32 buf_type; > > > > + le32 resource_id; > > > > + le64 timestamp; > > > > + le32 nr_data_size; > > > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\begin{description} > > > > +\item[\field{buf_type}] buf_type of the . > > > > +\item[\field{resource_id}] internal id of the resource. > > > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > > > + for synchronisation. > > > > +\item[\field{nr_data_size}] number of \field{data_size} entries. > > > > +\item[\field{data_size}] number of data bytes within a plane. > > > > +\end{description} > > > > + > > > > +\begin{lstlisting} > > > > +enum virtio_video_buffer_flag { > > > > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > > > > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > > > > + /* Encoder only */ > > > > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > > > > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > > > > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > > > > +}; > > > > + > > > > +struct virtio_video_resource_queue_resp { > > > > + struct virtio_video_ctrl_hdr hdr; > > > > + le64 timestamp; > > > > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > > > > + le32 size; /* Encoded size */ > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\begin{description} > > > > +\item[\field{timestamp}] an abstract sequence counter that can be used > > > > + for synchronisation. > > > > +\item[\field{flags}] mark specific buffers in the sequence. > > > > +\item[\field{size}] data size in the buffer (encoder only). > > > > +\end{description} > > > > + > > > > +The device sends a response to the queue request asynchronously when > > > > +it has finished processing the buffer. > > > > + > > > > +The device SHOULD mark a buffer that triggered a processing error with > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > > > > + > > > > +The device MUST mark the last buffer with the > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > > > +sequence. > > > > + > > > > +In case of encoder, to denote a particular frame type the devie MUST > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > > > > + > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > > > > + buffers back from the input or the output queue of the device. The > > > > + device SHOULD return all of the buffers from the respective queue as > > > > + soon as possible without pushing the buffers through the processing > > > > + pipeline. > > > > + > > > > +\begin{lstlisting} > > > > +struct virtio_video_queue_clear { > > > > + struct virtio_video_ctrl_hdr hdr; > > > > + le32 buf_type; > > > > + u8 padding[4]; > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\begin{description} > > > > +\item[\field{buf_type}] buffer type. > > > > +\end{description} > > > > + > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > > > > + output of a stream. > > > > + > > > > +\begin{lstlisting} > > > > +struct virtio_video_plane_format { > > > > + le32 plane_size; > > > > + le32 stride; > > > > + u8 padding[4]; > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\begin{description} > > > > +\item[\field{plane_size}] size of the plane in bytes. > > > > +\item[\field{stride}] stride used for the plane in bytes. > > > > +\end{description} > > > > + > > > > +\begin{lstlisting} > > > > +struct virtio_video_params { > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > > > > + le32 frame_width; > > > > + le32 frame_height; > > > > + le32 min_buffers; > > > > + le32 max_buffers; > > > > + le32 frame_rate; > > > > + struct virtio_video_crop { > > > > + le32 left; > > > > + le32 top; > > > > + le32 width; > > > > + le32 height; > > > > + } crop; > > > > + le32 num_planes; > > > > + struct virtio_video_plane_format > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > > > > +\end{lstlisting} > > > > + > > > > +\begin{description} > > > > +\item[\field{frame_width}] the value to get/set. > > > > +\item[\field{frame_height}] the value to get/set. > > > > +\item[\field{pixel_format}] the value to get/set. > > > > +\item[\field{min_buffers}] minimum buffers required to handle the > > > > + format (r/o). > > > > +\item[\field{max_buffers}] maximum buffers required to handle the > > > > + format (r/o). > > > > +\item[\field{frame_rate}] the value to get/set. > > > > +\item[\field{crop}] cropping (composing) rectangle. > > > > +\item[\field{num_planes}] number of planes used to store pixel data > > > > +(r/o). > > > > +\item[\field{plane_formats}] description of each plane. > > > > +\end{description} > > > > + > > > > +\begin{lstlisting} > > > > +struct virtio_video_get_params { > > > > + struct virtio_video_ctrl_hdr hdr; > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > > +}; > > > > + > > > > +struct virtio_video_get_params_resp { > > > > + struct virtio_video_ctrl_hdr hdr; > > > > + struct virtio_video_params params; > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\begin{description} > > > > +\item[\field{buf_type}] buffer type. > > > > +\item[\field{params}] parameter values. > > > > +\end{description} > > > > + > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > > > > + > > > > + > > > > +\begin{lstlisting} > > > > +struct virtio_video_set_params { > > > > + struct virtio_video_ctrl_hdr hdr; > > > > + struct virtio_video_params params; > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\begin{description} > > > > +\item[\field{params}] parameters to set. > > > > +\end{description} > > > > + > > > > +Setting stream parameters might have side effects within the device. > > > > +For example, the device MAY perform alignment of width and height, > > > > +change the number of planes it uses for the format, or do whatever > > > > +changes that are required to continue normal operation using the > > > > +updated parameters. It is up to the driver to check the parameter set > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > > > > + > > > > +\end{description} > > > > + > > > > +\subsubsection{Device Operation: eventq} > > > > + > > > > +The device can report events on the event queue. The driver initially > > > > +populates the queue with device-writeable buffers. When the device > > > > +needs to report an event, it fills a buffer and notifies the driver. > > > > +The driver consumes the report and adds a new buffer to the virtqueue. > > > > + > > > > +\begin{lstlisting} > > > > +enum virtio_video_event_type { > > > > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > > > > + /* For all functions */ > > > > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > > > > + /* For decoder only */ > > > > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > > > > +}; > > > > + > > > > +struct virtio_video_event { > > > > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > > > > + le32 stream_id; > > > > + u8 padding[4]; > > > > +}; > > > > +\end{lstlisting} > > > > + > > > > +\begin{description} > > > > +\item[\field{event_type}] type of the triggered event . > > > > +\item[\field{stream_id}] id of the source stream. > > > > +\end{description} > > > > + > > > > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > > > > +whenever it encounters new resolution data in the stream. This > > > > +includes the case of the initial device configuration after metadata > > > > +has been parsed and the case of dynamic resolution change. > >
Hi Keiichi, On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote: > Hi Dmitry, > > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi Keiichi, > > > > thanks for the updates, please see my comments below. > > > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote: > > > Hi Dmitry, > > > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp > > > <dmitry.sepp@opensynergy.com> > > > > wrote: > > > > Hi, > > > > > > > > a couple of new comments: > > > > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > > > > > The virtio video encoder device and decoder device provide > > > > > functionalities > > > > > to encode and decode video stream respectively. > > > > > Though video encoder and decoder are provided as different devices, > > > > > they > > > > > use a same protocol. > > > > > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > > > > --- > > > > > > > > > > content.tex | 1 + > > > > > virtio-video.tex | 579 > > > > > +++++++++++++++++++++++++++++++++++++++++++++++ > > > > > 2 files changed, 580 insertions(+) > > > > > create mode 100644 virtio-video.tex > > > > > > > > > > diff --git a/content.tex b/content.tex > > > > > index 556b373..9e56839 100644 > > > > > --- a/content.tex > > > > > +++ b/content.tex > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex} > > > > > > > > > > \input{virtio-fs.tex} > > > > > \input{virtio-rpmb.tex} > > > > > > > > > > +\input{virtio-video.tex} > > > > > > > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > > > > new file mode 100644 > > > > > index 0000000..30e728d > > > > > --- /dev/null > > > > > +++ b/virtio-video.tex > > > > > @@ -0,0 +1,579 @@ > > > > > +\section{Video Device}\label{sec:Device Types / Video Device} > > > > > + > > > > > +The virtio video encoder device and decoder device are virtual > > > > > devices > > > > > that +supports encoding and decoding respectively. Though the > > > > > encoder > > > > > and the decoder +are different devices, they use the same protocol. > > > > > + > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device / > > > > > Device > > > > > ID} > > > > > + > > > > > +\begin{description} > > > > > +\item[30] encoder device > > > > > +\item[31] decoder device > > > > > +\end{description} > > > > > + > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / > > > > > Virtqueues} + > > > > > +\begin{description} > > > > > +\item[0] controlq - queue for sending control commands. > > > > > +\item[1] eventq - queue for sending events happened in the device. > > > > > +\end{description} > > > > > + > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / > > > > > Feature > > > > > bits} + > > > > > +\begin{description} > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be > > > > > used > > > > > for > > > > > video + buffers. > > > > > +\end{description} > > > > > + > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video > > > > > Device > > > > > / Feature bits} + > > > > > +The device MUST offer at least one of feature bits. > > > > > + > > > > > +\subsection{Device configuration layout}\label{sec:Device Types / > > > > > Video > > > > > Device / Device configuration layout} + > > > > > +Video device configuration uses the following layout structure: > > > > > + > > > > > +\begin{lstlisting} > > > > > +struct virtio_video_config { > > > > > + le32 max_cap_len; > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\begin{description} > > > > > +\item[\field{max_cap_len}] defines the maximum length of a > > > > > descriptor > > > > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > > > > > + MUST set this value. > > > > > +\end{description} > > > > > + > > > > > +\subsection{Device Initialization}\label{sec:Device Types / Video > > > > > Device / > > > > > Device Initialization} + > > > > > +\devicenormative{\subsubsection}{Device Initialization}{Device > > > > > Types / > > > > > Video Device / Device Initialization} + > > > > > +The driver SHOULD query device capability by using the > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the > > > > > initial > > > > > +setup. > > > > > + > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device > > > > > / > > > > > Device Operation} + > > > > > +The driver allocates input and output buffers and queues the > > > > > buffers > > > > > +to the device. The device performs operations on the buffers > > > > > according > > > > > +to the function in question. > > > > > + > > > > > +\subsubsection{Device Operation: Create stream} > > > > > + > > > > > +To process buffers, the device needs to associate them with a > > > > > certain > > > > > +video stream (essentially, a context). Streams are created by > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > > > > +determined by the device. > > > > > + > > > > > +\subsubsection{Device Operation: Create buffers} > > > > > + > > > > > +Buffers are used to store the actual data as well as the relevant > > > > > +metadata. Scatter lists are supported, so the buffer doesn't need > > > > > to > > > > > +be contiguous in guest physical memory. > > > > > + > > > > > +\begin{itemize*} > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > > > > > + resource that is backed by a buffer allocated from the driver's > > > > > + memory. > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource > > > > > that > > > > > + is no longer needed. > > > > > +\end{itemize*} > > > > > + > > > > > +\subsubsection{Device Operation: Stream parameter control} > > > > > + > > > > > +\begin{itemize*} > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream > > > > > parameters > > > > > for + input and output streams from the device. > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream > > > > > parameters to > > > > > the + device. > > > > > +\item After setting stream parameters, the driver may issue > > > > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and > > > > > output > > > > > can be + changed implicitly by the device during the set operation. > > > > > +\end{itemize*} > > > > > + > > > > > +\subsubsection{Device Operation: Process buffers} > > > > > + > > > > > +\begin{itemize*} > > > > > +\item If the function and the buffer type require so, write data to > > > > > +the buffer memory. > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > > > > > +processing in the device. > > > > > +\item The request completes asynchronously when the device has > > > > > +finished with the buffer. > > > > > +\end{itemize*} > > > > > + > > > > > +\subsubsection{Device Operation: Buffer processing control} > > > > > + > > > > > +\begin{itemize*} > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process > > > > > and > > > > > + return all of the already queued buffers. > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return > > > > > back > > > > > + already queued buffers from the input or the output queue. This > > > > > also > > > > > + includes input or output buffers that can be currently owned by > > > > > the > > > > > + device's processing pipeline. > > > > > +\end{itemize*} > > > > > + > > > > > +\subsubsection{Device Operation: Asynchronous events} > > > > > + > > > > > +While processing buffers, the device can send asynchronous event > > > > > +notifications to the driver. The behaviour depends on the exact > > > > > +stream. For example, the decoder device sends a resolution change > > > > > +event when it encounters new resolution metadata in the stream. > > > > > + > > > > > +\subsubsection{Device Operation: Request header} > > > > > + > > > > > +All requests and responses on the control virt queue have a fixed > > > > > +header using the following layout structure and definitions: > > > > > + > > > > > +\begin{lstlisting} > > > > > +enum virtio_video_ctrl_type { > > > > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > > > > + > > > > > + /* request */ > > > > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > > > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > > > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > > > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > > > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > > > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > > > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > > > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > > > > + VIRTIO_VIDEO_T_SET_PARAMS, > > > > > + VIRTIO_VIDEO_T_GET_PARAMS, > > > > > + > > > > > + /* response */ > > > > > + VIRTIO_VIDEO_S_OK = 0x0200, > > > > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > > > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > > > > + > > > > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > > > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > > > > +}; > > > > > + > > > > > +struct virtio_video_ctrl_hdr { > > > > > + le32 type; > > > > > + le32 stream_id; > > > > > + le32 len; /* Length of the structure in bytes. */ > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\begin{description} > > > > > +\item[\field{type}] is the type of the driver request or the device > > > > > +response. > > > > > +\item[\field{stream_id}] specifies a target stream. > > > > > +\item[\field{len}] is the length of data in bytes, which includes > > > > > +length of the header. > > > > > +\end{description} > > > > > + > > > > > +\subsubsection{Device Operation: controlq} > > > > > + > > > > > +\begin{description} > > > > > + > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > > > > > +supported formats. > > > > > + > > > > > +The driver uses \field{struct virtio_video_get_capability} to send > > > > > a > > > > > +query request. > > > > > + > > > > > +\begin{lstlisting} > > > > > +enum virtio_video_buf_type { > > > > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > > > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > > > > +}; > > > > > + > > > > > +struct virtio_video_get_capability { > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > + enum virtio_video_buf_type buf_type; > > > > > +}; > > > > > +\end{lstlisting} > > > > > +\begin{description} > > > > > +\item[\field{buf_type}] is the buffer type that the driver asks > > > > > +information about. The driver MUST set either > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > > > > + > > > > > +The device responds a capability by using \field{struct > > > > > +virtio_video_get_capability_resp}. > > > > > +\begin{lstlisting} > > > > > +enum virtio_video_format { > > > > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > > > > + /* Raw formats */ > > > > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > > > > + VIRTIO_VIDEO_FORMAT_YUV420, > > > > > + VIRTIO_VIDEO_FORMAT_YVU420, > > > > > + > > > > > + /* Compressed formats */ > > > > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > > > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > > > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > > > > +}; > > > > > + > > > > > +enum virtio_video_profile { > > > > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > > > > + > > > > > + /* H.264 */ > > > > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > > > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN, > > > > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > > > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > > > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > > > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > > > > > + /* VP8 */ > > > > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > > > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN, > > > > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY, > > > > > + > > > > > + /* VP9 */ > > > > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN, > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > > +}; > > > > > + > > > > > +struct virtio_video_format_range { > > > > > + le32 min; > > > > > + le32 max; > > > > > + le32 step; > > > > > + u8 paddings[4]; > > > > > +}; > > > > > + > > > > > +struct virtio_video_format_desc { > > > > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > > > > > + le64 mask; > > > > > + struct virtio_video_format_range width; > > > > > + struct virtio_video_format_range height; > > > > > + le32 num_rates; > > > > > + u8 padding[4]; > > > > > + /* Followed by struct virtio_video_frame_rate frame_rates[] > > > > > */ > > > > > +}; > > > > > + > > > > > +struct virtio_video_get_capability_resp { > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > + le32 num_descs; > > > > > + /* Followed by struct virtio_video_format_desc desc[] */ > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +The format description \field{struct virtio_video_format_desc} > > > > > +includes the following fields: > > > > > +\begin{description} > > > > > +\item[\field{format}] specifies an image format. The device MUST > > > > > set > > > > > one > > > > > + of \field{enum virtio_video_format}. > > > > > +\item[\field{profile}] specifies a profile of the compressed image > > > > > format > > > > > + specified in \field{format}. The driver SHOULD ignore this value > > > > > if > > > > > + \field{format} is a raw format. > > > > > > > > So how should this be used? The spec does not define any way to set > > > > profile for the device. It is very important for encoder. > > > > > > Thank you for pointing this. > > > These points are overlooked, as I didn't care about encoder usage > > > enough. > > > > > > After thinking it again, I think it's not a very good idea to include > > > supported profiles and levels in a struct for capability. > > > This is because these values are available only for limited number of > > > formats. Also, it's true that we need to have a way to set these values > > > as > > > Dmitry pointed. > > > > Yes, you are right. In fact, the approach of the v1 spec to keep controls > > separately was not correct. > > > > > Instead, it would make more sense to have additional three types of > > > controls for profiles, levels, and bitrates: > > > (1) QUERY_CONTROL: Query values supported by the device > > > (2) GET_CONTROL: Read a value that is set in the device > > > (3) SET_CONTROL: Set a value in the device > > > > > > These operations are similar to V4L2 controls. > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and > > > G_CTRL in V4L2, respectively. > > > Also, (3) would be similar to enum virtio_video_control_type in the > > > virtio-video v1 driver implementation in > > > https://markmail.org/message/dwghwdqsbl3gsjxu . > > > > > > For QUERY_CONTROL, my idea is like this: > > > > > > enum virtio_video_control_type { > > > > > > VIRTIO_VIDEO_CONTROL_UNDEFINED = 0, > > > > > > VIRTIO_VIDEO_CONTROL_BITRATE = 0x100, > > > VIRTIO_VIDEO_CONTROL_PROFILE, > > > VIRTIO_VIDEO_CONTROL_LEVEL, > > > > > > }; > > > > > > struct virtio_video_query_control { > > > > > > struct virtio_video_ctrl_hdr hdr; > > > le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */ > > > le32 length; > > > /* Followed by additional data. > > > > > > * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE, > > > * the device must pass a codec format like H264 or VP9. > > > * The requred data must be defined in the specification. > > > */ > > > > > > }; > > > > > > struct virtio_video_query_control_resp { > > > > > > struct virtio_video_ctrl_hdr hdr; > > > le32 length; > > > u8 padding[4]; > > > /* Followed by data corresponds to the specified control. > > > > > > * The type of data must be defined in the spec. > > > * For example, if the driver queries profiles, this part should be > > > * an array of supported profiles of a given format. > > > */ > > > > > > }; > > > > > > WDYT? > > > > I think virtio_video_control_type should make sense. But I would disagree > > with the need to have new QUERY_CONTROL and GET_CONTROL. > > > > I assume the set of supported controls is fixed for some particular format > > on a given IP. So we'd propose to include controls into format > > descriptors, so we don't need to QUERY_CONTROL. This way (with > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. for > > profiles. > > > > struct virtio_video_format_frame { > > > > /* As proposed in Keiichi's prev email */ > > > > }; > > > > virtio_video_format_list { > > > > le32 num_entries; > > u8 padding[4]; > > /* Followed by le64 entries[] */ > > > > }; > > > > struct virtio_video_format_control { > > > > le32 type; > > u8 padding[4]; > > struct virtio_video_format_list values; > > > > }; > > > > struct virtio_video_format_desc { > > > > le64 mask; > > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > le32 planes_layout; /* See the thread [v2 0/1] */ > > le32 num_frames; > > le32 num_controls; > > /* Followed by struct virtio_video_format_frame frames[] */ > > /* Followed by struct virtio_video_format_control controls[] */ > > > > }; > > In my understanding, a set of supported levels depends on a profile. > H.264 spec says that levels are specified within each profile. > cf. "0.5 Profiles and levels" in > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU doesn't > seem to provide a way to query levels for each profile properly, though) > > So, if we want to have supported profiles and levels in format_desc as > your idea, each supported profile should have a list of supported > levels. > I suppose it makes the structure of video_format_desc too complicated. > I'd like to avoid this complexity caused by some specific formats. > > Instead, I'd like to keep virtio_video_format_desc minimal and have > QUERY_CONTROL to query format-specific values like profiles and > levels. > This design would make it easy to extend controls when we want to > support new types of formats. We will just need to define new > VIRTIO_VIDEO_CONTROL_*. > What do you think? Yes, I think we can keep it this way, this indeed provides more flexibility and this way we won't need to modify the 'format' parsing logic when more controls are added. Best regards, Dmitry. > > Best regards, > Keiichi > > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the > > current > > value locally, so there is no need to have GET_CONTROL. > > > > The only exception is the initial (default) control value in the device. > > But with the removal of 'function' and with addition of 'caps' instead, > > the way to provide defaults is gone. So I suppose for formats we'll be > > just using GET on driver start to get the 'defaults'. But the thing is > > that for formats there are other uses for GET, but for controls GET > > apparently does not make to much sense at runtime. > > > > > > Also, shouldn't the profile come together with level? Would make sense > > > > for > > > > encoders. > > > > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be > > > required when querying supported levels. > > > > So probably should be enumerated together, not queried, as per the comment > > above. > > > > > > > +\item[\field{mask}] is a bitset that represents the supported > > > > > + combination of input and output format. If \textit{i}-th bit is > > > > > set > > > > > + in \field{mask} of \textit{j}-th \field{struct > > > > > + virtio_video_format_desc} for input, the device supports encoding > > > > > or > > > > > + decoding from the \textit{j}-th input format to \textit{i}-th > > > > > output > > > > > + format. > > > > > +\item[\field{width, height}] represents a range of resolutions > > > > > + supported by the device. If its \field{step} is not applicable, > > > > > its > > > > > + \field{min} is equal to its \field{max}. > > > > > +\item[\field{num_rates}] is the length of an array > > > > > \field{frame_rates}. > > > > > In > > > > > case of decoder, the driver SHOULD ignore this value. > > > > > +\item[\field{frame_rates}] is an array of supported frame rates. > > > > > +\end{description} > > > > > + > > > > > > > > I'd guess frame rates depend on the resolution as well. This > > > > dependency > > > > was > > > > clear in the v1 spec, but in the v2 there is no dependency anymore. I > > > > think we need to update this. > > > > > > That's a good point. I missed that dependency when updating the > > > structures. > > > So, let me update the structs like the following: > > > > > > struct virtio_video_format_frame { > > > > > > struct virtio_video_format_range width; > > > struct virtio_video_format_range height; > > > le32 num_rates; > > > u8 padding[4]; > > > /* Followed by struct virtio_video_format_range frame_rates[] */ > > > > > > }; > > > > > > struct virtio_video_format_desc { > > > > > > le64 mask; > > > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > le32 planes_layout; /* See the thread [v2 0/1] */ > > > le32 num_frames; > > > u8 padding[4]; > > > /* Followed by struct virtio_video_format_frame frames[] */ > > > > > > }; > > > > Yes, I do agree with this approach. > > > > Best regards, > > Dmitry. > > > > > Best regards, > > > Keiichi. > > > > > > > Best regards, > > > > Dmitry. > > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > > > > > + within the device. > > > > > + > > > > > +\begin{lstlisting} > > > > > +enum virtio_video_mem_type { > > > > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > > > > +}; > > > > > + > > > > > +struct virtio_video_stream_create { > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types > > > > > */ > > > > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types > > > > > */ > > > > > + char debug_name[64]; > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\begin{description} > > > > > +\item[\field{in_mem_type}] is a type of buffer management for input > > > > > +buffers. The driver MUST set a value in \field{enum > > > > > +virtio_video_mem_type}. > > > > > +\item[\field{out_mem_type}] is a type of buffer management for > > > > > output > > > > > +buffers. The driver MUST set a value in \field{enum > > > > > +virtio_video_mem_type}. > > > > > +\item[\field{debug_name}] is a text string for a debug purpose. > > > > > +\end{description} > > > > > + > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream > > > > > (context) > > > > > + within the device. > > > > > + > > > > > +\begin{lstlisting} > > > > > +struct virtio_video_stream_destroy { > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > > > > > + queued buffers through the pipeline. > > > > > + > > > > > +\begin{lstlisting} > > > > > +struct virtio_video_stream_drain { > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > > > > > + within the device. > > > > > + > > > > > +\begin{lstlisting} > > > > > +struct virtio_video_mem_entry { > > > > > + le64 addr; > > > > > + le32 length; > > > > > + u8 padding[4]; > > > > > +}; > > > > > + > > > > > +struct virtio_video_resource_create { > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > + le32 resource_id; > > > > > + le32 nr_entries; > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\begin{description} > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > +\item[\field{nr_entries}] number of \field{struct > > > > > + virtio_video_mem_entry} memory entries. > > > > > +\end{description} > > > > > + > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource > > > > > descriptor > > > > > + within the device. > > > > > + > > > > > +\begin{lstlisting} > > > > > +struct virtio_video_resource_destroy { > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > + le32 resource_id; > > > > > + u8 padding[4]; > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\begin{description} > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > +\end{description} > > > > > + > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > > > > > +queue. > > > > > + > > > > > +\begin{lstlisting} > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > > > > + > > > > > +struct virtio_video_resource_queue { > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > + le32 buf_type; > > > > > + le32 resource_id; > > > > > + le64 timestamp; > > > > > + le32 nr_data_size; > > > > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\begin{description} > > > > > +\item[\field{buf_type}] buf_type of the . > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be > > > > > used > > > > > + for synchronisation. > > > > > +\item[\field{nr_data_size}] number of \field{data_size} entries. > > > > > +\item[\field{data_size}] number of data bytes within a plane. > > > > > +\end{description} > > > > > + > > > > > +\begin{lstlisting} > > > > > +enum virtio_video_buffer_flag { > > > > > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > > > > > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > > > > > + /* Encoder only */ > > > > > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > > > > > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > > > > > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > > > > > +}; > > > > > + > > > > > +struct virtio_video_resource_queue_resp { > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > + le64 timestamp; > > > > > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > > > > > + le32 size; /* Encoded size */ > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\begin{description} > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be > > > > > used > > > > > + for synchronisation. > > > > > +\item[\field{flags}] mark specific buffers in the sequence. > > > > > +\item[\field{size}] data size in the buffer (encoder only). > > > > > +\end{description} > > > > > + > > > > > +The device sends a response to the queue request asynchronously > > > > > when > > > > > +it has finished processing the buffer. > > > > > + > > > > > +The device SHOULD mark a buffer that triggered a processing error > > > > > with > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > > > > > + > > > > > +The device MUST mark the last buffer with the > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > > > > +sequence. > > > > > + > > > > > +In case of encoder, to denote a particular frame type the devie > > > > > MUST > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > > > > > + > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > > > > > + buffers back from the input or the output queue of the device. > > > > > The > > > > > + device SHOULD return all of the buffers from the respective queue > > > > > as > > > > > + soon as possible without pushing the buffers through the > > > > > processing > > > > > + pipeline. > > > > > + > > > > > +\begin{lstlisting} > > > > > +struct virtio_video_queue_clear { > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > + le32 buf_type; > > > > > + u8 padding[4]; > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\begin{description} > > > > > +\item[\field{buf_type}] buffer type. > > > > > +\end{description} > > > > > + > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > > > > > + output of a stream. > > > > > + > > > > > +\begin{lstlisting} > > > > > +struct virtio_video_plane_format { > > > > > + le32 plane_size; > > > > > + le32 stride; > > > > > + u8 padding[4]; > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\begin{description} > > > > > +\item[\field{plane_size}] size of the plane in bytes. > > > > > +\item[\field{stride}] stride used for the plane in bytes. > > > > > +\end{description} > > > > > + > > > > > +\begin{lstlisting} > > > > > +struct virtio_video_params { > > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > > > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > > > > > + le32 frame_width; > > > > > + le32 frame_height; > > > > > + le32 min_buffers; > > > > > + le32 max_buffers; > > > > > + le32 frame_rate; > > > > > + struct virtio_video_crop { > > > > > + le32 left; > > > > > + le32 top; > > > > > + le32 width; > > > > > + le32 height; > > > > > + } crop; > > > > > + le32 num_planes; > > > > > + struct virtio_video_plane_format > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\begin{description} > > > > > +\item[\field{frame_width}] the value to get/set. > > > > > +\item[\field{frame_height}] the value to get/set. > > > > > +\item[\field{pixel_format}] the value to get/set. > > > > > +\item[\field{min_buffers}] minimum buffers required to handle the > > > > > + format (r/o). > > > > > +\item[\field{max_buffers}] maximum buffers required to handle the > > > > > + format (r/o). > > > > > +\item[\field{frame_rate}] the value to get/set. > > > > > +\item[\field{crop}] cropping (composing) rectangle. > > > > > +\item[\field{num_planes}] number of planes used to store pixel data > > > > > +(r/o). > > > > > +\item[\field{plane_formats}] description of each plane. > > > > > +\end{description} > > > > > + > > > > > +\begin{lstlisting} > > > > > +struct virtio_video_get_params { > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > > > +}; > > > > > + > > > > > +struct virtio_video_get_params_resp { > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > + struct virtio_video_params params; > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\begin{description} > > > > > +\item[\field{buf_type}] buffer type. > > > > > +\item[\field{params}] parameter values. > > > > > +\end{description} > > > > > + > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > > > > > + > > > > > + > > > > > +\begin{lstlisting} > > > > > +struct virtio_video_set_params { > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > + struct virtio_video_params params; > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\begin{description} > > > > > +\item[\field{params}] parameters to set. > > > > > +\end{description} > > > > > + > > > > > +Setting stream parameters might have side effects within the > > > > > device. > > > > > +For example, the device MAY perform alignment of width and height, > > > > > +change the number of planes it uses for the format, or do whatever > > > > > +changes that are required to continue normal operation using the > > > > > +updated parameters. It is up to the driver to check the parameter > > > > > set > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > > > > > + > > > > > +\end{description} > > > > > + > > > > > +\subsubsection{Device Operation: eventq} > > > > > + > > > > > +The device can report events on the event queue. The driver > > > > > initially > > > > > +populates the queue with device-writeable buffers. When the device > > > > > +needs to report an event, it fills a buffer and notifies the > > > > > driver. > > > > > +The driver consumes the report and adds a new buffer to the > > > > > virtqueue. > > > > > + > > > > > +\begin{lstlisting} > > > > > +enum virtio_video_event_type { > > > > > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > > > > > + /* For all functions */ > > > > > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > > > > > + /* For decoder only */ > > > > > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > > > > > +}; > > > > > + > > > > > +struct virtio_video_event { > > > > > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > > > > > + le32 stream_id; > > > > > + u8 padding[4]; > > > > > +}; > > > > > +\end{lstlisting} > > > > > + > > > > > +\begin{description} > > > > > +\item[\field{event_type}] type of the triggered event . > > > > > +\item[\field{stream_id}] id of the source stream. > > > > > +\end{description} > > > > > + > > > > > +The device MUST send > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > > > > > +whenever it encounters new resolution data in the stream. This > > > > > +includes the case of the initial device configuration after > > > > > metadata > > > > > +has been parsed and the case of dynamic resolution change.
Hi Dmitry, On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi Keiichi, > > On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote: > > Hi Dmitry, > > > > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com> > wrote: > > > Hi Keiichi, > > > > > > thanks for the updates, please see my comments below. > > > > > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote: > > > > Hi Dmitry, > > > > > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp > > > > <dmitry.sepp@opensynergy.com> > > > > > > wrote: > > > > > Hi, > > > > > > > > > > a couple of new comments: > > > > > > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > > > > > > > The virtio video encoder device and decoder device provide > > > > > > functionalities > > > > > > to encode and decode video stream respectively. > > > > > > Though video encoder and decoder are provided as different devices, > > > > > > they > > > > > > use a same protocol. > > > > > > > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > > > > > --- > > > > > > > > > > > > content.tex | 1 + > > > > > > virtio-video.tex | 579 > > > > > > +++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > 2 files changed, 580 insertions(+) > > > > > > create mode 100644 virtio-video.tex > > > > > > > > > > > > diff --git a/content.tex b/content.tex > > > > > > index 556b373..9e56839 100644 > > > > > > --- a/content.tex > > > > > > +++ b/content.tex > > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex} > > > > > > > > > > > > \input{virtio-fs.tex} > > > > > > \input{virtio-rpmb.tex} > > > > > > > > > > > > +\input{virtio-video.tex} > > > > > > > > > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > > > > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > > > > > new file mode 100644 > > > > > > index 0000000..30e728d > > > > > > --- /dev/null > > > > > > +++ b/virtio-video.tex > > > > > > @@ -0,0 +1,579 @@ > > > > > > +\section{Video Device}\label{sec:Device Types / Video Device} > > > > > > + > > > > > > +The virtio video encoder device and decoder device are virtual > > > > > > devices > > > > > > that +supports encoding and decoding respectively. Though the > > > > > > encoder > > > > > > and the decoder +are different devices, they use the same protocol. > > > > > > + > > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device / > > > > > > Device > > > > > > ID} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[30] encoder device > > > > > > +\item[31] decoder device > > > > > > +\end{description} > > > > > > + > > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / > > > > > > Virtqueues} + > > > > > > +\begin{description} > > > > > > +\item[0] controlq - queue for sending control commands. > > > > > > +\item[1] eventq - queue for sending events happened in the device. > > > > > > +\end{description} > > > > > > + > > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / > > > > > > Feature > > > > > > bits} + > > > > > > +\begin{description} > > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be > > > > > > used > > > > > > for > > > > > > video + buffers. > > > > > > +\end{description} > > > > > > + > > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video > > > > > > Device > > > > > > / Feature bits} + > > > > > > +The device MUST offer at least one of feature bits. > > > > > > + > > > > > > +\subsection{Device configuration layout}\label{sec:Device Types / > > > > > > Video > > > > > > Device / Device configuration layout} + > > > > > > +Video device configuration uses the following layout structure: > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +struct virtio_video_config { > > > > > > + le32 max_cap_len; > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[\field{max_cap_len}] defines the maximum length of a > > > > > > descriptor > > > > > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > > > > > > + MUST set this value. > > > > > > +\end{description} > > > > > > + > > > > > > +\subsection{Device Initialization}\label{sec:Device Types / Video > > > > > > Device / > > > > > > Device Initialization} + > > > > > > +\devicenormative{\subsubsection}{Device Initialization}{Device > > > > > > Types / > > > > > > Video Device / Device Initialization} + > > > > > > +The driver SHOULD query device capability by using the > > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the > > > > > > initial > > > > > > +setup. > > > > > > + > > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device > > > > > > / > > > > > > Device Operation} + > > > > > > +The driver allocates input and output buffers and queues the > > > > > > buffers > > > > > > +to the device. The device performs operations on the buffers > > > > > > according > > > > > > +to the function in question. > > > > > > + > > > > > > +\subsubsection{Device Operation: Create stream} > > > > > > + > > > > > > +To process buffers, the device needs to associate them with a > > > > > > certain > > > > > > +video stream (essentially, a context). Streams are created by > > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > > > > > +determined by the device. > > > > > > + > > > > > > +\subsubsection{Device Operation: Create buffers} > > > > > > + > > > > > > +Buffers are used to store the actual data as well as the relevant > > > > > > +metadata. Scatter lists are supported, so the buffer doesn't need > > > > > > to > > > > > > +be contiguous in guest physical memory. > > > > > > + > > > > > > +\begin{itemize*} > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > > > > > > + resource that is backed by a buffer allocated from the driver's > > > > > > + memory. > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource > > > > > > that > > > > > > + is no longer needed. > > > > > > +\end{itemize*} > > > > > > + > > > > > > +\subsubsection{Device Operation: Stream parameter control} > > > > > > + > > > > > > +\begin{itemize*} > > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream > > > > > > parameters > > > > > > for + input and output streams from the device. > > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream > > > > > > parameters to > > > > > > the + device. > > > > > > +\item After setting stream parameters, the driver may issue > > > > > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and > > > > > > output > > > > > > can be + changed implicitly by the device during the set operation. > > > > > > +\end{itemize*} > > > > > > + > > > > > > +\subsubsection{Device Operation: Process buffers} > > > > > > + > > > > > > +\begin{itemize*} > > > > > > +\item If the function and the buffer type require so, write data to > > > > > > +the buffer memory. > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > > > > > > +processing in the device. > > > > > > +\item The request completes asynchronously when the device has > > > > > > +finished with the buffer. > > > > > > +\end{itemize*} > > > > > > + > > > > > > +\subsubsection{Device Operation: Buffer processing control} > > > > > > + > > > > > > +\begin{itemize*} > > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process > > > > > > and > > > > > > + return all of the already queued buffers. > > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return > > > > > > back > > > > > > + already queued buffers from the input or the output queue. This > > > > > > also > > > > > > + includes input or output buffers that can be currently owned by > > > > > > the > > > > > > + device's processing pipeline. > > > > > > +\end{itemize*} > > > > > > + > > > > > > +\subsubsection{Device Operation: Asynchronous events} > > > > > > + > > > > > > +While processing buffers, the device can send asynchronous event > > > > > > +notifications to the driver. The behaviour depends on the exact > > > > > > +stream. For example, the decoder device sends a resolution change > > > > > > +event when it encounters new resolution metadata in the stream. > > > > > > + > > > > > > +\subsubsection{Device Operation: Request header} > > > > > > + > > > > > > +All requests and responses on the control virt queue have a fixed > > > > > > +header using the following layout structure and definitions: > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +enum virtio_video_ctrl_type { > > > > > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > > > > > + > > > > > > + /* request */ > > > > > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > > > > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > > > > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > > > > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > > > > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > > > > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > > > > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > > > > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > > > > > + VIRTIO_VIDEO_T_SET_PARAMS, > > > > > > + VIRTIO_VIDEO_T_GET_PARAMS, > > > > > > + > > > > > > + /* response */ > > > > > > + VIRTIO_VIDEO_S_OK = 0x0200, > > > > > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > > > > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > > > > > + > > > > > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > > > > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > > > > > +}; > > > > > > + > > > > > > +struct virtio_video_ctrl_hdr { > > > > > > + le32 type; > > > > > > + le32 stream_id; > > > > > > + le32 len; /* Length of the structure in bytes. */ > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[\field{type}] is the type of the driver request or the device > > > > > > +response. > > > > > > +\item[\field{stream_id}] specifies a target stream. > > > > > > +\item[\field{len}] is the length of data in bytes, which includes > > > > > > +length of the header. > > > > > > +\end{description} > > > > > > + > > > > > > +\subsubsection{Device Operation: controlq} > > > > > > + > > > > > > +\begin{description} > > > > > > + > > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > > > > > > +supported formats. > > > > > > + > > > > > > +The driver uses \field{struct virtio_video_get_capability} to send > > > > > > a > > > > > > +query request. > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +enum virtio_video_buf_type { > > > > > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > > > > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > > > > > +}; > > > > > > + > > > > > > +struct virtio_video_get_capability { > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > + enum virtio_video_buf_type buf_type; > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > +\begin{description} > > > > > > +\item[\field{buf_type}] is the buffer type that the driver asks > > > > > > +information about. The driver MUST set either > > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > > > > > + > > > > > > +The device responds a capability by using \field{struct > > > > > > +virtio_video_get_capability_resp}. > > > > > > +\begin{lstlisting} > > > > > > +enum virtio_video_format { > > > > > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > > > > > + /* Raw formats */ > > > > > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > > > > > + VIRTIO_VIDEO_FORMAT_YUV420, > > > > > > + VIRTIO_VIDEO_FORMAT_YVU420, > > > > > > + > > > > > > + /* Compressed formats */ > > > > > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > > > > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > > > > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > > > > > +}; > > > > > > + > > > > > > +enum virtio_video_profile { > > > > > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > > > > > + > > > > > > + /* H.264 */ > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > > > > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN, > > > > > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > > > > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > > > > > > + /* VP8 */ > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = > > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN, > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = > > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY, > > > > > > + > > > > > > + /* VP9 */ > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = > > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN, > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = > > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > > > +}; > > > > > > + > > > > > > +struct virtio_video_format_range { > > > > > > + le32 min; > > > > > > + le32 max; > > > > > > + le32 step; > > > > > > + u8 paddings[4]; > > > > > > +}; > > > > > > + > > > > > > +struct virtio_video_format_desc { > > > > > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > > > > > > + le64 mask; > > > > > > + struct virtio_video_format_range width; > > > > > > + struct virtio_video_format_range height; > > > > > > + le32 num_rates; > > > > > > + u8 padding[4]; > > > > > > + /* Followed by struct virtio_video_frame_rate frame_rates[] > > > > > > */ > > > > > > +}; > > > > > > + > > > > > > +struct virtio_video_get_capability_resp { > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > + le32 num_descs; > > > > > > + /* Followed by struct virtio_video_format_desc desc[] */ > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +The format description \field{struct virtio_video_format_desc} > > > > > > +includes the following fields: > > > > > > +\begin{description} > > > > > > +\item[\field{format}] specifies an image format. The device MUST > > > > > > set > > > > > > one > > > > > > + of \field{enum virtio_video_format}. > > > > > > +\item[\field{profile}] specifies a profile of the compressed image > > > > > > format > > > > > > + specified in \field{format}. The driver SHOULD ignore this value > > > > > > if > > > > > > + \field{format} is a raw format. > > > > > > > > > > So how should this be used? The spec does not define any way to set > > > > > profile for the device. It is very important for encoder. > > > > > > > > Thank you for pointing this. > > > > These points are overlooked, as I didn't care about encoder usage > > > > enough. > > > > > > > > After thinking it again, I think it's not a very good idea to include > > > > supported profiles and levels in a struct for capability. > > > > This is because these values are available only for limited number of > > > > formats. Also, it's true that we need to have a way to set these values > > > > as > > > > Dmitry pointed. > > > > > > Yes, you are right. In fact, the approach of the v1 spec to keep controls > > > separately was not correct. > > > > > > > Instead, it would make more sense to have additional three types of > > > > controls for profiles, levels, and bitrates: > > > > (1) QUERY_CONTROL: Query values supported by the device > > > > (2) GET_CONTROL: Read a value that is set in the device > > > > (3) SET_CONTROL: Set a value in the device > > > > > > > > These operations are similar to V4L2 controls. > > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and > > > > G_CTRL in V4L2, respectively. > > > > Also, (3) would be similar to enum virtio_video_control_type in the > > > > virtio-video v1 driver implementation in > > > > https://markmail.org/message/dwghwdqsbl3gsjxu . > > > > > > > > For QUERY_CONTROL, my idea is like this: > > > > > > > > enum virtio_video_control_type { > > > > > > > > VIRTIO_VIDEO_CONTROL_UNDEFINED = 0, > > > > > > > > VIRTIO_VIDEO_CONTROL_BITRATE = 0x100, > > > > VIRTIO_VIDEO_CONTROL_PROFILE, > > > > VIRTIO_VIDEO_CONTROL_LEVEL, > > > > > > > > }; > > > > > > > > struct virtio_video_query_control { > > > > > > > > struct virtio_video_ctrl_hdr hdr; > > > > le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */ > > > > le32 length; > > > > /* Followed by additional data. > > > > > > > > * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE, > > > > * the device must pass a codec format like H264 or VP9. > > > > * The requred data must be defined in the specification. > > > > */ > > > > > > > > }; > > > > > > > > struct virtio_video_query_control_resp { > > > > > > > > struct virtio_video_ctrl_hdr hdr; > > > > le32 length; > > > > u8 padding[4]; > > > > /* Followed by data corresponds to the specified control. > > > > > > > > * The type of data must be defined in the spec. > > > > * For example, if the driver queries profiles, this part should be > > > > * an array of supported profiles of a given format. > > > > */ > > > > > > > > }; > > > > > > > > WDYT? > > > > > > I think virtio_video_control_type should make sense. But I would disagree > > > with the need to have new QUERY_CONTROL and GET_CONTROL. > > > > > > I assume the set of supported controls is fixed for some particular format > > > on a given IP. So we'd propose to include controls into format > > > descriptors, so we don't need to QUERY_CONTROL. This way (with > > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. for > > > profiles. > > > > > > struct virtio_video_format_frame { > > > > > > /* As proposed in Keiichi's prev email */ > > > > > > }; > > > > > > virtio_video_format_list { > > > > > > le32 num_entries; > > > u8 padding[4]; > > > /* Followed by le64 entries[] */ > > > > > > }; > > > > > > struct virtio_video_format_control { > > > > > > le32 type; > > > u8 padding[4]; > > > struct virtio_video_format_list values; > > > > > > }; > > > > > > struct virtio_video_format_desc { > > > > > > le64 mask; > > > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > le32 planes_layout; /* See the thread [v2 0/1] */ > > > le32 num_frames; > > > le32 num_controls; > > > /* Followed by struct virtio_video_format_frame frames[] */ > > > /* Followed by struct virtio_video_format_control controls[] */ > > > > > > }; > > > > In my understanding, a set of supported levels depends on a profile. > > H.264 spec says that levels are specified within each profile. > > cf. "0.5 Profiles and levels" in > > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU doesn't > > seem to provide a way to query levels for each profile properly, though) > > > > So, if we want to have supported profiles and levels in format_desc as > > your idea, each supported profile should have a list of supported > > levels. > > I suppose it makes the structure of video_format_desc too complicated. > > I'd like to avoid this complexity caused by some specific formats. > > > > Instead, I'd like to keep virtio_video_format_desc minimal and have > > QUERY_CONTROL to query format-specific values like profiles and > > levels. > > This design would make it easy to extend controls when we want to > > support new types of formats. We will just need to define new > > VIRTIO_VIDEO_CONTROL_*. > > What do you think? > > Yes, I think we can keep it this way, this indeed provides more flexibility and > this way we won't need to modify the 'format' parsing logic when more controls > are added. Thanks! I'm preparing the next version of the patch. I have one concern about terminology. We have two different things called "controls" now: 1) Messages passed via controlq. i.e. "ctrl" in 'enum virtio_video_ctrl_type' and 'virtio_video_ctrl_hdr'. 2) Commands for profiles, levels, and bitrates similar to V4L2 controls. e.g.., "CONTROL" in VIRTIO_VIDEO_CONTROL_* I feel calling both "controls" is somewhat confusing and I am wondering if we can't rename either of them. For example, how about renaming 2) to "command"? Then, we'll have {QUERY ,GET, SET}_COMMAND. It may sound strange, though. Do you have any idea? Best regards, Keiichi > > Best regards, > Dmitry. > > > > > Best regards, > > Keiichi > > > > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the > > > current > > > value locally, so there is no need to have GET_CONTROL. > > > > > > The only exception is the initial (default) control value in the device. > > > But with the removal of 'function' and with addition of 'caps' instead, > > > the way to provide defaults is gone. So I suppose for formats we'll be > > > just using GET on driver start to get the 'defaults'. But the thing is > > > that for formats there are other uses for GET, but for controls GET > > > apparently does not make to much sense at runtime. > > > > > > > > Also, shouldn't the profile come together with level? Would make sense > > > > > for > > > > > encoders. > > > > > > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be > > > > required when querying supported levels. > > > > > > So probably should be enumerated together, not queried, as per the comment > > > above. > > > > > > > > > +\item[\field{mask}] is a bitset that represents the supported > > > > > > + combination of input and output format. If \textit{i}-th bit is > > > > > > set > > > > > > + in \field{mask} of \textit{j}-th \field{struct > > > > > > + virtio_video_format_desc} for input, the device supports encoding > > > > > > or > > > > > > + decoding from the \textit{j}-th input format to \textit{i}-th > > > > > > output > > > > > > + format. > > > > > > +\item[\field{width, height}] represents a range of resolutions > > > > > > + supported by the device. If its \field{step} is not applicable, > > > > > > its > > > > > > + \field{min} is equal to its \field{max}. > > > > > > +\item[\field{num_rates}] is the length of an array > > > > > > \field{frame_rates}. > > > > > > In > > > > > > case of decoder, the driver SHOULD ignore this value. > > > > > > +\item[\field{frame_rates}] is an array of supported frame rates. > > > > > > +\end{description} > > > > > > + > > > > > > > > > > I'd guess frame rates depend on the resolution as well. This > > > > > dependency > > > > > was > > > > > clear in the v1 spec, but in the v2 there is no dependency anymore. I > > > > > think we need to update this. > > > > > > > > That's a good point. I missed that dependency when updating the > > > > structures. > > > > So, let me update the structs like the following: > > > > > > > > struct virtio_video_format_frame { > > > > > > > > struct virtio_video_format_range width; > > > > struct virtio_video_format_range height; > > > > le32 num_rates; > > > > u8 padding[4]; > > > > /* Followed by struct virtio_video_format_range frame_rates[] */ > > > > > > > > }; > > > > > > > > struct virtio_video_format_desc { > > > > > > > > le64 mask; > > > > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > le32 planes_layout; /* See the thread [v2 0/1] */ > > > > le32 num_frames; > > > > u8 padding[4]; > > > > /* Followed by struct virtio_video_format_frame frames[] */ > > > > > > > > }; > > > > > > Yes, I do agree with this approach. > > > > > > Best regards, > > > Dmitry. > > > > > > > Best regards, > > > > Keiichi. > > > > > > > > > Best regards, > > > > > Dmitry. > > > > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > > > > > > + within the device. > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +enum virtio_video_mem_type { > > > > > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > > > > > +}; > > > > > > + > > > > > > +struct virtio_video_stream_create { > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types > > > > > > */ > > > > > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types > > > > > > */ > > > > > > + char debug_name[64]; > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[\field{in_mem_type}] is a type of buffer management for input > > > > > > +buffers. The driver MUST set a value in \field{enum > > > > > > +virtio_video_mem_type}. > > > > > > +\item[\field{out_mem_type}] is a type of buffer management for > > > > > > output > > > > > > +buffers. The driver MUST set a value in \field{enum > > > > > > +virtio_video_mem_type}. > > > > > > +\item[\field{debug_name}] is a text string for a debug purpose. > > > > > > +\end{description} > > > > > > + > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream > > > > > > (context) > > > > > > + within the device. > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +struct virtio_video_stream_destroy { > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > > > > > > + queued buffers through the pipeline. > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +struct virtio_video_stream_drain { > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > > > > > > + within the device. > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +struct virtio_video_mem_entry { > > > > > > + le64 addr; > > > > > > + le32 length; > > > > > > + u8 padding[4]; > > > > > > +}; > > > > > > + > > > > > > +struct virtio_video_resource_create { > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > + le32 resource_id; > > > > > > + le32 nr_entries; > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > +\item[\field{nr_entries}] number of \field{struct > > > > > > + virtio_video_mem_entry} memory entries. > > > > > > +\end{description} > > > > > > + > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource > > > > > > descriptor > > > > > > + within the device. > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +struct virtio_video_resource_destroy { > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > + le32 resource_id; > > > > > > + u8 padding[4]; > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > +\end{description} > > > > > > + > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > > > > > > +queue. > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > > > > > + > > > > > > +struct virtio_video_resource_queue { > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > + le32 buf_type; > > > > > > + le32 resource_id; > > > > > > + le64 timestamp; > > > > > > + le32 nr_data_size; > > > > > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[\field{buf_type}] buf_type of the . > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be > > > > > > used > > > > > > + for synchronisation. > > > > > > +\item[\field{nr_data_size}] number of \field{data_size} entries. > > > > > > +\item[\field{data_size}] number of data bytes within a plane. > > > > > > +\end{description} > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +enum virtio_video_buffer_flag { > > > > > > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > > > > > > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > > > > > > + /* Encoder only */ > > > > > > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > > > > > > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > > > > > > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > > > > > > +}; > > > > > > + > > > > > > +struct virtio_video_resource_queue_resp { > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > + le64 timestamp; > > > > > > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > > > > > > + le32 size; /* Encoded size */ > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be > > > > > > used > > > > > > + for synchronisation. > > > > > > +\item[\field{flags}] mark specific buffers in the sequence. > > > > > > +\item[\field{size}] data size in the buffer (encoder only). > > > > > > +\end{description} > > > > > > + > > > > > > +The device sends a response to the queue request asynchronously > > > > > > when > > > > > > +it has finished processing the buffer. > > > > > > + > > > > > > +The device SHOULD mark a buffer that triggered a processing error > > > > > > with > > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > > > > > > + > > > > > > +The device MUST mark the last buffer with the > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > > > > > +sequence. > > > > > > + > > > > > > +In case of encoder, to denote a particular frame type the devie > > > > > > MUST > > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > > > > > > + > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > > > > > > + buffers back from the input or the output queue of the device. > > > > > > The > > > > > > + device SHOULD return all of the buffers from the respective queue > > > > > > as > > > > > > + soon as possible without pushing the buffers through the > > > > > > processing > > > > > > + pipeline. > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +struct virtio_video_queue_clear { > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > + le32 buf_type; > > > > > > + u8 padding[4]; > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[\field{buf_type}] buffer type. > > > > > > +\end{description} > > > > > > + > > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > > > > > > + output of a stream. > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +struct virtio_video_plane_format { > > > > > > + le32 plane_size; > > > > > > + le32 stride; > > > > > > + u8 padding[4]; > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[\field{plane_size}] size of the plane in bytes. > > > > > > +\item[\field{stride}] stride used for the plane in bytes. > > > > > > +\end{description} > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +struct virtio_video_params { > > > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > > > > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > > > > > > + le32 frame_width; > > > > > > + le32 frame_height; > > > > > > + le32 min_buffers; > > > > > > + le32 max_buffers; > > > > > > + le32 frame_rate; > > > > > > + struct virtio_video_crop { > > > > > > + le32 left; > > > > > > + le32 top; > > > > > > + le32 width; > > > > > > + le32 height; > > > > > > + } crop; > > > > > > + le32 num_planes; > > > > > > + struct virtio_video_plane_format > > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[\field{frame_width}] the value to get/set. > > > > > > +\item[\field{frame_height}] the value to get/set. > > > > > > +\item[\field{pixel_format}] the value to get/set. > > > > > > +\item[\field{min_buffers}] minimum buffers required to handle the > > > > > > + format (r/o). > > > > > > +\item[\field{max_buffers}] maximum buffers required to handle the > > > > > > + format (r/o). > > > > > > +\item[\field{frame_rate}] the value to get/set. > > > > > > +\item[\field{crop}] cropping (composing) rectangle. > > > > > > +\item[\field{num_planes}] number of planes used to store pixel data > > > > > > +(r/o). > > > > > > +\item[\field{plane_formats}] description of each plane. > > > > > > +\end{description} > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +struct virtio_video_get_params { > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > > > > +}; > > > > > > + > > > > > > +struct virtio_video_get_params_resp { > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > + struct virtio_video_params params; > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[\field{buf_type}] buffer type. > > > > > > +\item[\field{params}] parameter values. > > > > > > +\end{description} > > > > > > + > > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > > > > > > + > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +struct virtio_video_set_params { > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > + struct virtio_video_params params; > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[\field{params}] parameters to set. > > > > > > +\end{description} > > > > > > + > > > > > > +Setting stream parameters might have side effects within the > > > > > > device. > > > > > > +For example, the device MAY perform alignment of width and height, > > > > > > +change the number of planes it uses for the format, or do whatever > > > > > > +changes that are required to continue normal operation using the > > > > > > +updated parameters. It is up to the driver to check the parameter > > > > > > set > > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > > > > > > + > > > > > > +\end{description} > > > > > > + > > > > > > +\subsubsection{Device Operation: eventq} > > > > > > + > > > > > > +The device can report events on the event queue. The driver > > > > > > initially > > > > > > +populates the queue with device-writeable buffers. When the device > > > > > > +needs to report an event, it fills a buffer and notifies the > > > > > > driver. > > > > > > +The driver consumes the report and adds a new buffer to the > > > > > > virtqueue. > > > > > > + > > > > > > +\begin{lstlisting} > > > > > > +enum virtio_video_event_type { > > > > > > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > > > > > > + /* For all functions */ > > > > > > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > > > > > > + /* For decoder only */ > > > > > > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > > > > > > +}; > > > > > > + > > > > > > +struct virtio_video_event { > > > > > > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > > > > > > + le32 stream_id; > > > > > > + u8 padding[4]; > > > > > > +}; > > > > > > +\end{lstlisting} > > > > > > + > > > > > > +\begin{description} > > > > > > +\item[\field{event_type}] type of the triggered event . > > > > > > +\item[\field{stream_id}] id of the source stream. > > > > > > +\end{description} > > > > > > + > > > > > > +The device MUST send > > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > > > > > > +whenever it encounters new resolution data in the stream. This > > > > > > +includes the case of the initial device configuration after > > > > > > metadata > > > > > > +has been parsed and the case of dynamic resolution change. > >
Hi Dmitry, On Wed, Dec 18, 2019 at 10:02 PM Keiichi Watanabe <keiichiw@chromium.org> wrote: > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > The virtio video encoder device and decoder device provide functionalities to > encode and decode video stream respectively. > Though video encoder and decoder are provided as different devices, they use a > same protocol. > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > --- > content.tex | 1 + > virtio-video.tex | 579 +++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 580 insertions(+) > create mode 100644 virtio-video.tex > > diff --git a/content.tex b/content.tex > index 556b373..9e56839 100644 > --- a/content.tex > +++ b/content.tex > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device > \input{virtio-vsock.tex} > \input{virtio-fs.tex} > \input{virtio-rpmb.tex} > +\input{virtio-video.tex} > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > diff --git a/virtio-video.tex b/virtio-video.tex > new file mode 100644 > index 0000000..30e728d > --- /dev/null > +++ b/virtio-video.tex > @@ -0,0 +1,579 @@ > +\section{Video Device}\label{sec:Device Types / Video Device} > + > +The virtio video encoder device and decoder device are virtual devices that > +supports encoding and decoding respectively. Though the encoder and the decoder > +are different devices, they use the same protocol. > + > +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID} > + > +\begin{description} > +\item[30] encoder device > +\item[31] decoder device > +\end{description} > + > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues} > + > +\begin{description} > +\item[0] controlq - queue for sending control commands. > +\item[1] eventq - queue for sending events happened in the device. > +\end{description} > + > +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature bits} > + > +\begin{description} > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for video > + buffers. > +\end{description} > + > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} > + > +The device MUST offer at least one of feature bits. > + > +\subsection{Device configuration layout}\label{sec:Device Types / Video Device / Device configuration layout} > + > +Video device configuration uses the following layout structure: > + > +\begin{lstlisting} > +struct virtio_video_config { > + le32 max_cap_len; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{max_cap_len}] defines the maximum length of a descriptor > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > + MUST set this value. > +\end{description} > + > +\subsection{Device Initialization}\label{sec:Device Types / Video Device / Device Initialization} > + > +\devicenormative{\subsubsection}{Device Initialization}{Device Types / Video Device / Device Initialization} > + > +The driver SHOULD query device capability by using the > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial > +setup. > + > +\subsection{Device Operation}\label{sec:Device Types / Video Device / Device Operation} > + > +The driver allocates input and output buffers and queues the buffers > +to the device. The device performs operations on the buffers according > +to the function in question. > + > +\subsubsection{Device Operation: Create stream} > + > +To process buffers, the device needs to associate them with a certain > +video stream (essentially, a context). Streams are created by > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > +determined by the device. > + > +\subsubsection{Device Operation: Create buffers} > + > +Buffers are used to store the actual data as well as the relevant > +metadata. Scatter lists are supported, so the buffer doesn't need to > +be contiguous in guest physical memory. > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > + resource that is backed by a buffer allocated from the driver's > + memory. > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that > + is no longer needed. > +\end{itemize*} > + > +\subsubsection{Device Operation: Stream parameter control} > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters for > + input and output streams from the device. > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the > + device. > +\item After setting stream parameters, the driver may issue > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can be > + changed implicitly by the device during the set operation. > +\end{itemize*} > + > +\subsubsection{Device Operation: Process buffers} > + > +\begin{itemize*} > +\item If the function and the buffer type require so, write data to > +the buffer memory. > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > +processing in the device. > +\item The request completes asynchronously when the device has > +finished with the buffer. > +\end{itemize*} > + > +\subsubsection{Device Operation: Buffer processing control} > + > +\begin{itemize*} > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and > + return all of the already queued buffers. > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back > + already queued buffers from the input or the output queue. This also > + includes input or output buffers that can be currently owned by the > + device's processing pipeline. > +\end{itemize*} > + > +\subsubsection{Device Operation: Asynchronous events} > + > +While processing buffers, the device can send asynchronous event > +notifications to the driver. The behaviour depends on the exact > +stream. For example, the decoder device sends a resolution change > +event when it encounters new resolution metadata in the stream. > + > +\subsubsection{Device Operation: Request header} > + > +All requests and responses on the control virt queue have a fixed > +header using the following layout structure and definitions: > + > +\begin{lstlisting} > +enum virtio_video_ctrl_type { > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > + > + /* request */ > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > + VIRTIO_VIDEO_T_STREAM_CREATE, > + VIRTIO_VIDEO_T_STREAM_DESTROY, > + VIRTIO_VIDEO_T_STREAM_DRAIN, > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > + VIRTIO_VIDEO_T_SET_PARAMS, > + VIRTIO_VIDEO_T_GET_PARAMS, > + > + /* response */ > + VIRTIO_VIDEO_S_OK = 0x0200, > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > + > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > +}; I have a question about naming of these constants. What's the meaning of "T" and "S" for requests and responses, respectively? Though I copied these values from your first version, I'm wondering if we can't call VIRTIO_VIDEO_REQ_* and VIRTIO_VIDEO_RESP_* instead. Best regards, Keiichi > + > +struct virtio_video_ctrl_hdr { > + le32 type; > + le32 stream_id; > + le32 len; /* Length of the structure in bytes. */ > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{type}] is the type of the driver request or the device > +response. > +\item[\field{stream_id}] specifies a target stream. > +\item[\field{len}] is the length of data in bytes, which includes > +length of the header. > +\end{description} > + > +\subsubsection{Device Operation: controlq} > + > +\begin{description} > + > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > +supported formats. > + > +The driver uses \field{struct virtio_video_get_capability} to send a > +query request. > + > +\begin{lstlisting} > +enum virtio_video_buf_type { > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > +}; > + > +struct virtio_video_get_capability { > + struct virtio_video_ctrl_hdr hdr; > + enum virtio_video_buf_type buf_type; > +}; > +\end{lstlisting} > +\begin{description} > +\item[\field{buf_type}] is the buffer type that the driver asks > +information about. The driver MUST set either > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. > +\end{description} > + > +The device responds a capability by using \field{struct > +virtio_video_get_capability_resp}. > +\begin{lstlisting} > +enum virtio_video_format { > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > + /* Raw formats */ > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > + VIRTIO_VIDEO_FORMAT_YUV420, > + VIRTIO_VIDEO_FORMAT_YVU420, > + > + /* Compressed formats */ > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > +}; > + > +enum virtio_video_profile { > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > + > + /* H.264 */ > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_BASELINE, > + VIRTIO_VIDEO_PROFILE_H264_MAIN, > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > + VIRTIO_VIDEO_PROFILE_H264_MAX = VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > + > + /* VP8 */ > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, > + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, > + > + /* VP9 */ > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > + VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > +}; > + > +struct virtio_video_format_range { > + le32 min; > + le32 max; > + le32 step; > + u8 paddings[4]; > +}; > + > +struct virtio_video_format_desc { > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > + le64 mask; > + struct virtio_video_format_range width; > + struct virtio_video_format_range height; > + le32 num_rates; > + u8 padding[4]; > + /* Followed by struct virtio_video_frame_rate frame_rates[] */ > +}; > + > +struct virtio_video_get_capability_resp { > + struct virtio_video_ctrl_hdr hdr; > + le32 num_descs; > + /* Followed by struct virtio_video_format_desc desc[] */ > +}; > +\end{lstlisting} > + > +The format description \field{struct virtio_video_format_desc} > +includes the following fields: > +\begin{description} > +\item[\field{format}] specifies an image format. The device MUST set one > + of \field{enum virtio_video_format}. > +\item[\field{profile}] specifies a profile of the compressed image format > + specified in \field{format}. The driver SHOULD ignore this value if > + \field{format} is a raw format. > +\item[\field{mask}] is a bitset that represents the supported > + combination of input and output format. If \textit{i}-th bit is set > + in \field{mask} of \textit{j}-th \field{struct > + virtio_video_format_desc} for input, the device supports encoding or > + decoding from the \textit{j}-th input format to \textit{i}-th output > + format. > +\item[\field{width, height}] represents a range of resolutions > + supported by the device. If its \field{step} is not applicable, its > + \field{min} is equal to its \field{max}. > +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In case of decoder, the driver SHOULD ignore this value. > +\item[\field{frame_rates}] is an array of supported frame rates. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > + within the device. > + > +\begin{lstlisting} > +enum virtio_video_mem_type { > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > +}; > + > +struct virtio_video_stream_create { > + struct virtio_video_ctrl_hdr hdr; > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ > + char debug_name[64]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{in_mem_type}] is a type of buffer management for input > +buffers. The driver MUST set a value in \field{enum > +virtio_video_mem_type}. > +\item[\field{out_mem_type}] is a type of buffer management for output > +buffers. The driver MUST set a value in \field{enum > +virtio_video_mem_type}. > +\item[\field{debug_name}] is a text string for a debug purpose. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_stream_destroy { > + struct virtio_video_ctrl_hdr hdr; > +}; > +\end{lstlisting} > + > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > + queued buffers through the pipeline. > + > +\begin{lstlisting} > +struct virtio_video_stream_drain { > + struct virtio_video_ctrl_hdr hdr; > +}; > +\end{lstlisting} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_mem_entry { > + le64 addr; > + le32 length; > + u8 padding[4]; > +}; > + > +struct virtio_video_resource_create { > + struct virtio_video_ctrl_hdr hdr; > + le32 resource_id; > + le32 nr_entries; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{resource_id}] internal id of the resource. > +\item[\field{nr_entries}] number of \field{struct > + virtio_video_mem_entry} memory entries. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor > + within the device. > + > +\begin{lstlisting} > +struct virtio_video_resource_destroy { > + struct virtio_video_ctrl_hdr hdr; > + le32 resource_id; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{resource_id}] internal id of the resource. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > +queue. > + > +\begin{lstlisting} > +#define VIRTIO_VIDEO_MAX_PLANES 8 > + > +struct virtio_video_resource_queue { > + struct virtio_video_ctrl_hdr hdr; > + le32 buf_type; > + le32 resource_id; > + le64 timestamp; > + le32 nr_data_size; > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{buf_type}] buf_type of the . > +\item[\field{resource_id}] internal id of the resource. > +\item[\field{timestamp}] an abstract sequence counter that can be used > + for synchronisation. > +\item[\field{nr_data_size}] number of \field{data_size} entries. > +\item[\field{data_size}] number of data bytes within a plane. > +\end{description} > + > +\begin{lstlisting} > +enum virtio_video_buffer_flag { > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > + /* Encoder only */ > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > +}; > + > +struct virtio_video_resource_queue_resp { > + struct virtio_video_ctrl_hdr hdr; > + le64 timestamp; > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > + le32 size; /* Encoded size */ > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{timestamp}] an abstract sequence counter that can be used > + for synchronisation. > +\item[\field{flags}] mark specific buffers in the sequence. > +\item[\field{size}] data size in the buffer (encoder only). > +\end{description} > + > +The device sends a response to the queue request asynchronously when > +it has finished processing the buffer. > + > +The device SHOULD mark a buffer that triggered a processing error with > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > + > +The device MUST mark the last buffer with the > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > +sequence. > + > +In case of encoder, to denote a particular frame type the devie MUST > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > + > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > + buffers back from the input or the output queue of the device. The > + device SHOULD return all of the buffers from the respective queue as > + soon as possible without pushing the buffers through the processing > + pipeline. > + > +\begin{lstlisting} > +struct virtio_video_queue_clear { > + struct virtio_video_ctrl_hdr hdr; > + le32 buf_type; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{buf_type}] buffer type. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > + output of a stream. > + > +\begin{lstlisting} > +struct virtio_video_plane_format { > + le32 plane_size; > + le32 stride; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{plane_size}] size of the plane in bytes. > +\item[\field{stride}] stride used for the plane in bytes. > +\end{description} > + > +\begin{lstlisting} > +struct virtio_video_params { > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > + le32 frame_width; > + le32 frame_height; > + le32 min_buffers; > + le32 max_buffers; > + le32 frame_rate; > + struct virtio_video_crop { > + le32 left; > + le32 top; > + le32 width; > + le32 height; > + } crop; > + le32 num_planes; > + struct virtio_video_plane_format plane_formats[VIRTIO_VIDEO_MAX_PLANES]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{frame_width}] the value to get/set. > +\item[\field{frame_height}] the value to get/set. > +\item[\field{pixel_format}] the value to get/set. > +\item[\field{min_buffers}] minimum buffers required to handle the > + format (r/o). > +\item[\field{max_buffers}] maximum buffers required to handle the > + format (r/o). > +\item[\field{frame_rate}] the value to get/set. > +\item[\field{crop}] cropping (composing) rectangle. > +\item[\field{num_planes}] number of planes used to store pixel data > +(r/o). > +\item[\field{plane_formats}] description of each plane. > +\end{description} > + > +\begin{lstlisting} > +struct virtio_video_get_params { > + struct virtio_video_ctrl_hdr hdr; > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > +}; > + > +struct virtio_video_get_params_resp { > + struct virtio_video_ctrl_hdr hdr; > + struct virtio_video_params params; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{buf_type}] buffer type. > +\item[\field{params}] parameter values. > +\end{description} > + > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > + > + > +\begin{lstlisting} > +struct virtio_video_set_params { > + struct virtio_video_ctrl_hdr hdr; > + struct virtio_video_params params; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{params}] parameters to set. > +\end{description} > + > +Setting stream parameters might have side effects within the device. > +For example, the device MAY perform alignment of width and height, > +change the number of planes it uses for the format, or do whatever > +changes that are required to continue normal operation using the > +updated parameters. It is up to the driver to check the parameter set > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > + > +\end{description} > + > +\subsubsection{Device Operation: eventq} > + > +The device can report events on the event queue. The driver initially > +populates the queue with device-writeable buffers. When the device > +needs to report an event, it fills a buffer and notifies the driver. > +The driver consumes the report and adds a new buffer to the virtqueue. > + > +\begin{lstlisting} > +enum virtio_video_event_type { > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > + /* For all functions */ > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > + /* For decoder only */ > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > +}; > + > +struct virtio_video_event { > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > + le32 stream_id; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +\begin{description} > +\item[\field{event_type}] type of the triggered event . > +\item[\field{stream_id}] id of the source stream. > +\end{description} > + > +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > +whenever it encounters new resolution data in the stream. This > +includes the case of the initial device configuration after metadata > +has been parsed and the case of dynamic resolution change. > -- > 2.24.1.735.g03f4e72817-goog >
On Wed, Jan 8, 2020 at 9:15 PM Keiichi Watanabe <keiichiw@chromium.org> wrote: > > Hi Dmitry, > > On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > > > Hi Keiichi, > > > > On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote: > > > Hi Dmitry, > > > > > > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com> > > wrote: > > > > Hi Keiichi, > > > > > > > > thanks for the updates, please see my comments below. > > > > > > > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote: > > > > > Hi Dmitry, > > > > > > > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp > > > > > <dmitry.sepp@opensynergy.com> > > > > > > > > wrote: > > > > > > Hi, > > > > > > > > > > > > a couple of new comments: > > > > > > > > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > > > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > > > > > > > > > The virtio video encoder device and decoder device provide > > > > > > > functionalities > > > > > > > to encode and decode video stream respectively. > > > > > > > Though video encoder and decoder are provided as different devices, > > > > > > > they > > > > > > > use a same protocol. > > > > > > > > > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > > > > > > --- > > > > > > > > > > > > > > content.tex | 1 + > > > > > > > virtio-video.tex | 579 > > > > > > > +++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > > 2 files changed, 580 insertions(+) > > > > > > > create mode 100644 virtio-video.tex > > > > > > > > > > > > > > diff --git a/content.tex b/content.tex > > > > > > > index 556b373..9e56839 100644 > > > > > > > --- a/content.tex > > > > > > > +++ b/content.tex > > > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex} > > > > > > > > > > > > > > \input{virtio-fs.tex} > > > > > > > \input{virtio-rpmb.tex} > > > > > > > > > > > > > > +\input{virtio-video.tex} > > > > > > > > > > > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > > > > > > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > > > > > > new file mode 100644 > > > > > > > index 0000000..30e728d > > > > > > > --- /dev/null > > > > > > > +++ b/virtio-video.tex > > > > > > > @@ -0,0 +1,579 @@ > > > > > > > +\section{Video Device}\label{sec:Device Types / Video Device} > > > > > > > + > > > > > > > +The virtio video encoder device and decoder device are virtual > > > > > > > devices > > > > > > > that +supports encoding and decoding respectively. Though the > > > > > > > encoder > > > > > > > and the decoder +are different devices, they use the same protocol. > > > > > > > + > > > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device / > > > > > > > Device > > > > > > > ID} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[30] encoder device > > > > > > > +\item[31] decoder device > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / > > > > > > > Virtqueues} + > > > > > > > +\begin{description} > > > > > > > +\item[0] controlq - queue for sending control commands. > > > > > > > +\item[1] eventq - queue for sending events happened in the device. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / > > > > > > > Feature > > > > > > > bits} + > > > > > > > +\begin{description} > > > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be > > > > > > > used > > > > > > > for > > > > > > > video + buffers. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video > > > > > > > Device > > > > > > > / Feature bits} + > > > > > > > +The device MUST offer at least one of feature bits. > > > > > > > + > > > > > > > +\subsection{Device configuration layout}\label{sec:Device Types / > > > > > > > Video > > > > > > > Device / Device configuration layout} + > > > > > > > +Video device configuration uses the following layout structure: > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +struct virtio_video_config { > > > > > > > + le32 max_cap_len; > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[\field{max_cap_len}] defines the maximum length of a > > > > > > > descriptor > > > > > > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > > > > > > > + MUST set this value. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\subsection{Device Initialization}\label{sec:Device Types / Video > > > > > > > Device / > > > > > > > Device Initialization} + > > > > > > > +\devicenormative{\subsubsection}{Device Initialization}{Device > > > > > > > Types / > > > > > > > Video Device / Device Initialization} + > > > > > > > +The driver SHOULD query device capability by using the > > > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the > > > > > > > initial > > > > > > > +setup. > > > > > > > + > > > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device > > > > > > > / > > > > > > > Device Operation} + > > > > > > > +The driver allocates input and output buffers and queues the > > > > > > > buffers > > > > > > > +to the device. The device performs operations on the buffers > > > > > > > according > > > > > > > +to the function in question. > > > > > > > + > > > > > > > +\subsubsection{Device Operation: Create stream} > > > > > > > + > > > > > > > +To process buffers, the device needs to associate them with a > > > > > > > certain > > > > > > > +video stream (essentially, a context). Streams are created by > > > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > > > > > > +determined by the device. > > > > > > > + > > > > > > > +\subsubsection{Device Operation: Create buffers} > > > > > > > + > > > > > > > +Buffers are used to store the actual data as well as the relevant > > > > > > > +metadata. Scatter lists are supported, so the buffer doesn't need > > > > > > > to > > > > > > > +be contiguous in guest physical memory. > > > > > > > + > > > > > > > +\begin{itemize*} > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > > > > > > > + resource that is backed by a buffer allocated from the driver's > > > > > > > + memory. > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource > > > > > > > that > > > > > > > + is no longer needed. > > > > > > > +\end{itemize*} > > > > > > > + > > > > > > > +\subsubsection{Device Operation: Stream parameter control} > > > > > > > + > > > > > > > +\begin{itemize*} > > > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream > > > > > > > parameters > > > > > > > for + input and output streams from the device. > > > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream > > > > > > > parameters to > > > > > > > the + device. > > > > > > > +\item After setting stream parameters, the driver may issue > > > > > > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and > > > > > > > output > > > > > > > can be + changed implicitly by the device during the set operation. > > > > > > > +\end{itemize*} > > > > > > > + > > > > > > > +\subsubsection{Device Operation: Process buffers} > > > > > > > + > > > > > > > +\begin{itemize*} > > > > > > > +\item If the function and the buffer type require so, write data to > > > > > > > +the buffer memory. > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > > > > > > > +processing in the device. > > > > > > > +\item The request completes asynchronously when the device has > > > > > > > +finished with the buffer. > > > > > > > +\end{itemize*} > > > > > > > + > > > > > > > +\subsubsection{Device Operation: Buffer processing control} > > > > > > > + > > > > > > > +\begin{itemize*} > > > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process > > > > > > > and > > > > > > > + return all of the already queued buffers. > > > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return > > > > > > > back > > > > > > > + already queued buffers from the input or the output queue. This > > > > > > > also > > > > > > > + includes input or output buffers that can be currently owned by > > > > > > > the > > > > > > > + device's processing pipeline. > > > > > > > +\end{itemize*} > > > > > > > + > > > > > > > +\subsubsection{Device Operation: Asynchronous events} > > > > > > > + > > > > > > > +While processing buffers, the device can send asynchronous event > > > > > > > +notifications to the driver. The behaviour depends on the exact > > > > > > > +stream. For example, the decoder device sends a resolution change > > > > > > > +event when it encounters new resolution metadata in the stream. > > > > > > > + > > > > > > > +\subsubsection{Device Operation: Request header} > > > > > > > + > > > > > > > +All requests and responses on the control virt queue have a fixed > > > > > > > +header using the following layout structure and definitions: > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +enum virtio_video_ctrl_type { > > > > > > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > > > > > > + > > > > > > > + /* request */ > > > > > > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > > > > > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > > > > > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > > > > > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > > > > > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > > > > > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > > > > > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > > > > > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > > > > > > + VIRTIO_VIDEO_T_SET_PARAMS, > > > > > > > + VIRTIO_VIDEO_T_GET_PARAMS, > > > > > > > + > > > > > > > + /* response */ > > > > > > > + VIRTIO_VIDEO_S_OK = 0x0200, > > > > > > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > > > > > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > > > > > > + > > > > > > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > > > > > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > > > > > > +}; > > > > > > > + > > > > > > > +struct virtio_video_ctrl_hdr { > > > > > > > + le32 type; > > > > > > > + le32 stream_id; > > > > > > > + le32 len; /* Length of the structure in bytes. */ > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[\field{type}] is the type of the driver request or the device > > > > > > > +response. > > > > > > > +\item[\field{stream_id}] specifies a target stream. > > > > > > > +\item[\field{len}] is the length of data in bytes, which includes > > > > > > > +length of the header. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\subsubsection{Device Operation: controlq} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > + > > > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > > > > > > > +supported formats. > > > > > > > + > > > > > > > +The driver uses \field{struct virtio_video_get_capability} to send > > > > > > > a > > > > > > > +query request. > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +enum virtio_video_buf_type { > > > > > > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > > > > > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > > > > > > +}; > > > > > > > + > > > > > > > +struct virtio_video_get_capability { > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > + enum virtio_video_buf_type buf_type; > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > +\begin{description} > > > > > > > +\item[\field{buf_type}] is the buffer type that the driver asks > > > > > > > +information about. The driver MUST set either > > > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > > > > > > + > > > > > > > +The device responds a capability by using \field{struct > > > > > > > +virtio_video_get_capability_resp}. > > > > > > > +\begin{lstlisting} > > > > > > > +enum virtio_video_format { > > > > > > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > > > > > > + /* Raw formats */ > > > > > > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > > > > > > + VIRTIO_VIDEO_FORMAT_YUV420, > > > > > > > + VIRTIO_VIDEO_FORMAT_YVU420, > > > > > > > + > > > > > > > + /* Compressed formats */ > > > > > > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > > > > > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > > > > > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > > > > > > +}; > > > > > > > + > > > > > > > +enum virtio_video_profile { > > > > > > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > > > > > > + > > > > > > > + /* H.264 */ > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN, > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > > > > > > > + /* VP8 */ > > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN, > > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY, > > > > > > > + > > > > > > > + /* VP9 */ > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN, > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > > > > +}; > > > > > > > + > > > > > > > +struct virtio_video_format_range { > > > > > > > + le32 min; > > > > > > > + le32 max; > > > > > > > + le32 step; > > > > > > > + u8 paddings[4]; > > > > > > > +}; > > > > > > > + > > > > > > > +struct virtio_video_format_desc { > > > > > > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > > > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > > > > > > > + le64 mask; > > > > > > > + struct virtio_video_format_range width; > > > > > > > + struct virtio_video_format_range height; > > > > > > > + le32 num_rates; > > > > > > > + u8 padding[4]; > > > > > > > + /* Followed by struct virtio_video_frame_rate frame_rates[] > > > > > > > */ > > > > > > > +}; > > > > > > > + > > > > > > > +struct virtio_video_get_capability_resp { > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > + le32 num_descs; > > > > > > > + /* Followed by struct virtio_video_format_desc desc[] */ > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +The format description \field{struct virtio_video_format_desc} > > > > > > > +includes the following fields: > > > > > > > +\begin{description} > > > > > > > +\item[\field{format}] specifies an image format. The device MUST > > > > > > > set > > > > > > > one > > > > > > > + of \field{enum virtio_video_format}. > > > > > > > +\item[\field{profile}] specifies a profile of the compressed image > > > > > > > format > > > > > > > + specified in \field{format}. The driver SHOULD ignore this value > > > > > > > if > > > > > > > + \field{format} is a raw format. > > > > > > > > > > > > So how should this be used? The spec does not define any way to set > > > > > > profile for the device. It is very important for encoder. > > > > > > > > > > Thank you for pointing this. > > > > > These points are overlooked, as I didn't care about encoder usage > > > > > enough. > > > > > > > > > > After thinking it again, I think it's not a very good idea to include > > > > > supported profiles and levels in a struct for capability. > > > > > This is because these values are available only for limited number of > > > > > formats. Also, it's true that we need to have a way to set these values > > > > > as > > > > > Dmitry pointed. > > > > > > > > Yes, you are right. In fact, the approach of the v1 spec to keep controls > > > > separately was not correct. > > > > > > > > > Instead, it would make more sense to have additional three types of > > > > > controls for profiles, levels, and bitrates: > > > > > (1) QUERY_CONTROL: Query values supported by the device > > > > > (2) GET_CONTROL: Read a value that is set in the device > > > > > (3) SET_CONTROL: Set a value in the device > > > > > > > > > > These operations are similar to V4L2 controls. > > > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and > > > > > G_CTRL in V4L2, respectively. > > > > > Also, (3) would be similar to enum virtio_video_control_type in the > > > > > virtio-video v1 driver implementation in > > > > > https://markmail.org/message/dwghwdqsbl3gsjxu . > > > > > > > > > > For QUERY_CONTROL, my idea is like this: > > > > > > > > > > enum virtio_video_control_type { > > > > > > > > > > VIRTIO_VIDEO_CONTROL_UNDEFINED = 0, > > > > > > > > > > VIRTIO_VIDEO_CONTROL_BITRATE = 0x100, > > > > > VIRTIO_VIDEO_CONTROL_PROFILE, > > > > > VIRTIO_VIDEO_CONTROL_LEVEL, > > > > > > > > > > }; > > > > > > > > > > struct virtio_video_query_control { > > > > > > > > > > struct virtio_video_ctrl_hdr hdr; > > > > > le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */ > > > > > le32 length; > > > > > /* Followed by additional data. > > > > > > > > > > * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE, > > > > > * the device must pass a codec format like H264 or VP9. > > > > > * The requred data must be defined in the specification. > > > > > */ > > > > > > > > > > }; > > > > > > > > > > struct virtio_video_query_control_resp { > > > > > > > > > > struct virtio_video_ctrl_hdr hdr; > > > > > le32 length; > > > > > u8 padding[4]; > > > > > /* Followed by data corresponds to the specified control. > > > > > > > > > > * The type of data must be defined in the spec. > > > > > * For example, if the driver queries profiles, this part should be > > > > > * an array of supported profiles of a given format. > > > > > */ > > > > > > > > > > }; > > > > > > > > > > WDYT? > > > > > > > > I think virtio_video_control_type should make sense. But I would disagree > > > > with the need to have new QUERY_CONTROL and GET_CONTROL. > > > > > > > > I assume the set of supported controls is fixed for some particular format > > > > on a given IP. So we'd propose to include controls into format > > > > descriptors, so we don't need to QUERY_CONTROL. This way (with > > > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. for > > > > profiles. > > > > > > > > struct virtio_video_format_frame { > > > > > > > > /* As proposed in Keiichi's prev email */ > > > > > > > > }; > > > > > > > > virtio_video_format_list { > > > > > > > > le32 num_entries; > > > > u8 padding[4]; > > > > /* Followed by le64 entries[] */ > > > > > > > > }; > > > > > > > > struct virtio_video_format_control { > > > > > > > > le32 type; > > > > u8 padding[4]; > > > > struct virtio_video_format_list values; > > > > > > > > }; > > > > > > > > struct virtio_video_format_desc { > > > > > > > > le64 mask; > > > > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > le32 planes_layout; /* See the thread [v2 0/1] */ > > > > le32 num_frames; > > > > le32 num_controls; > > > > /* Followed by struct virtio_video_format_frame frames[] */ > > > > /* Followed by struct virtio_video_format_control controls[] */ > > > > > > > > }; > > > > > > In my understanding, a set of supported levels depends on a profile. > > > H.264 spec says that levels are specified within each profile. > > > cf. "0.5 Profiles and levels" in > > > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU doesn't > > > seem to provide a way to query levels for each profile properly, though) > > > > > > So, if we want to have supported profiles and levels in format_desc as > > > your idea, each supported profile should have a list of supported > > > levels. > > > I suppose it makes the structure of video_format_desc too complicated. > > > I'd like to avoid this complexity caused by some specific formats. > > > > > > Instead, I'd like to keep virtio_video_format_desc minimal and have > > > QUERY_CONTROL to query format-specific values like profiles and > > > levels. > > > This design would make it easy to extend controls when we want to > > > support new types of formats. We will just need to define new > > > VIRTIO_VIDEO_CONTROL_*. > > > What do you think? > > > > Yes, I think we can keep it this way, this indeed provides more flexibility and > > this way we won't need to modify the 'format' parsing logic when more controls > > are added. > > Thanks! I'm preparing the next version of the patch. > > I have one concern about terminology. > We have two different things called "controls" now: > 1) Messages passed via controlq. i.e. "ctrl" in 'enum > virtio_video_ctrl_type' and 'virtio_video_ctrl_hdr'. > 2) Commands for profiles, levels, and bitrates similar to V4L2 > controls. e.g.., "CONTROL" in VIRTIO_VIDEO_CONTROL_* > > I feel calling both "controls" is somewhat confusing and I am > wondering if we can't rename either of them. > > For example, how about renaming 2) to "command"? Then, we'll have > {QUERY ,GET, SET}_COMMAND. > It may sound strange, though. > Do you have any idea? "Command" sound like executing an action, but in this context we're thinking more about parameters or, exactly, controls. The term "control" is not limited to V4L2 and actually used in other subsystems as well, for example the Android Camera HAL. How about renaming 1) to "commandq" and then having the other symbols use the abbreviated "cmd"? Best regards, Tomasz > > Best regards, > Keiichi > > > > > Best regards, > > Dmitry. > > > > > > > > Best regards, > > > Keiichi > > > > > > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the > > > > current > > > > value locally, so there is no need to have GET_CONTROL. > > > > > > > > The only exception is the initial (default) control value in the device. > > > > But with the removal of 'function' and with addition of 'caps' instead, > > > > the way to provide defaults is gone. So I suppose for formats we'll be > > > > just using GET on driver start to get the 'defaults'. But the thing is > > > > that for formats there are other uses for GET, but for controls GET > > > > apparently does not make to much sense at runtime. > > > > > > > > > > Also, shouldn't the profile come together with level? Would make sense > > > > > > for > > > > > > encoders. > > > > > > > > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be > > > > > required when querying supported levels. > > > > > > > > So probably should be enumerated together, not queried, as per the comment > > > > above. > > > > > > > > > > > +\item[\field{mask}] is a bitset that represents the supported > > > > > > > + combination of input and output format. If \textit{i}-th bit is > > > > > > > set > > > > > > > + in \field{mask} of \textit{j}-th \field{struct > > > > > > > + virtio_video_format_desc} for input, the device supports encoding > > > > > > > or > > > > > > > + decoding from the \textit{j}-th input format to \textit{i}-th > > > > > > > output > > > > > > > + format. > > > > > > > +\item[\field{width, height}] represents a range of resolutions > > > > > > > + supported by the device. If its \field{step} is not applicable, > > > > > > > its > > > > > > > + \field{min} is equal to its \field{max}. > > > > > > > +\item[\field{num_rates}] is the length of an array > > > > > > > \field{frame_rates}. > > > > > > > In > > > > > > > case of decoder, the driver SHOULD ignore this value. > > > > > > > +\item[\field{frame_rates}] is an array of supported frame rates. > > > > > > > +\end{description} > > > > > > > + > > > > > > > > > > > > I'd guess frame rates depend on the resolution as well. This > > > > > > dependency > > > > > > was > > > > > > clear in the v1 spec, but in the v2 there is no dependency anymore. I > > > > > > think we need to update this. > > > > > > > > > > That's a good point. I missed that dependency when updating the > > > > > structures. > > > > > So, let me update the structs like the following: > > > > > > > > > > struct virtio_video_format_frame { > > > > > > > > > > struct virtio_video_format_range width; > > > > > struct virtio_video_format_range height; > > > > > le32 num_rates; > > > > > u8 padding[4]; > > > > > /* Followed by struct virtio_video_format_range frame_rates[] */ > > > > > > > > > > }; > > > > > > > > > > struct virtio_video_format_desc { > > > > > > > > > > le64 mask; > > > > > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > > le32 planes_layout; /* See the thread [v2 0/1] */ > > > > > le32 num_frames; > > > > > u8 padding[4]; > > > > > /* Followed by struct virtio_video_format_frame frames[] */ > > > > > > > > > > }; > > > > > > > > Yes, I do agree with this approach. > > > > > > > > Best regards, > > > > Dmitry. > > > > > > > > > Best regards, > > > > > Keiichi. > > > > > > > > > > > Best regards, > > > > > > Dmitry. > > > > > > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > > > > > > > + within the device. > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +enum virtio_video_mem_type { > > > > > > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > > > > > > +}; > > > > > > > + > > > > > > > +struct virtio_video_stream_create { > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types > > > > > > > */ > > > > > > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types > > > > > > > */ > > > > > > > + char debug_name[64]; > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[\field{in_mem_type}] is a type of buffer management for input > > > > > > > +buffers. The driver MUST set a value in \field{enum > > > > > > > +virtio_video_mem_type}. > > > > > > > +\item[\field{out_mem_type}] is a type of buffer management for > > > > > > > output > > > > > > > +buffers. The driver MUST set a value in \field{enum > > > > > > > +virtio_video_mem_type}. > > > > > > > +\item[\field{debug_name}] is a text string for a debug purpose. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream > > > > > > > (context) > > > > > > > + within the device. > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +struct virtio_video_stream_destroy { > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > > > > > > > + queued buffers through the pipeline. > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +struct virtio_video_stream_drain { > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > > > > > > > + within the device. > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +struct virtio_video_mem_entry { > > > > > > > + le64 addr; > > > > > > > + le32 length; > > > > > > > + u8 padding[4]; > > > > > > > +}; > > > > > > > + > > > > > > > +struct virtio_video_resource_create { > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > + le32 resource_id; > > > > > > > + le32 nr_entries; > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > > +\item[\field{nr_entries}] number of \field{struct > > > > > > > + virtio_video_mem_entry} memory entries. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource > > > > > > > descriptor > > > > > > > + within the device. > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +struct virtio_video_resource_destroy { > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > + le32 resource_id; > > > > > > > + u8 padding[4]; > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > > > > > > > +queue. > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > > > > > > + > > > > > > > +struct virtio_video_resource_queue { > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > + le32 buf_type; > > > > > > > + le32 resource_id; > > > > > > > + le64 timestamp; > > > > > > > + le32 nr_data_size; > > > > > > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[\field{buf_type}] buf_type of the . > > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be > > > > > > > used > > > > > > > + for synchronisation. > > > > > > > +\item[\field{nr_data_size}] number of \field{data_size} entries. > > > > > > > +\item[\field{data_size}] number of data bytes within a plane. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +enum virtio_video_buffer_flag { > > > > > > > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > > > > > > > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > > > > > > > + /* Encoder only */ > > > > > > > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > > > > > > > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > > > > > > > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > > > > > > > +}; > > > > > > > + > > > > > > > +struct virtio_video_resource_queue_resp { > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > + le64 timestamp; > > > > > > > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > > > > > > > + le32 size; /* Encoded size */ > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be > > > > > > > used > > > > > > > + for synchronisation. > > > > > > > +\item[\field{flags}] mark specific buffers in the sequence. > > > > > > > +\item[\field{size}] data size in the buffer (encoder only). > > > > > > > +\end{description} > > > > > > > + > > > > > > > +The device sends a response to the queue request asynchronously > > > > > > > when > > > > > > > +it has finished processing the buffer. > > > > > > > + > > > > > > > +The device SHOULD mark a buffer that triggered a processing error > > > > > > > with > > > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > > > > > > > + > > > > > > > +The device MUST mark the last buffer with the > > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > > > > > > +sequence. > > > > > > > + > > > > > > > +In case of encoder, to denote a particular frame type the devie > > > > > > > MUST > > > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > > > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > > > > > > > + > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > > > > > > > + buffers back from the input or the output queue of the device. > > > > > > > The > > > > > > > + device SHOULD return all of the buffers from the respective queue > > > > > > > as > > > > > > > + soon as possible without pushing the buffers through the > > > > > > > processing > > > > > > > + pipeline. > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +struct virtio_video_queue_clear { > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > + le32 buf_type; > > > > > > > + u8 padding[4]; > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[\field{buf_type}] buffer type. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > > > > > > > + output of a stream. > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +struct virtio_video_plane_format { > > > > > > > + le32 plane_size; > > > > > > > + le32 stride; > > > > > > > + u8 padding[4]; > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[\field{plane_size}] size of the plane in bytes. > > > > > > > +\item[\field{stride}] stride used for the plane in bytes. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +struct virtio_video_params { > > > > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > > > > > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > > > > > > > + le32 frame_width; > > > > > > > + le32 frame_height; > > > > > > > + le32 min_buffers; > > > > > > > + le32 max_buffers; > > > > > > > + le32 frame_rate; > > > > > > > + struct virtio_video_crop { > > > > > > > + le32 left; > > > > > > > + le32 top; > > > > > > > + le32 width; > > > > > > > + le32 height; > > > > > > > + } crop; > > > > > > > + le32 num_planes; > > > > > > > + struct virtio_video_plane_format > > > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[\field{frame_width}] the value to get/set. > > > > > > > +\item[\field{frame_height}] the value to get/set. > > > > > > > +\item[\field{pixel_format}] the value to get/set. > > > > > > > +\item[\field{min_buffers}] minimum buffers required to handle the > > > > > > > + format (r/o). > > > > > > > +\item[\field{max_buffers}] maximum buffers required to handle the > > > > > > > + format (r/o). > > > > > > > +\item[\field{frame_rate}] the value to get/set. > > > > > > > +\item[\field{crop}] cropping (composing) rectangle. > > > > > > > +\item[\field{num_planes}] number of planes used to store pixel data > > > > > > > +(r/o). > > > > > > > +\item[\field{plane_formats}] description of each plane. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +struct virtio_video_get_params { > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > > > > > +}; > > > > > > > + > > > > > > > +struct virtio_video_get_params_resp { > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > + struct virtio_video_params params; > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[\field{buf_type}] buffer type. > > > > > > > +\item[\field{params}] parameter values. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > > > > > > > + > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +struct virtio_video_set_params { > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > + struct virtio_video_params params; > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[\field{params}] parameters to set. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +Setting stream parameters might have side effects within the > > > > > > > device. > > > > > > > +For example, the device MAY perform alignment of width and height, > > > > > > > +change the number of planes it uses for the format, or do whatever > > > > > > > +changes that are required to continue normal operation using the > > > > > > > +updated parameters. It is up to the driver to check the parameter > > > > > > > set > > > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > > > > > > > + > > > > > > > +\end{description} > > > > > > > + > > > > > > > +\subsubsection{Device Operation: eventq} > > > > > > > + > > > > > > > +The device can report events on the event queue. The driver > > > > > > > initially > > > > > > > +populates the queue with device-writeable buffers. When the device > > > > > > > +needs to report an event, it fills a buffer and notifies the > > > > > > > driver. > > > > > > > +The driver consumes the report and adds a new buffer to the > > > > > > > virtqueue. > > > > > > > + > > > > > > > +\begin{lstlisting} > > > > > > > +enum virtio_video_event_type { > > > > > > > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > > > > > > > + /* For all functions */ > > > > > > > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > > > > > > > + /* For decoder only */ > > > > > > > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > > > > > > > +}; > > > > > > > + > > > > > > > +struct virtio_video_event { > > > > > > > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > > > > > > > + le32 stream_id; > > > > > > > + u8 padding[4]; > > > > > > > +}; > > > > > > > +\end{lstlisting} > > > > > > > + > > > > > > > +\begin{description} > > > > > > > +\item[\field{event_type}] type of the triggered event . > > > > > > > +\item[\field{stream_id}] id of the source stream. > > > > > > > +\end{description} > > > > > > > + > > > > > > > +The device MUST send > > > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > > > > > > > +whenever it encounters new resolution data in the stream. This > > > > > > > +includes the case of the initial device configuration after > > > > > > > metadata > > > > > > > +has been parsed and the case of dynamic resolution change. > > > >
On Wed, Jan 8, 2020 at 9:46 PM Tomasz Figa <tfiga@chromium.org> wrote: > > On Wed, Jan 8, 2020 at 9:15 PM Keiichi Watanabe <keiichiw@chromium.org> wrote: > > > > Hi Dmitry, > > > > On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > > > > > Hi Keiichi, > > > > > > On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote: > > > > Hi Dmitry, > > > > > > > > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > wrote: > > > > > Hi Keiichi, > > > > > > > > > > thanks for the updates, please see my comments below. > > > > > > > > > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote: > > > > > > Hi Dmitry, > > > > > > > > > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp > > > > > > <dmitry.sepp@opensynergy.com> > > > > > > > > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > > > a couple of new comments: > > > > > > > > > > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > > > > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > > > > > > > > > > > The virtio video encoder device and decoder device provide > > > > > > > > functionalities > > > > > > > > to encode and decode video stream respectively. > > > > > > > > Though video encoder and decoder are provided as different devices, > > > > > > > > they > > > > > > > > use a same protocol. > > > > > > > > > > > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > > > > > > > --- > > > > > > > > > > > > > > > > content.tex | 1 + > > > > > > > > virtio-video.tex | 579 > > > > > > > > +++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > > > 2 files changed, 580 insertions(+) > > > > > > > > create mode 100644 virtio-video.tex > > > > > > > > > > > > > > > > diff --git a/content.tex b/content.tex > > > > > > > > index 556b373..9e56839 100644 > > > > > > > > --- a/content.tex > > > > > > > > +++ b/content.tex > > > > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > > > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex} > > > > > > > > > > > > > > > > \input{virtio-fs.tex} > > > > > > > > \input{virtio-rpmb.tex} > > > > > > > > > > > > > > > > +\input{virtio-video.tex} > > > > > > > > > > > > > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > > > > > > > > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > > > > > > > new file mode 100644 > > > > > > > > index 0000000..30e728d > > > > > > > > --- /dev/null > > > > > > > > +++ b/virtio-video.tex > > > > > > > > @@ -0,0 +1,579 @@ > > > > > > > > +\section{Video Device}\label{sec:Device Types / Video Device} > > > > > > > > + > > > > > > > > +The virtio video encoder device and decoder device are virtual > > > > > > > > devices > > > > > > > > that +supports encoding and decoding respectively. Though the > > > > > > > > encoder > > > > > > > > and the decoder +are different devices, they use the same protocol. > > > > > > > > + > > > > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device / > > > > > > > > Device > > > > > > > > ID} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[30] encoder device > > > > > > > > +\item[31] decoder device > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device / > > > > > > > > Virtqueues} + > > > > > > > > +\begin{description} > > > > > > > > +\item[0] controlq - queue for sending control commands. > > > > > > > > +\item[1] eventq - queue for sending events happened in the device. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video Device / > > > > > > > > Feature > > > > > > > > bits} + > > > > > > > > +\begin{description} > > > > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be > > > > > > > > used > > > > > > > > for > > > > > > > > video + buffers. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video > > > > > > > > Device > > > > > > > > / Feature bits} + > > > > > > > > +The device MUST offer at least one of feature bits. > > > > > > > > + > > > > > > > > +\subsection{Device configuration layout}\label{sec:Device Types / > > > > > > > > Video > > > > > > > > Device / Device configuration layout} + > > > > > > > > +Video device configuration uses the following layout structure: > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_config { > > > > > > > > + le32 max_cap_len; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{max_cap_len}] defines the maximum length of a > > > > > > > > descriptor > > > > > > > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device > > > > > > > > + MUST set this value. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\subsection{Device Initialization}\label{sec:Device Types / Video > > > > > > > > Device / > > > > > > > > Device Initialization} + > > > > > > > > +\devicenormative{\subsubsection}{Device Initialization}{Device > > > > > > > > Types / > > > > > > > > Video Device / Device Initialization} + > > > > > > > > +The driver SHOULD query device capability by using the > > > > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the > > > > > > > > initial > > > > > > > > +setup. > > > > > > > > + > > > > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video Device > > > > > > > > / > > > > > > > > Device Operation} + > > > > > > > > +The driver allocates input and output buffers and queues the > > > > > > > > buffers > > > > > > > > +to the device. The device performs operations on the buffers > > > > > > > > according > > > > > > > > +to the function in question. > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Create stream} > > > > > > > > + > > > > > > > > +To process buffers, the device needs to associate them with a > > > > > > > > certain > > > > > > > > +video stream (essentially, a context). Streams are created by > > > > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > > > > > > > +determined by the device. > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Create buffers} > > > > > > > > + > > > > > > > > +Buffers are used to store the actual data as well as the relevant > > > > > > > > +metadata. Scatter lists are supported, so the buffer doesn't need > > > > > > > > to > > > > > > > > +be contiguous in guest physical memory. > > > > > > > > + > > > > > > > > +\begin{itemize*} > > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video > > > > > > > > + resource that is backed by a buffer allocated from the driver's > > > > > > > > + memory. > > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource > > > > > > > > that > > > > > > > > + is no longer needed. > > > > > > > > +\end{itemize*} > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Stream parameter control} > > > > > > > > + > > > > > > > > +\begin{itemize*} > > > > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream > > > > > > > > parameters > > > > > > > > for + input and output streams from the device. > > > > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream > > > > > > > > parameters to > > > > > > > > the + device. > > > > > > > > +\item After setting stream parameters, the driver may issue > > > > > > > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and > > > > > > > > output > > > > > > > > can be + changed implicitly by the device during the set operation. > > > > > > > > +\end{itemize*} > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Process buffers} > > > > > > > > + > > > > > > > > +\begin{itemize*} > > > > > > > > +\item If the function and the buffer type require so, write data to > > > > > > > > +the buffer memory. > > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for > > > > > > > > +processing in the device. > > > > > > > > +\item The request completes asynchronously when the device has > > > > > > > > +finished with the buffer. > > > > > > > > +\end{itemize*} > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Buffer processing control} > > > > > > > > + > > > > > > > > +\begin{itemize*} > > > > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process > > > > > > > > and > > > > > > > > + return all of the already queued buffers. > > > > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return > > > > > > > > back > > > > > > > > + already queued buffers from the input or the output queue. This > > > > > > > > also > > > > > > > > + includes input or output buffers that can be currently owned by > > > > > > > > the > > > > > > > > + device's processing pipeline. > > > > > > > > +\end{itemize*} > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Asynchronous events} > > > > > > > > + > > > > > > > > +While processing buffers, the device can send asynchronous event > > > > > > > > +notifications to the driver. The behaviour depends on the exact > > > > > > > > +stream. For example, the decoder device sends a resolution change > > > > > > > > +event when it encounters new resolution metadata in the stream. > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Request header} > > > > > > > > + > > > > > > > > +All requests and responses on the control virt queue have a fixed > > > > > > > > +header using the following layout structure and definitions: > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +enum virtio_video_ctrl_type { > > > > > > > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > > > > > > > + > > > > > > > > + /* request */ > > > > > > > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > > > > > > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > > > > > > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > > > > > > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > > > > > > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > > > > > > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > > > > > > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > > > > > > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > > > > > > > + VIRTIO_VIDEO_T_SET_PARAMS, > > > > > > > > + VIRTIO_VIDEO_T_GET_PARAMS, > > > > > > > > + > > > > > > > > + /* response */ > > > > > > > > + VIRTIO_VIDEO_S_OK = 0x0200, > > > > > > > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > > > > > > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > > > > > > > + > > > > > > > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > > > > > > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_ctrl_hdr { > > > > > > > > + le32 type; > > > > > > > > + le32 stream_id; > > > > > > > > + le32 len; /* Length of the structure in bytes. */ > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{type}] is the type of the driver request or the device > > > > > > > > +response. > > > > > > > > +\item[\field{stream_id}] specifies a target stream. > > > > > > > > +\item[\field{len}] is the length of data in bytes, which includes > > > > > > > > +length of the header. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: controlq} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about > > > > > > > > +supported formats. > > > > > > > > + > > > > > > > > +The driver uses \field{struct virtio_video_get_capability} to send > > > > > > > > a > > > > > > > > +query request. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +enum virtio_video_buf_type { > > > > > > > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > > > > > > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_get_capability { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + enum virtio_video_buf_type buf_type; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{buf_type}] is the buffer type that the driver asks > > > > > > > > +information about. The driver MUST set either > > > > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > > > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > > > > > > > + > > > > > > > > +The device responds a capability by using \field{struct > > > > > > > > +virtio_video_get_capability_resp}. > > > > > > > > +\begin{lstlisting} > > > > > > > > +enum virtio_video_format { > > > > > > > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > > > > > > > + /* Raw formats */ > > > > > > > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > > > > > > > + VIRTIO_VIDEO_FORMAT_YUV420, > > > > > > > > + VIRTIO_VIDEO_FORMAT_YVU420, > > > > > > > > + > > > > > > > > + /* Compressed formats */ > > > > > > > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > > > > > > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > > > > > > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +enum virtio_video_profile { > > > > > > > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > > > > > > > + > > > > > > > > + /* H.264 */ > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > > > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > > > > > > > > + /* VP8 */ > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = > > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = > > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY, > > > > > > > > + > > > > > > > > + /* VP9 */ > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = > > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = > > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_format_range { > > > > > > > > + le32 min; > > > > > > > > + le32 max; > > > > > > > > + le32 step; > > > > > > > > + u8 paddings[4]; > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_format_desc { > > > > > > > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > > > > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ > > > > > > > > + le64 mask; > > > > > > > > + struct virtio_video_format_range width; > > > > > > > > + struct virtio_video_format_range height; > > > > > > > > + le32 num_rates; > > > > > > > > + u8 padding[4]; > > > > > > > > + /* Followed by struct virtio_video_frame_rate frame_rates[] > > > > > > > > */ > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_get_capability_resp { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 num_descs; > > > > > > > > + /* Followed by struct virtio_video_format_desc desc[] */ > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +The format description \field{struct virtio_video_format_desc} > > > > > > > > +includes the following fields: > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{format}] specifies an image format. The device MUST > > > > > > > > set > > > > > > > > one > > > > > > > > + of \field{enum virtio_video_format}. > > > > > > > > +\item[\field{profile}] specifies a profile of the compressed image > > > > > > > > format > > > > > > > > + specified in \field{format}. The driver SHOULD ignore this value > > > > > > > > if > > > > > > > > + \field{format} is a raw format. > > > > > > > > > > > > > > So how should this be used? The spec does not define any way to set > > > > > > > profile for the device. It is very important for encoder. > > > > > > > > > > > > Thank you for pointing this. > > > > > > These points are overlooked, as I didn't care about encoder usage > > > > > > enough. > > > > > > > > > > > > After thinking it again, I think it's not a very good idea to include > > > > > > supported profiles and levels in a struct for capability. > > > > > > This is because these values are available only for limited number of > > > > > > formats. Also, it's true that we need to have a way to set these values > > > > > > as > > > > > > Dmitry pointed. > > > > > > > > > > Yes, you are right. In fact, the approach of the v1 spec to keep controls > > > > > separately was not correct. > > > > > > > > > > > Instead, it would make more sense to have additional three types of > > > > > > controls for profiles, levels, and bitrates: > > > > > > (1) QUERY_CONTROL: Query values supported by the device > > > > > > (2) GET_CONTROL: Read a value that is set in the device > > > > > > (3) SET_CONTROL: Set a value in the device > > > > > > > > > > > > These operations are similar to V4L2 controls. > > > > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, and > > > > > > G_CTRL in V4L2, respectively. > > > > > > Also, (3) would be similar to enum virtio_video_control_type in the > > > > > > virtio-video v1 driver implementation in > > > > > > https://markmail.org/message/dwghwdqsbl3gsjxu . > > > > > > > > > > > > For QUERY_CONTROL, my idea is like this: > > > > > > > > > > > > enum virtio_video_control_type { > > > > > > > > > > > > VIRTIO_VIDEO_CONTROL_UNDEFINED = 0, > > > > > > > > > > > > VIRTIO_VIDEO_CONTROL_BITRATE = 0x100, > > > > > > VIRTIO_VIDEO_CONTROL_PROFILE, > > > > > > VIRTIO_VIDEO_CONTROL_LEVEL, > > > > > > > > > > > > }; > > > > > > > > > > > > struct virtio_video_query_control { > > > > > > > > > > > > struct virtio_video_ctrl_hdr hdr; > > > > > > le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */ > > > > > > le32 length; > > > > > > /* Followed by additional data. > > > > > > > > > > > > * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE, > > > > > > * the device must pass a codec format like H264 or VP9. > > > > > > * The requred data must be defined in the specification. > > > > > > */ > > > > > > > > > > > > }; > > > > > > > > > > > > struct virtio_video_query_control_resp { > > > > > > > > > > > > struct virtio_video_ctrl_hdr hdr; > > > > > > le32 length; > > > > > > u8 padding[4]; > > > > > > /* Followed by data corresponds to the specified control. > > > > > > > > > > > > * The type of data must be defined in the spec. > > > > > > * For example, if the driver queries profiles, this part should be > > > > > > * an array of supported profiles of a given format. > > > > > > */ > > > > > > > > > > > > }; > > > > > > > > > > > > WDYT? > > > > > > > > > > I think virtio_video_control_type should make sense. But I would disagree > > > > > with the need to have new QUERY_CONTROL and GET_CONTROL. > > > > > > > > > > I assume the set of supported controls is fixed for some particular format > > > > > on a given IP. So we'd propose to include controls into format > > > > > descriptors, so we don't need to QUERY_CONTROL. This way (with > > > > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. for > > > > > profiles. > > > > > > > > > > struct virtio_video_format_frame { > > > > > > > > > > /* As proposed in Keiichi's prev email */ > > > > > > > > > > }; > > > > > > > > > > virtio_video_format_list { > > > > > > > > > > le32 num_entries; > > > > > u8 padding[4]; > > > > > /* Followed by le64 entries[] */ > > > > > > > > > > }; > > > > > > > > > > struct virtio_video_format_control { > > > > > > > > > > le32 type; > > > > > u8 padding[4]; > > > > > struct virtio_video_format_list values; > > > > > > > > > > }; > > > > > > > > > > struct virtio_video_format_desc { > > > > > > > > > > le64 mask; > > > > > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > > le32 planes_layout; /* See the thread [v2 0/1] */ > > > > > le32 num_frames; > > > > > le32 num_controls; > > > > > /* Followed by struct virtio_video_format_frame frames[] */ > > > > > /* Followed by struct virtio_video_format_control controls[] */ > > > > > > > > > > }; > > > > > > > > In my understanding, a set of supported levels depends on a profile. > > > > H.264 spec says that levels are specified within each profile. > > > > cf. "0.5 Profiles and levels" in > > > > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU doesn't > > > > seem to provide a way to query levels for each profile properly, though) > > > > > > > > So, if we want to have supported profiles and levels in format_desc as > > > > your idea, each supported profile should have a list of supported > > > > levels. > > > > I suppose it makes the structure of video_format_desc too complicated. > > > > I'd like to avoid this complexity caused by some specific formats. > > > > > > > > Instead, I'd like to keep virtio_video_format_desc minimal and have > > > > QUERY_CONTROL to query format-specific values like profiles and > > > > levels. > > > > This design would make it easy to extend controls when we want to > > > > support new types of formats. We will just need to define new > > > > VIRTIO_VIDEO_CONTROL_*. > > > > What do you think? > > > > > > Yes, I think we can keep it this way, this indeed provides more flexibility and > > > this way we won't need to modify the 'format' parsing logic when more controls > > > are added. > > > > Thanks! I'm preparing the next version of the patch. > > > > I have one concern about terminology. > > We have two different things called "controls" now: > > 1) Messages passed via controlq. i.e. "ctrl" in 'enum > > virtio_video_ctrl_type' and 'virtio_video_ctrl_hdr'. > > 2) Commands for profiles, levels, and bitrates similar to V4L2 > > controls. e.g.., "CONTROL" in VIRTIO_VIDEO_CONTROL_* > > > > I feel calling both "controls" is somewhat confusing and I am > > wondering if we can't rename either of them. > > > > For example, how about renaming 2) to "command"? Then, we'll have > > {QUERY ,GET, SET}_COMMAND. > > It may sound strange, though. > > Do you have any idea? > > "Command" sound like executing an action, but in this context we're > thinking more about parameters or, exactly, controls. The term > "control" is not limited to V4L2 and actually used in other subsystems > as well, for example the Android Camera HAL. > > How about renaming 1) to "commandq" and then having the other symbols > use the abbreviated "cmd"? Thanks! Keep calling 2) "control" and renaming 1) sounds better now. Actually, virtio-gpu's requests are called "command". (The queue name is "controlq", though) One concern is that the name "controlq" is commonly used among some virtio devices such as virtio-net and virtio-gpu. However, it won't make more sense to stick to using the same queue name as other virtio protocols. Keiichi > > Best regards, > Tomasz > > > > > Best regards, > > Keiichi > > > > > > > > Best regards, > > > Dmitry. > > > > > > > > > > > Best regards, > > > > Keiichi > > > > > > > > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the > > > > > current > > > > > value locally, so there is no need to have GET_CONTROL. > > > > > > > > > > The only exception is the initial (default) control value in the device. > > > > > But with the removal of 'function' and with addition of 'caps' instead, > > > > > the way to provide defaults is gone. So I suppose for formats we'll be > > > > > just using GET on driver start to get the 'defaults'. But the thing is > > > > > that for formats there are other uses for GET, but for controls GET > > > > > apparently does not make to much sense at runtime. > > > > > > > > > > > > Also, shouldn't the profile come together with level? Would make sense > > > > > > > for > > > > > > > encoders. > > > > > > > > > > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be > > > > > > required when querying supported levels. > > > > > > > > > > So probably should be enumerated together, not queried, as per the comment > > > > > above. > > > > > > > > > > > > > +\item[\field{mask}] is a bitset that represents the supported > > > > > > > > + combination of input and output format. If \textit{i}-th bit is > > > > > > > > set > > > > > > > > + in \field{mask} of \textit{j}-th \field{struct > > > > > > > > + virtio_video_format_desc} for input, the device supports encoding > > > > > > > > or > > > > > > > > + decoding from the \textit{j}-th input format to \textit{i}-th > > > > > > > > output > > > > > > > > + format. > > > > > > > > +\item[\field{width, height}] represents a range of resolutions > > > > > > > > + supported by the device. If its \field{step} is not applicable, > > > > > > > > its > > > > > > > > + \field{min} is equal to its \field{max}. > > > > > > > > +\item[\field{num_rates}] is the length of an array > > > > > > > > \field{frame_rates}. > > > > > > > > In > > > > > > > > case of decoder, the driver SHOULD ignore this value. > > > > > > > > +\item[\field{frame_rates}] is an array of supported frame rates. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > > > > > > > I'd guess frame rates depend on the resolution as well. This > > > > > > > dependency > > > > > > > was > > > > > > > clear in the v1 spec, but in the v2 there is no dependency anymore. I > > > > > > > think we need to update this. > > > > > > > > > > > > That's a good point. I missed that dependency when updating the > > > > > > structures. > > > > > > So, let me update the structs like the following: > > > > > > > > > > > > struct virtio_video_format_frame { > > > > > > > > > > > > struct virtio_video_format_range width; > > > > > > struct virtio_video_format_range height; > > > > > > le32 num_rates; > > > > > > u8 padding[4]; > > > > > > /* Followed by struct virtio_video_format_range frame_rates[] */ > > > > > > > > > > > > }; > > > > > > > > > > > > struct virtio_video_format_desc { > > > > > > > > > > > > le64 mask; > > > > > > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > > > le32 planes_layout; /* See the thread [v2 0/1] */ > > > > > > le32 num_frames; > > > > > > u8 padding[4]; > > > > > > /* Followed by struct virtio_video_format_frame frames[] */ > > > > > > > > > > > > }; > > > > > > > > > > Yes, I do agree with this approach. > > > > > > > > > > Best regards, > > > > > Dmitry. > > > > > > > > > > > Best regards, > > > > > > Keiichi. > > > > > > > > > > > > > Best regards, > > > > > > > Dmitry. > > > > > > > > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) > > > > > > > > + within the device. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +enum virtio_video_mem_type { > > > > > > > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_stream_create { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types > > > > > > > > */ > > > > > > > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types > > > > > > > > */ > > > > > > > > + char debug_name[64]; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{in_mem_type}] is a type of buffer management for input > > > > > > > > +buffers. The driver MUST set a value in \field{enum > > > > > > > > +virtio_video_mem_type}. > > > > > > > > +\item[\field{out_mem_type}] is a type of buffer management for > > > > > > > > output > > > > > > > > +buffers. The driver MUST set a value in \field{enum > > > > > > > > +virtio_video_mem_type}. > > > > > > > > +\item[\field{debug_name}] is a text string for a debug purpose. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream > > > > > > > > (context) > > > > > > > > + within the device. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_stream_destroy { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the > > > > > > > > + queued buffers through the pipeline. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_stream_drain { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor > > > > > > > > + within the device. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_mem_entry { > > > > > > > > + le64 addr; > > > > > > > > + le32 length; > > > > > > > > + u8 padding[4]; > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_resource_create { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 resource_id; > > > > > > > > + le32 nr_entries; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > > > +\item[\field{nr_entries}] number of \field{struct > > > > > > > > + virtio_video_mem_entry} memory entries. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource > > > > > > > > descriptor > > > > > > > > + within the device. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_resource_destroy { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 resource_id; > > > > > > > > + u8 padding[4]; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's > > > > > > > > +queue. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > > > > > > > + > > > > > > > > +struct virtio_video_resource_queue { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 buf_type; > > > > > > > > + le32 resource_id; > > > > > > > > + le64 timestamp; > > > > > > > > + le32 nr_data_size; > > > > > > > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{buf_type}] buf_type of the . > > > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be > > > > > > > > used > > > > > > > > + for synchronisation. > > > > > > > > +\item[\field{nr_data_size}] number of \field{data_size} entries. > > > > > > > > +\item[\field{data_size}] number of data bytes within a plane. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +enum virtio_video_buffer_flag { > > > > > > > > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > > > > > > > > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > > > > > > > > + /* Encoder only */ > > > > > > > > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > > > > > > > > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > > > > > > > > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_resource_queue_resp { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le64 timestamp; > > > > > > > > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > > > > > > > > + le32 size; /* Encoded size */ > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that can be > > > > > > > > used > > > > > > > > + for synchronisation. > > > > > > > > +\item[\field{flags}] mark specific buffers in the sequence. > > > > > > > > +\item[\field{size}] data size in the buffer (encoder only). > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +The device sends a response to the queue request asynchronously > > > > > > > > when > > > > > > > > +it has finished processing the buffer. > > > > > > > > + > > > > > > > > +The device SHOULD mark a buffer that triggered a processing error > > > > > > > > with > > > > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > > > > > > > > + > > > > > > > > +The device MUST mark the last buffer with the > > > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain > > > > > > > > +sequence. > > > > > > > > + > > > > > > > > +In case of encoder, to denote a particular frame type the devie > > > > > > > > MUST > > > > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > > > > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued > > > > > > > > + buffers back from the input or the output queue of the device. > > > > > > > > The > > > > > > > > + device SHOULD return all of the buffers from the respective queue > > > > > > > > as > > > > > > > > + soon as possible without pushing the buffers through the > > > > > > > > processing > > > > > > > > + pipeline. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_queue_clear { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 buf_type; > > > > > > > > + u8 padding[4]; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{buf_type}] buffer type. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the > > > > > > > > + output of a stream. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_plane_format { > > > > > > > > + le32 plane_size; > > > > > > > > + le32 stride; > > > > > > > > + u8 padding[4]; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{plane_size}] size of the plane in bytes. > > > > > > > > +\item[\field{stride}] stride used for the plane in bytes. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_params { > > > > > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > > > > > > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ > > > > > > > > + le32 frame_width; > > > > > > > > + le32 frame_height; > > > > > > > > + le32 min_buffers; > > > > > > > > + le32 max_buffers; > > > > > > > > + le32 frame_rate; > > > > > > > > + struct virtio_video_crop { > > > > > > > > + le32 left; > > > > > > > > + le32 top; > > > > > > > > + le32 width; > > > > > > > > + le32 height; > > > > > > > > + } crop; > > > > > > > > + le32 num_planes; > > > > > > > > + struct virtio_video_plane_format > > > > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{frame_width}] the value to get/set. > > > > > > > > +\item[\field{frame_height}] the value to get/set. > > > > > > > > +\item[\field{pixel_format}] the value to get/set. > > > > > > > > +\item[\field{min_buffers}] minimum buffers required to handle the > > > > > > > > + format (r/o). > > > > > > > > +\item[\field{max_buffers}] maximum buffers required to handle the > > > > > > > > + format (r/o). > > > > > > > > +\item[\field{frame_rate}] the value to get/set. > > > > > > > > +\item[\field{crop}] cropping (composing) rectangle. > > > > > > > > +\item[\field{num_planes}] number of planes used to store pixel data > > > > > > > > +(r/o). > > > > > > > > +\item[\field{plane_formats}] description of each plane. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_get_params { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_get_params_resp { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + struct virtio_video_params params; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{buf_type}] buffer type. > > > > > > > > +\item[\field{params}] parameter values. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. > > > > > > > > + > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_set_params { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + struct virtio_video_params params; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{params}] parameters to set. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +Setting stream parameters might have side effects within the > > > > > > > > device. > > > > > > > > +For example, the device MAY perform alignment of width and height, > > > > > > > > +change the number of planes it uses for the format, or do whatever > > > > > > > > +changes that are required to continue normal operation using the > > > > > > > > +updated parameters. It is up to the driver to check the parameter > > > > > > > > set > > > > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > > > > > > > > + > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: eventq} > > > > > > > > + > > > > > > > > +The device can report events on the event queue. The driver > > > > > > > > initially > > > > > > > > +populates the queue with device-writeable buffers. When the device > > > > > > > > +needs to report an event, it fills a buffer and notifies the > > > > > > > > driver. > > > > > > > > +The driver consumes the report and adds a new buffer to the > > > > > > > > virtqueue. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +enum virtio_video_event_type { > > > > > > > > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > > > > > > > > + /* For all functions */ > > > > > > > > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > > > > > > > > + /* For decoder only */ > > > > > > > > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_event { > > > > > > > > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ > > > > > > > > + le32 stream_id; > > > > > > > > + u8 padding[4]; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{event_type}] type of the triggered event . > > > > > > > > +\item[\field{stream_id}] id of the source stream. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +The device MUST send > > > > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > > > > > > > > +whenever it encounters new resolution data in the stream. This > > > > > > > > +includes the case of the initial device configuration after > > > > > > > > metadata > > > > > > > > +has been parsed and the case of dynamic resolution change. > > > > > >
Hi Tomasz, Keiichi, On Mittwoch, 8. Januar 2020 13:46:25 CET Tomasz Figa wrote: > On Wed, Jan 8, 2020 at 9:15 PM Keiichi Watanabe <keiichiw@chromium.org> wrote: > > Hi Dmitry, > > > > On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > > Hi Keiichi, > > > > > > On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote: > > > > Hi Dmitry, > > > > > > > > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp > > > > <dmitry.sepp@opensynergy.com> > > > > > > wrote: > > > > > Hi Keiichi, > > > > > > > > > > thanks for the updates, please see my comments below. > > > > > > > > > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote: > > > > > > Hi Dmitry, > > > > > > > > > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp > > > > > > <dmitry.sepp@opensynergy.com> > > > > > > > > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > > > a couple of new comments: > > > > > > > > > > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe wrote: > > > > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > > > > > > > > > > > The virtio video encoder device and decoder device provide > > > > > > > > functionalities > > > > > > > > to encode and decode video stream respectively. > > > > > > > > Though video encoder and decoder are provided as different > > > > > > > > devices, > > > > > > > > they > > > > > > > > use a same protocol. > > > > > > > > > > > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > > > > > > > --- > > > > > > > > > > > > > > > > content.tex | 1 + > > > > > > > > virtio-video.tex | 579 > > > > > > > > +++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > > > 2 files changed, 580 insertions(+) > > > > > > > > create mode 100644 virtio-video.tex > > > > > > > > > > > > > > > > diff --git a/content.tex b/content.tex > > > > > > > > index 556b373..9e56839 100644 > > > > > > > > --- a/content.tex > > > > > > > > +++ b/content.tex > > > > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > > > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex} > > > > > > > > > > > > > > > > \input{virtio-fs.tex} > > > > > > > > \input{virtio-rpmb.tex} > > > > > > > > > > > > > > > > +\input{virtio-video.tex} > > > > > > > > > > > > > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature > > > > > > > > Bits} > > > > > > > > > > > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > > > > > > > new file mode 100644 > > > > > > > > index 0000000..30e728d > > > > > > > > --- /dev/null > > > > > > > > +++ b/virtio-video.tex > > > > > > > > @@ -0,0 +1,579 @@ > > > > > > > > +\section{Video Device}\label{sec:Device Types / Video Device} > > > > > > > > + > > > > > > > > +The virtio video encoder device and decoder device are > > > > > > > > virtual > > > > > > > > devices > > > > > > > > that +supports encoding and decoding respectively. Though the > > > > > > > > encoder > > > > > > > > and the decoder +are different devices, they use the same > > > > > > > > protocol. > > > > > > > > + > > > > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device > > > > > > > > / > > > > > > > > Device > > > > > > > > ID} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[30] encoder device > > > > > > > > +\item[31] decoder device > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device > > > > > > > > / > > > > > > > > Virtqueues} + > > > > > > > > +\begin{description} > > > > > > > > +\item[0] controlq - queue for sending control commands. > > > > > > > > +\item[1] eventq - queue for sending events happened in the > > > > > > > > device. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video > > > > > > > > Device / > > > > > > > > Feature > > > > > > > > bits} + > > > > > > > > +\begin{description} > > > > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages > > > > > > > > can be > > > > > > > > used > > > > > > > > for > > > > > > > > video + buffers. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / > > > > > > > > Video > > > > > > > > Device > > > > > > > > / Feature bits} + > > > > > > > > +The device MUST offer at least one of feature bits. > > > > > > > > + > > > > > > > > +\subsection{Device configuration layout}\label{sec:Device > > > > > > > > Types / > > > > > > > > Video > > > > > > > > Device / Device configuration layout} + > > > > > > > > +Video device configuration uses the following layout > > > > > > > > structure: > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_config { > > > > > > > > + le32 max_cap_len; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{max_cap_len}] defines the maximum length of a > > > > > > > > descriptor > > > > > > > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The > > > > > > > > device > > > > > > > > + MUST set this value. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\subsection{Device Initialization}\label{sec:Device Types / > > > > > > > > Video > > > > > > > > Device / > > > > > > > > Device Initialization} + > > > > > > > > +\devicenormative{\subsubsection}{Device > > > > > > > > Initialization}{Device > > > > > > > > Types / > > > > > > > > Video Device / Device Initialization} + > > > > > > > > +The driver SHOULD query device capability by using the > > > > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for > > > > > > > > the > > > > > > > > initial > > > > > > > > +setup. > > > > > > > > + > > > > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video > > > > > > > > Device > > > > > > > > / > > > > > > > > Device Operation} + > > > > > > > > +The driver allocates input and output buffers and queues the > > > > > > > > buffers > > > > > > > > +to the device. The device performs operations on the buffers > > > > > > > > according > > > > > > > > +to the function in question. > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Create stream} > > > > > > > > + > > > > > > > > +To process buffers, the device needs to associate them with a > > > > > > > > certain > > > > > > > > +video stream (essentially, a context). Streams are created by > > > > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > > > > > > > +determined by the device. > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Create buffers} > > > > > > > > + > > > > > > > > +Buffers are used to store the actual data as well as the > > > > > > > > relevant > > > > > > > > +metadata. Scatter lists are supported, so the buffer doesn't > > > > > > > > need > > > > > > > > to > > > > > > > > +be contiguous in guest physical memory. > > > > > > > > + > > > > > > > > +\begin{itemize*} > > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio > > > > > > > > video > > > > > > > > + resource that is backed by a buffer allocated from the > > > > > > > > driver's > > > > > > > > + memory. > > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a > > > > > > > > resource > > > > > > > > that > > > > > > > > + is no longer needed. > > > > > > > > +\end{itemize*} > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Stream parameter control} > > > > > > > > + > > > > > > > > +\begin{itemize*} > > > > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream > > > > > > > > parameters > > > > > > > > for + input and output streams from the device. > > > > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream > > > > > > > > parameters to > > > > > > > > the + device. > > > > > > > > +\item After setting stream parameters, the driver may issue > > > > > > > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input > > > > > > > > and > > > > > > > > output > > > > > > > > can be + changed implicitly by the device during the set > > > > > > > > operation. > > > > > > > > +\end{itemize*} > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Process buffers} > > > > > > > > + > > > > > > > > +\begin{itemize*} > > > > > > > > +\item If the function and the buffer type require so, write > > > > > > > > data to > > > > > > > > +the buffer memory. > > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer > > > > > > > > for > > > > > > > > +processing in the device. > > > > > > > > +\item The request completes asynchronously when the device > > > > > > > > has > > > > > > > > +finished with the buffer. > > > > > > > > +\end{itemize*} > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Buffer processing control} > > > > > > > > + > > > > > > > > +\begin{itemize*} > > > > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to > > > > > > > > process > > > > > > > > and > > > > > > > > + return all of the already queued buffers. > > > > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to > > > > > > > > return > > > > > > > > back > > > > > > > > + already queued buffers from the input or the output queue. > > > > > > > > This > > > > > > > > also > > > > > > > > + includes input or output buffers that can be currently > > > > > > > > owned by > > > > > > > > the > > > > > > > > + device's processing pipeline. > > > > > > > > +\end{itemize*} > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Asynchronous events} > > > > > > > > + > > > > > > > > +While processing buffers, the device can send asynchronous > > > > > > > > event > > > > > > > > +notifications to the driver. The behaviour depends on the > > > > > > > > exact > > > > > > > > +stream. For example, the decoder device sends a resolution > > > > > > > > change > > > > > > > > +event when it encounters new resolution metadata in the > > > > > > > > stream. > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: Request header} > > > > > > > > + > > > > > > > > +All requests and responses on the control virt queue have a > > > > > > > > fixed > > > > > > > > +header using the following layout structure and definitions: > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +enum virtio_video_ctrl_type { > > > > > > > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > > > > > > > + > > > > > > > > + /* request */ > > > > > > > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > > > > > > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > > > > > > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > > > > > > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > > > > > > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > > > > > > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > > > > > > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > > > > > > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > > > > > > > + VIRTIO_VIDEO_T_SET_PARAMS, > > > > > > > > + VIRTIO_VIDEO_T_GET_PARAMS, > > > > > > > > + > > > > > > > > + /* response */ > > > > > > > > + VIRTIO_VIDEO_S_OK = 0x0200, > > > > > > > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > > > > > > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > > > > > > > + > > > > > > > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > > > > > > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_ctrl_hdr { > > > > > > > > + le32 type; > > > > > > > > + le32 stream_id; > > > > > > > > + le32 len; /* Length of the structure in bytes. */ > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{type}] is the type of the driver request or the > > > > > > > > device > > > > > > > > +response. > > > > > > > > +\item[\field{stream_id}] specifies a target stream. > > > > > > > > +\item[\field{len}] is the length of data in bytes, which > > > > > > > > includes > > > > > > > > +length of the header. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: controlq} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information > > > > > > > > about > > > > > > > > +supported formats. > > > > > > > > + > > > > > > > > +The driver uses \field{struct virtio_video_get_capability} to > > > > > > > > send > > > > > > > > a > > > > > > > > +query request. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +enum virtio_video_buf_type { > > > > > > > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > > > > > > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_get_capability { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + enum virtio_video_buf_type buf_type; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{buf_type}] is the buffer type that the driver > > > > > > > > asks > > > > > > > > +information about. The driver MUST set either > > > > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > > > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > > > > > > > + > > > > > > > > +The device responds a capability by using \field{struct > > > > > > > > +virtio_video_get_capability_resp}. > > > > > > > > +\begin{lstlisting} > > > > > > > > +enum virtio_video_format { > > > > > > > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > > > > > > > + /* Raw formats */ > > > > > > > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > > > > > > > + VIRTIO_VIDEO_FORMAT_YUV420, > > > > > > > > + VIRTIO_VIDEO_FORMAT_YVU420, > > > > > > > > + > > > > > > > > + /* Compressed formats */ > > > > > > > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > > > > > > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > > > > > > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +enum virtio_video_profile { > > > > > > > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > > > > > > > + > > > > > > > > + /* H.264 */ > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > > > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > > > > > > > > + /* VP8 */ > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = > > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = > > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY, > > > > > > > > + > > > > > > > > + /* VP9 */ > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = > > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = > > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_format_range { > > > > > > > > + le32 min; > > > > > > > > + le32 max; > > > > > > > > + le32 step; > > > > > > > > + u8 paddings[4]; > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_format_desc { > > > > > > > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types > > > > > > > > */ > > > > > > > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types > > > > > > > > */ > > > > > > > > + le64 mask; > > > > > > > > + struct virtio_video_format_range width; > > > > > > > > + struct virtio_video_format_range height; > > > > > > > > + le32 num_rates; > > > > > > > > + u8 padding[4]; > > > > > > > > + /* Followed by struct virtio_video_frame_rate > > > > > > > > frame_rates[] > > > > > > > > */ > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_get_capability_resp { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 num_descs; > > > > > > > > + /* Followed by struct virtio_video_format_desc desc[] > > > > > > > > */ > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +The format description \field{struct > > > > > > > > virtio_video_format_desc} > > > > > > > > +includes the following fields: > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{format}] specifies an image format. The device > > > > > > > > MUST > > > > > > > > set > > > > > > > > one > > > > > > > > + of \field{enum virtio_video_format}. > > > > > > > > +\item[\field{profile}] specifies a profile of the compressed > > > > > > > > image > > > > > > > > format > > > > > > > > + specified in \field{format}. The driver SHOULD ignore this > > > > > > > > value > > > > > > > > if > > > > > > > > + \field{format} is a raw format. > > > > > > > > > > > > > > So how should this be used? The spec does not define any way to > > > > > > > set > > > > > > > profile for the device. It is very important for encoder. > > > > > > > > > > > > Thank you for pointing this. > > > > > > These points are overlooked, as I didn't care about encoder usage > > > > > > enough. > > > > > > > > > > > > After thinking it again, I think it's not a very good idea to > > > > > > include > > > > > > supported profiles and levels in a struct for capability. > > > > > > This is because these values are available only for limited number > > > > > > of > > > > > > formats. Also, it's true that we need to have a way to set these > > > > > > values > > > > > > as > > > > > > Dmitry pointed. > > > > > > > > > > Yes, you are right. In fact, the approach of the v1 spec to keep > > > > > controls > > > > > separately was not correct. > > > > > > > > > > > Instead, it would make more sense to have additional three types > > > > > > of > > > > > > controls for profiles, levels, and bitrates: > > > > > > (1) QUERY_CONTROL: Query values supported by the device > > > > > > (2) GET_CONTROL: Read a value that is set in the device > > > > > > (3) SET_CONTROL: Set a value in the device > > > > > > > > > > > > These operations are similar to V4L2 controls. > > > > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, > > > > > > and > > > > > > G_CTRL in V4L2, respectively. > > > > > > Also, (3) would be similar to enum virtio_video_control_type in > > > > > > the > > > > > > virtio-video v1 driver implementation in > > > > > > https://markmail.org/message/dwghwdqsbl3gsjxu . > > > > > > > > > > > > For QUERY_CONTROL, my idea is like this: > > > > > > > > > > > > enum virtio_video_control_type { > > > > > > > > > > > > VIRTIO_VIDEO_CONTROL_UNDEFINED = 0, > > > > > > > > > > > > VIRTIO_VIDEO_CONTROL_BITRATE = 0x100, > > > > > > VIRTIO_VIDEO_CONTROL_PROFILE, > > > > > > VIRTIO_VIDEO_CONTROL_LEVEL, > > > > > > > > > > > > }; > > > > > > > > > > > > struct virtio_video_query_control { > > > > > > > > > > > > struct virtio_video_ctrl_hdr hdr; > > > > > > le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */ > > > > > > le32 length; > > > > > > /* Followed by additional data. > > > > > > > > > > > > * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE, > > > > > > * the device must pass a codec format like H264 or VP9. > > > > > > * The requred data must be defined in the specification. > > > > > > */ > > > > > > > > > > > > }; > > > > > > > > > > > > struct virtio_video_query_control_resp { > > > > > > > > > > > > struct virtio_video_ctrl_hdr hdr; > > > > > > le32 length; > > > > > > u8 padding[4]; > > > > > > /* Followed by data corresponds to the specified control. > > > > > > > > > > > > * The type of data must be defined in the spec. > > > > > > * For example, if the driver queries profiles, this part should > > > > > > be > > > > > > * an array of supported profiles of a given format. > > > > > > */ > > > > > > > > > > > > }; > > > > > > > > > > > > WDYT? > > > > > > > > > > I think virtio_video_control_type should make sense. But I would > > > > > disagree > > > > > with the need to have new QUERY_CONTROL and GET_CONTROL. > > > > > > > > > > I assume the set of supported controls is fixed for some particular > > > > > format > > > > > on a given IP. So we'd propose to include controls into format > > > > > descriptors, so we don't need to QUERY_CONTROL. This way (with > > > > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. > > > > > for > > > > > profiles. > > > > > > > > > > struct virtio_video_format_frame { > > > > > > > > > > /* As proposed in Keiichi's prev email */ > > > > > > > > > > }; > > > > > > > > > > virtio_video_format_list { > > > > > > > > > > le32 num_entries; > > > > > u8 padding[4]; > > > > > /* Followed by le64 entries[] */ > > > > > > > > > > }; > > > > > > > > > > struct virtio_video_format_control { > > > > > > > > > > le32 type; > > > > > u8 padding[4]; > > > > > struct virtio_video_format_list values; > > > > > > > > > > }; > > > > > > > > > > struct virtio_video_format_desc { > > > > > > > > > > le64 mask; > > > > > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > > le32 planes_layout; /* See the thread [v2 0/1] */ > > > > > le32 num_frames; > > > > > le32 num_controls; > > > > > /* Followed by struct virtio_video_format_frame frames[] */ > > > > > /* Followed by struct virtio_video_format_control controls[] > > > > > */ > > > > > > > > > > }; > > > > > > > > In my understanding, a set of supported levels depends on a profile. > > > > H.264 spec says that levels are specified within each profile. > > > > cf. "0.5 Profiles and levels" in > > > > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU > > > > doesn't > > > > seem to provide a way to query levels for each profile properly, > > > > though) > > > > > > > > So, if we want to have supported profiles and levels in format_desc as > > > > your idea, each supported profile should have a list of supported > > > > levels. > > > > I suppose it makes the structure of video_format_desc too complicated. > > > > I'd like to avoid this complexity caused by some specific formats. > > > > > > > > Instead, I'd like to keep virtio_video_format_desc minimal and have > > > > QUERY_CONTROL to query format-specific values like profiles and > > > > levels. > > > > This design would make it easy to extend controls when we want to > > > > support new types of formats. We will just need to define new > > > > VIRTIO_VIDEO_CONTROL_*. > > > > What do you think? > > > > > > Yes, I think we can keep it this way, this indeed provides more > > > flexibility and this way we won't need to modify the 'format' parsing > > > logic when more controls are added. > > > > Thanks! I'm preparing the next version of the patch. > > > > I have one concern about terminology. > > We have two different things called "controls" now: > > 1) Messages passed via controlq. i.e. "ctrl" in 'enum > > virtio_video_ctrl_type' and 'virtio_video_ctrl_hdr'. > > 2) Commands for profiles, levels, and bitrates similar to V4L2 > > controls. e.g.., "CONTROL" in VIRTIO_VIDEO_CONTROL_* > > > > I feel calling both "controls" is somewhat confusing and I am > > wondering if we can't rename either of them. > > > > For example, how about renaming 2) to "command"? Then, we'll have > > {QUERY ,GET, SET}_COMMAND. > > It may sound strange, though. > > Do you have any idea? > > "Command" sound like executing an action, but in this context we're > thinking more about parameters or, exactly, controls. The term > "control" is not limited to V4L2 and actually used in other subsystems > as well, for example the Android Camera HAL. > > How about renaming 1) to "commandq" and then having the other symbols > use the abbreviated "cmd"? First I thought about renaming 'virtio_video_ctrl_hdr' to 'virtio_video_req_hdr', but then 'reqq' wouldn't look nice. So I'd be ok to go with 'commandq' and 'cmd'. T is for type, S is for status. I was looking at the virtio-iommu spec around that time, I think I derived it from there. Regards, Dmitry > > Best regards, > Tomasz > > > Best regards, > > Keiichi > > > > > Best regards, > > > Dmitry. > > > > > > > Best regards, > > > > Keiichi > > > > > > > > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the > > > > > current > > > > > value locally, so there is no need to have GET_CONTROL. > > > > > > > > > > The only exception is the initial (default) control value in the > > > > > device. > > > > > But with the removal of 'function' and with addition of 'caps' > > > > > instead, > > > > > the way to provide defaults is gone. So I suppose for formats we'll > > > > > be > > > > > just using GET on driver start to get the 'defaults'. But the thing > > > > > is > > > > > that for formats there are other uses for GET, but for controls GET > > > > > apparently does not make to much sense at runtime. > > > > > > > > > > > > Also, shouldn't the profile come together with level? Would make > > > > > > > sense > > > > > > > for > > > > > > > encoders. > > > > > > > > > > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be > > > > > > required when querying supported levels. > > > > > > > > > > So probably should be enumerated together, not queried, as per the > > > > > comment > > > > > above. > > > > > > > > > > > > > +\item[\field{mask}] is a bitset that represents the supported > > > > > > > > + combination of input and output format. If \textit{i}-th > > > > > > > > bit is > > > > > > > > set > > > > > > > > + in \field{mask} of \textit{j}-th \field{struct > > > > > > > > + virtio_video_format_desc} for input, the device supports > > > > > > > > encoding > > > > > > > > or > > > > > > > > + decoding from the \textit{j}-th input format to > > > > > > > > \textit{i}-th > > > > > > > > output > > > > > > > > + format. > > > > > > > > +\item[\field{width, height}] represents a range of > > > > > > > > resolutions > > > > > > > > + supported by the device. If its \field{step} is not > > > > > > > > applicable, > > > > > > > > its > > > > > > > > + \field{min} is equal to its \field{max}. > > > > > > > > +\item[\field{num_rates}] is the length of an array > > > > > > > > \field{frame_rates}. > > > > > > > > In > > > > > > > > case of decoder, the driver SHOULD ignore this value. > > > > > > > > +\item[\field{frame_rates}] is an array of supported frame > > > > > > > > rates. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > > > > > > > I'd guess frame rates depend on the resolution as well. This > > > > > > > dependency > > > > > > > was > > > > > > > clear in the v1 spec, but in the v2 there is no dependency > > > > > > > anymore. I > > > > > > > think we need to update this. > > > > > > > > > > > > That's a good point. I missed that dependency when updating the > > > > > > structures. > > > > > > So, let me update the structs like the following: > > > > > > > > > > > > struct virtio_video_format_frame { > > > > > > > > > > > > struct virtio_video_format_range width; > > > > > > struct virtio_video_format_range height; > > > > > > le32 num_rates; > > > > > > u8 padding[4]; > > > > > > /* Followed by struct virtio_video_format_range > > > > > > frame_rates[] */ > > > > > > > > > > > > }; > > > > > > > > > > > > struct virtio_video_format_desc { > > > > > > > > > > > > le64 mask; > > > > > > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > > > le32 planes_layout; /* See the thread [v2 0/1] */ > > > > > > le32 num_frames; > > > > > > u8 padding[4]; > > > > > > /* Followed by struct virtio_video_format_frame frames[] > > > > > > */ > > > > > > > > > > > > }; > > > > > > > > > > Yes, I do agree with this approach. > > > > > > > > > > Best regards, > > > > > Dmitry. > > > > > > > > > > > Best regards, > > > > > > Keiichi. > > > > > > > > > > > > > Best regards, > > > > > > > Dmitry. > > > > > > > > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream > > > > > > > > (context) > > > > > > > > + within the device. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +enum virtio_video_mem_type { > > > > > > > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_stream_create { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* > > > > > > > > types > > > > > > > > */ > > > > > > > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* > > > > > > > > types > > > > > > > > */ > > > > > > > > + char debug_name[64]; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{in_mem_type}] is a type of buffer management for > > > > > > > > input > > > > > > > > +buffers. The driver MUST set a value in \field{enum > > > > > > > > +virtio_video_mem_type}. > > > > > > > > +\item[\field{out_mem_type}] is a type of buffer management > > > > > > > > for > > > > > > > > output > > > > > > > > +buffers. The driver MUST set a value in \field{enum > > > > > > > > +virtio_video_mem_type}. > > > > > > > > +\item[\field{debug_name}] is a text string for a debug > > > > > > > > purpose. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream > > > > > > > > (context) > > > > > > > > + within the device. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_stream_destroy { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all > > > > > > > > the > > > > > > > > + queued buffers through the pipeline. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_stream_drain { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource > > > > > > > > descriptor > > > > > > > > + within the device. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_mem_entry { > > > > > > > > + le64 addr; > > > > > > > > + le32 length; > > > > > > > > + u8 padding[4]; > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_resource_create { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 resource_id; > > > > > > > > + le32 nr_entries; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > > > +\item[\field{nr_entries}] number of \field{struct > > > > > > > > + virtio_video_mem_entry} memory entries. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource > > > > > > > > descriptor > > > > > > > > + within the device. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_resource_destroy { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 resource_id; > > > > > > > > + u8 padding[4]; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the > > > > > > > > device's > > > > > > > > +queue. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > > > > > > > + > > > > > > > > +struct virtio_video_resource_queue { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 buf_type; > > > > > > > > + le32 resource_id; > > > > > > > > + le64 timestamp; > > > > > > > > + le32 nr_data_size; > > > > > > > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{buf_type}] buf_type of the . > > > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that > > > > > > > > can be > > > > > > > > used > > > > > > > > + for synchronisation. > > > > > > > > +\item[\field{nr_data_size}] number of \field{data_size} > > > > > > > > entries. > > > > > > > > +\item[\field{data_size}] number of data bytes within a plane. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +enum virtio_video_buffer_flag { > > > > > > > > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > > > > > > > > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > > > > > > > > + /* Encoder only */ > > > > > > > > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > > > > > > > > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > > > > > > > > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_resource_queue_resp { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le64 timestamp; > > > > > > > > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > > > > > > > > + le32 size; /* Encoded size */ > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that > > > > > > > > can be > > > > > > > > used > > > > > > > > + for synchronisation. > > > > > > > > +\item[\field{flags}] mark specific buffers in the sequence. > > > > > > > > +\item[\field{size}] data size in the buffer (encoder only). > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +The device sends a response to the queue request > > > > > > > > asynchronously > > > > > > > > when > > > > > > > > +it has finished processing the buffer. > > > > > > > > + > > > > > > > > +The device SHOULD mark a buffer that triggered a processing > > > > > > > > error > > > > > > > > with > > > > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > > > > > > > > + > > > > > > > > +The device MUST mark the last buffer with the > > > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the > > > > > > > > drain > > > > > > > > +sequence. > > > > > > > > + > > > > > > > > +In case of encoder, to denote a particular frame type the > > > > > > > > devie > > > > > > > > MUST > > > > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > > > > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already > > > > > > > > queued > > > > > > > > + buffers back from the input or the output queue of the > > > > > > > > device. > > > > > > > > The > > > > > > > > + device SHOULD return all of the buffers from the respective > > > > > > > > queue > > > > > > > > as > > > > > > > > + soon as possible without pushing the buffers through the > > > > > > > > processing > > > > > > > > + pipeline. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_queue_clear { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 buf_type; > > > > > > > > + u8 padding[4]; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{buf_type}] buffer type. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input > > > > > > > > or the > > > > > > > > + output of a stream. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_plane_format { > > > > > > > > + le32 plane_size; > > > > > > > > + le32 stride; > > > > > > > > + u8 padding[4]; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{plane_size}] size of the plane in bytes. > > > > > > > > +\item[\field{stride}] stride used for the plane in bytes. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_params { > > > > > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* > > > > > > > > types */ > > > > > > > > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types > > > > > > > > */ > > > > > > > > + le32 frame_width; > > > > > > > > + le32 frame_height; > > > > > > > > + le32 min_buffers; > > > > > > > > + le32 max_buffers; > > > > > > > > + le32 frame_rate; > > > > > > > > + struct virtio_video_crop { > > > > > > > > + le32 left; > > > > > > > > + le32 top; > > > > > > > > + le32 width; > > > > > > > > + le32 height; > > > > > > > > + } crop; > > > > > > > > + le32 num_planes; > > > > > > > > + struct virtio_video_plane_format > > > > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{frame_width}] the value to get/set. > > > > > > > > +\item[\field{frame_height}] the value to get/set. > > > > > > > > +\item[\field{pixel_format}] the value to get/set. > > > > > > > > +\item[\field{min_buffers}] minimum buffers required to handle > > > > > > > > the > > > > > > > > + format (r/o). > > > > > > > > +\item[\field{max_buffers}] maximum buffers required to handle > > > > > > > > the > > > > > > > > + format (r/o). > > > > > > > > +\item[\field{frame_rate}] the value to get/set. > > > > > > > > +\item[\field{crop}] cropping (composing) rectangle. > > > > > > > > +\item[\field{num_planes}] number of planes used to store > > > > > > > > pixel data > > > > > > > > +(r/o). > > > > > > > > +\item[\field{plane_formats}] description of each plane. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_get_params { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* > > > > > > > > types */ > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_get_params_resp { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + struct virtio_video_params params; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{buf_type}] buffer type. > > > > > > > > +\item[\field{params}] parameter values. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a > > > > > > > > stream. > > > > > > > > + > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +struct virtio_video_set_params { > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > + struct virtio_video_params params; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{params}] parameters to set. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +Setting stream parameters might have side effects within the > > > > > > > > device. > > > > > > > > +For example, the device MAY perform alignment of width and > > > > > > > > height, > > > > > > > > +change the number of planes it uses for the format, or do > > > > > > > > whatever > > > > > > > > +changes that are required to continue normal operation using > > > > > > > > the > > > > > > > > +updated parameters. It is up to the driver to check the > > > > > > > > parameter > > > > > > > > set > > > > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > > > > > > > > + > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +\subsubsection{Device Operation: eventq} > > > > > > > > + > > > > > > > > +The device can report events on the event queue. The driver > > > > > > > > initially > > > > > > > > +populates the queue with device-writeable buffers. When the > > > > > > > > device > > > > > > > > +needs to report an event, it fills a buffer and notifies the > > > > > > > > driver. > > > > > > > > +The driver consumes the report and adds a new buffer to the > > > > > > > > virtqueue. > > > > > > > > + > > > > > > > > +\begin{lstlisting} > > > > > > > > +enum virtio_video_event_type { > > > > > > > > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > > > > > > > > + /* For all functions */ > > > > > > > > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > > > > > > > > + /* For decoder only */ > > > > > > > > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = > > > > > > > > 0x0200, > > > > > > > > +}; > > > > > > > > + > > > > > > > > +struct virtio_video_event { > > > > > > > > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* > > > > > > > > types */ > > > > > > > > + le32 stream_id; > > > > > > > > + u8 padding[4]; > > > > > > > > +}; > > > > > > > > +\end{lstlisting} > > > > > > > > + > > > > > > > > +\begin{description} > > > > > > > > +\item[\field{event_type}] type of the triggered event . > > > > > > > > +\item[\field{stream_id}] id of the source stream. > > > > > > > > +\end{description} > > > > > > > > + > > > > > > > > +The device MUST send > > > > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > > > > > > > > +whenever it encounters new resolution data in the stream. > > > > > > > > This > > > > > > > > +includes the case of the initial device configuration after > > > > > > > > metadata > > > > > > > > +has been parsed and the case of dynamic resolution change.
On Wed, Jan 8, 2020 at 10:11 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi Tomasz, Keiichi, > > On Mittwoch, 8. Januar 2020 13:46:25 CET Tomasz Figa wrote: > > On Wed, Jan 8, 2020 at 9:15 PM Keiichi Watanabe <keiichiw@chromium.org> > wrote: > > > Hi Dmitry, > > > > > > On Wed, Jan 8, 2020 at 7:00 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> > wrote: > > > > Hi Keiichi, > > > > > > > > On Mittwoch, 8. Januar 2020 07:59:22 CET Keiichi Watanabe wrote: > > > > > Hi Dmitry, > > > > > > > > > > On Wed, Jan 8, 2020 at 1:50 AM Dmitry Sepp > > > > > <dmitry.sepp@opensynergy.com> > > > > > > > > wrote: > > > > > > Hi Keiichi, > > > > > > > > > > > > thanks for the updates, please see my comments below. > > > > > > > > > > > > On Dienstag, 7. Januar 2020 14:24:31 CET Keiichi Watanabe wrote: > > > > > > > Hi Dmitry, > > > > > > > > > > > > > > On Mon, Jan 6, 2020 at 11:59 PM Dmitry Sepp > > > > > > > <dmitry.sepp@opensynergy.com> > > > > > > > > > > > > wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > a couple of new comments: > > > > > > > > > > > > > > > > On Mittwoch, 18. Dezember 2019 14:02:14 CET Keiichi Watanabe > wrote: > > > > > > > > > From: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > > > > > > > > > > > > > The virtio video encoder device and decoder device provide > > > > > > > > > functionalities > > > > > > > > > to encode and decode video stream respectively. > > > > > > > > > Though video encoder and decoder are provided as different > > > > > > > > > devices, > > > > > > > > > they > > > > > > > > > use a same protocol. > > > > > > > > > > > > > > > > > > Signed-off-by: Dmitry Sepp <dmitry.sepp@opensynergy.com> > > > > > > > > > Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org> > > > > > > > > > --- > > > > > > > > > > > > > > > > > > content.tex | 1 + > > > > > > > > > virtio-video.tex | 579 > > > > > > > > > +++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > > > > 2 files changed, 580 insertions(+) > > > > > > > > > create mode 100644 virtio-video.tex > > > > > > > > > > > > > > > > > > diff --git a/content.tex b/content.tex > > > > > > > > > index 556b373..9e56839 100644 > > > > > > > > > --- a/content.tex > > > > > > > > > +++ b/content.tex > > > > > > > > > @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing > > > > > > > > > Requirements}\label{sec:Device \input{virtio-vsock.tex} > > > > > > > > > > > > > > > > > > \input{virtio-fs.tex} > > > > > > > > > \input{virtio-rpmb.tex} > > > > > > > > > > > > > > > > > > +\input{virtio-video.tex} > > > > > > > > > > > > > > > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature > > > > > > > > > Bits} > > > > > > > > > > > > > > > > > > diff --git a/virtio-video.tex b/virtio-video.tex > > > > > > > > > new file mode 100644 > > > > > > > > > index 0000000..30e728d > > > > > > > > > --- /dev/null > > > > > > > > > +++ b/virtio-video.tex > > > > > > > > > @@ -0,0 +1,579 @@ > > > > > > > > > +\section{Video Device}\label{sec:Device Types / Video Device} > > > > > > > > > + > > > > > > > > > +The virtio video encoder device and decoder device are > > > > > > > > > virtual > > > > > > > > > devices > > > > > > > > > that +supports encoding and decoding respectively. Though the > > > > > > > > > encoder > > > > > > > > > and the decoder +are different devices, they use the same > > > > > > > > > protocol. > > > > > > > > > + > > > > > > > > > +\subsection{Device ID}\label{sec:Device Types / Video Device > > > > > > > > > / > > > > > > > > > Device > > > > > > > > > ID} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[30] encoder device > > > > > > > > > +\item[31] decoder device > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\subsection{Virtqueues}\label{sec:Device Types / Video Device > > > > > > > > > / > > > > > > > > > Virtqueues} + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[0] controlq - queue for sending control commands. > > > > > > > > > +\item[1] eventq - queue for sending events happened in the > > > > > > > > > device. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\subsection{Feature bits}\label{sec:Device Types / Video > > > > > > > > > Device / > > > > > > > > > Feature > > > > > > > > > bits} + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages > > > > > > > > > can be > > > > > > > > > used > > > > > > > > > for > > > > > > > > > video + buffers. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / > > > > > > > > > Video > > > > > > > > > Device > > > > > > > > > / Feature bits} + > > > > > > > > > +The device MUST offer at least one of feature bits. > > > > > > > > > + > > > > > > > > > +\subsection{Device configuration layout}\label{sec:Device > > > > > > > > > Types / > > > > > > > > > Video > > > > > > > > > Device / Device configuration layout} + > > > > > > > > > +Video device configuration uses the following layout > > > > > > > > > structure: > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +struct virtio_video_config { > > > > > > > > > + le32 max_cap_len; > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{max_cap_len}] defines the maximum length of a > > > > > > > > > descriptor > > > > > > > > > + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The > > > > > > > > > device > > > > > > > > > + MUST set this value. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\subsection{Device Initialization}\label{sec:Device Types / > > > > > > > > > Video > > > > > > > > > Device / > > > > > > > > > Device Initialization} + > > > > > > > > > +\devicenormative{\subsubsection}{Device > > > > > > > > > Initialization}{Device > > > > > > > > > Types / > > > > > > > > > Video Device / Device Initialization} + > > > > > > > > > +The driver SHOULD query device capability by using the > > > > > > > > > +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for > > > > > > > > > the > > > > > > > > > initial > > > > > > > > > +setup. > > > > > > > > > + > > > > > > > > > +\subsection{Device Operation}\label{sec:Device Types / Video > > > > > > > > > Device > > > > > > > > > / > > > > > > > > > Device Operation} + > > > > > > > > > +The driver allocates input and output buffers and queues the > > > > > > > > > buffers > > > > > > > > > +to the device. The device performs operations on the buffers > > > > > > > > > according > > > > > > > > > +to the function in question. > > > > > > > > > + > > > > > > > > > +\subsubsection{Device Operation: Create stream} > > > > > > > > > + > > > > > > > > > +To process buffers, the device needs to associate them with a > > > > > > > > > certain > > > > > > > > > +video stream (essentially, a context). Streams are created by > > > > > > > > > +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters > > > > > > > > > +determined by the device. > > > > > > > > > + > > > > > > > > > +\subsubsection{Device Operation: Create buffers} > > > > > > > > > + > > > > > > > > > +Buffers are used to store the actual data as well as the > > > > > > > > > relevant > > > > > > > > > +metadata. Scatter lists are supported, so the buffer doesn't > > > > > > > > > need > > > > > > > > > to > > > > > > > > > +be contiguous in guest physical memory. > > > > > > > > > + > > > > > > > > > +\begin{itemize*} > > > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio > > > > > > > > > video > > > > > > > > > + resource that is backed by a buffer allocated from the > > > > > > > > > driver's > > > > > > > > > + memory. > > > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a > > > > > > > > > resource > > > > > > > > > that > > > > > > > > > + is no longer needed. > > > > > > > > > +\end{itemize*} > > > > > > > > > + > > > > > > > > > +\subsubsection{Device Operation: Stream parameter control} > > > > > > > > > + > > > > > > > > > +\begin{itemize*} > > > > > > > > > +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream > > > > > > > > > parameters > > > > > > > > > for + input and output streams from the device. > > > > > > > > > +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream > > > > > > > > > parameters to > > > > > > > > > the + device. > > > > > > > > > +\item After setting stream parameters, the driver may issue > > > > > > > > > + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input > > > > > > > > > and > > > > > > > > > output > > > > > > > > > can be + changed implicitly by the device during the set > > > > > > > > > operation. > > > > > > > > > +\end{itemize*} > > > > > > > > > + > > > > > > > > > +\subsubsection{Device Operation: Process buffers} > > > > > > > > > + > > > > > > > > > +\begin{itemize*} > > > > > > > > > +\item If the function and the buffer type require so, write > > > > > > > > > data to > > > > > > > > > +the buffer memory. > > > > > > > > > +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer > > > > > > > > > for > > > > > > > > > +processing in the device. > > > > > > > > > +\item The request completes asynchronously when the device > > > > > > > > > has > > > > > > > > > +finished with the buffer. > > > > > > > > > +\end{itemize*} > > > > > > > > > + > > > > > > > > > +\subsubsection{Device Operation: Buffer processing control} > > > > > > > > > + > > > > > > > > > +\begin{itemize*} > > > > > > > > > +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to > > > > > > > > > process > > > > > > > > > and > > > > > > > > > + return all of the already queued buffers. > > > > > > > > > +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to > > > > > > > > > return > > > > > > > > > back > > > > > > > > > + already queued buffers from the input or the output queue. > > > > > > > > > This > > > > > > > > > also > > > > > > > > > + includes input or output buffers that can be currently > > > > > > > > > owned by > > > > > > > > > the > > > > > > > > > + device's processing pipeline. > > > > > > > > > +\end{itemize*} > > > > > > > > > + > > > > > > > > > +\subsubsection{Device Operation: Asynchronous events} > > > > > > > > > + > > > > > > > > > +While processing buffers, the device can send asynchronous > > > > > > > > > event > > > > > > > > > +notifications to the driver. The behaviour depends on the > > > > > > > > > exact > > > > > > > > > +stream. For example, the decoder device sends a resolution > > > > > > > > > change > > > > > > > > > +event when it encounters new resolution metadata in the > > > > > > > > > stream. > > > > > > > > > + > > > > > > > > > +\subsubsection{Device Operation: Request header} > > > > > > > > > + > > > > > > > > > +All requests and responses on the control virt queue have a > > > > > > > > > fixed > > > > > > > > > +header using the following layout structure and definitions: > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +enum virtio_video_ctrl_type { > > > > > > > > > + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, > > > > > > > > > + > > > > > > > > > + /* request */ > > > > > > > > > + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, > > > > > > > > > + VIRTIO_VIDEO_T_STREAM_CREATE, > > > > > > > > > + VIRTIO_VIDEO_T_STREAM_DESTROY, > > > > > > > > > + VIRTIO_VIDEO_T_STREAM_DRAIN, > > > > > > > > > + VIRTIO_VIDEO_T_RESOURCE_CREATE, > > > > > > > > > + VIRTIO_VIDEO_T_RESOURCE_DESTROY, > > > > > > > > > + VIRTIO_VIDEO_T_RESOURCE_QUEUE, > > > > > > > > > + VIRTIO_VIDEO_T_QUEUE_CLEAR, > > > > > > > > > + VIRTIO_VIDEO_T_SET_PARAMS, > > > > > > > > > + VIRTIO_VIDEO_T_GET_PARAMS, > > > > > > > > > + > > > > > > > > > + /* response */ > > > > > > > > > + VIRTIO_VIDEO_S_OK = 0x0200, > > > > > > > > > + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, > > > > > > > > > + VIRTIO_VIDEO_S_OK_GET_PARAMS, > > > > > > > > > + > > > > > > > > > + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, > > > > > > > > > + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, > > > > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, > > > > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, > > > > > > > > > + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, > > > > > > > > > +}; > > > > > > > > > + > > > > > > > > > +struct virtio_video_ctrl_hdr { > > > > > > > > > + le32 type; > > > > > > > > > + le32 stream_id; > > > > > > > > > + le32 len; /* Length of the structure in bytes. */ > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{type}] is the type of the driver request or the > > > > > > > > > device > > > > > > > > > +response. > > > > > > > > > +\item[\field{stream_id}] specifies a target stream. > > > > > > > > > +\item[\field{len}] is the length of data in bytes, which > > > > > > > > > includes > > > > > > > > > +length of the header. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\subsubsection{Device Operation: controlq} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > + > > > > > > > > > +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information > > > > > > > > > about > > > > > > > > > +supported formats. > > > > > > > > > + > > > > > > > > > +The driver uses \field{struct virtio_video_get_capability} to > > > > > > > > > send > > > > > > > > > a > > > > > > > > > +query request. > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +enum virtio_video_buf_type { > > > > > > > > > + VIRTIO_VIDEO_BUF_TYPE_INPUT, > > > > > > > > > + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, > > > > > > > > > +}; > > > > > > > > > + > > > > > > > > > +struct virtio_video_get_capability { > > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > > + enum virtio_video_buf_type buf_type; > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{buf_type}] is the buffer type that the driver > > > > > > > > > asks > > > > > > > > > +information about. The driver MUST set either > > > > > > > > > +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or > > > > > > > > > \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} > > > > > > > > > + > > > > > > > > > +The device responds a capability by using \field{struct > > > > > > > > > +virtio_video_get_capability_resp}. > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +enum virtio_video_format { > > > > > > > > > + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, > > > > > > > > > + /* Raw formats */ > > > > > > > > > + VIRTIO_VIDEO_FORMAT_NV12 = 1, > > > > > > > > > + VIRTIO_VIDEO_FORMAT_YUV420, > > > > > > > > > + VIRTIO_VIDEO_FORMAT_YVU420, > > > > > > > > > + > > > > > > > > > + /* Compressed formats */ > > > > > > > > > + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, > > > > > > > > > + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, > > > > > > > > > + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, > > > > > > > > > +}; > > > > > > > > > + > > > > > > > > > +enum virtio_video_profile { > > > > > > > > > + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, > > > > > > > > > + > > > > > > > > > + /* H.264 */ > > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_BASELINE = > > > > > > > > > VIRTIO_VIDEO_PROFILE_H264_BASELINE, + > > > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MAIN, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_H264_MAX = > > > > > > > > > VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + > > > > > > > > > + /* VP8 */ > > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_ANY = > > > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_MIN, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP8_MAX = > > > > > > > > > VIRTIO_VIDEO_PROFILE_VP8_ANY, > > > > > > > > > + > > > > > > > > > + /* VP9 */ > > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = > > > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_MIN, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > > > > > > + VIRTIO_VIDEO_PROFILE_VP9_MAX = > > > > > > > > > VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, > > > > > > > > > +}; > > > > > > > > > + > > > > > > > > > +struct virtio_video_format_range { > > > > > > > > > + le32 min; > > > > > > > > > + le32 max; > > > > > > > > > + le32 step; > > > > > > > > > + u8 paddings[4]; > > > > > > > > > +}; > > > > > > > > > + > > > > > > > > > +struct virtio_video_format_desc { > > > > > > > > > + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types > > > > > > > > > */ > > > > > > > > > + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types > > > > > > > > > */ > > > > > > > > > + le64 mask; > > > > > > > > > + struct virtio_video_format_range width; > > > > > > > > > + struct virtio_video_format_range height; > > > > > > > > > + le32 num_rates; > > > > > > > > > + u8 padding[4]; > > > > > > > > > + /* Followed by struct virtio_video_frame_rate > > > > > > > > > frame_rates[] > > > > > > > > > */ > > > > > > > > > +}; > > > > > > > > > + > > > > > > > > > +struct virtio_video_get_capability_resp { > > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > > + le32 num_descs; > > > > > > > > > + /* Followed by struct virtio_video_format_desc desc[] > > > > > > > > > */ > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +The format description \field{struct > > > > > > > > > virtio_video_format_desc} > > > > > > > > > +includes the following fields: > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{format}] specifies an image format. The device > > > > > > > > > MUST > > > > > > > > > set > > > > > > > > > one > > > > > > > > > + of \field{enum virtio_video_format}. > > > > > > > > > +\item[\field{profile}] specifies a profile of the compressed > > > > > > > > > image > > > > > > > > > format > > > > > > > > > + specified in \field{format}. The driver SHOULD ignore this > > > > > > > > > value > > > > > > > > > if > > > > > > > > > + \field{format} is a raw format. > > > > > > > > > > > > > > > > So how should this be used? The spec does not define any way to > > > > > > > > set > > > > > > > > profile for the device. It is very important for encoder. > > > > > > > > > > > > > > Thank you for pointing this. > > > > > > > These points are overlooked, as I didn't care about encoder usage > > > > > > > enough. > > > > > > > > > > > > > > After thinking it again, I think it's not a very good idea to > > > > > > > include > > > > > > > supported profiles and levels in a struct for capability. > > > > > > > This is because these values are available only for limited number > > > > > > > of > > > > > > > formats. Also, it's true that we need to have a way to set these > > > > > > > values > > > > > > > as > > > > > > > Dmitry pointed. > > > > > > > > > > > > Yes, you are right. In fact, the approach of the v1 spec to keep > > > > > > controls > > > > > > separately was not correct. > > > > > > > > > > > > > Instead, it would make more sense to have additional three types > > > > > > > of > > > > > > > controls for profiles, levels, and bitrates: > > > > > > > (1) QUERY_CONTROL: Query values supported by the device > > > > > > > (2) GET_CONTROL: Read a value that is set in the device > > > > > > > (3) SET_CONTROL: Set a value in the device > > > > > > > > > > > > > > These operations are similar to V4L2 controls. > > > > > > > (1), (2) and (3) would correspond VIDIOC_QUERY{CTRL,MENU}, S_CTRL, > > > > > > > and > > > > > > > G_CTRL in V4L2, respectively. > > > > > > > Also, (3) would be similar to enum virtio_video_control_type in > > > > > > > the > > > > > > > virtio-video v1 driver implementation in > > > > > > > https://markmail.org/message/dwghwdqsbl3gsjxu . > > > > > > > > > > > > > > For QUERY_CONTROL, my idea is like this: > > > > > > > > > > > > > > enum virtio_video_control_type { > > > > > > > > > > > > > > VIRTIO_VIDEO_CONTROL_UNDEFINED = 0, > > > > > > > > > > > > > > VIRTIO_VIDEO_CONTROL_BITRATE = 0x100, > > > > > > > VIRTIO_VIDEO_CONTROL_PROFILE, > > > > > > > VIRTIO_VIDEO_CONTROL_LEVEL, > > > > > > > > > > > > > > }; > > > > > > > > > > > > > > struct virtio_video_query_control { > > > > > > > > > > > > > > struct virtio_video_ctrl_hdr hdr; > > > > > > > le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */ > > > > > > > le32 length; > > > > > > > /* Followed by additional data. > > > > > > > > > > > > > > * If |control| is VIRTIO_VIDEO_CONTROL_PROFILE, > > > > > > > * the device must pass a codec format like H264 or VP9. > > > > > > > * The requred data must be defined in the specification. > > > > > > > */ > > > > > > > > > > > > > > }; > > > > > > > > > > > > > > struct virtio_video_query_control_resp { > > > > > > > > > > > > > > struct virtio_video_ctrl_hdr hdr; > > > > > > > le32 length; > > > > > > > u8 padding[4]; > > > > > > > /* Followed by data corresponds to the specified control. > > > > > > > > > > > > > > * The type of data must be defined in the spec. > > > > > > > * For example, if the driver queries profiles, this part should > > > > > > > be > > > > > > > * an array of supported profiles of a given format. > > > > > > > */ > > > > > > > > > > > > > > }; > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > I think virtio_video_control_type should make sense. But I would > > > > > > disagree > > > > > > with the need to have new QUERY_CONTROL and GET_CONTROL. > > > > > > > > > > > > I assume the set of supported controls is fixed for some particular > > > > > > format > > > > > > on a given IP. So we'd propose to include controls into format > > > > > > descriptors, so we don't need to QUERY_CONTROL. This way (with > > > > > > 'virtio_video_format_list') we can define not-contiguos ranges, e.g. > > > > > > for > > > > > > profiles. > > > > > > > > > > > > struct virtio_video_format_frame { > > > > > > > > > > > > /* As proposed in Keiichi's prev email */ > > > > > > > > > > > > }; > > > > > > > > > > > > virtio_video_format_list { > > > > > > > > > > > > le32 num_entries; > > > > > > u8 padding[4]; > > > > > > /* Followed by le64 entries[] */ > > > > > > > > > > > > }; > > > > > > > > > > > > struct virtio_video_format_control { > > > > > > > > > > > > le32 type; > > > > > > u8 padding[4]; > > > > > > struct virtio_video_format_list values; > > > > > > > > > > > > }; > > > > > > > > > > > > struct virtio_video_format_desc { > > > > > > > > > > > > le64 mask; > > > > > > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > > > le32 planes_layout; /* See the thread [v2 0/1] */ > > > > > > le32 num_frames; > > > > > > le32 num_controls; > > > > > > /* Followed by struct virtio_video_format_frame frames[] */ > > > > > > /* Followed by struct virtio_video_format_control controls[] > > > > > > */ > > > > > > > > > > > > }; > > > > > > > > > > In my understanding, a set of supported levels depends on a profile. > > > > > H.264 spec says that levels are specified within each profile. > > > > > cf. "0.5 Profiles and levels" in > > > > > https://www.itu.int/rec/T-REC-H.264-201906-I/en (V4L2's QUERYMENU > > > > > doesn't > > > > > seem to provide a way to query levels for each profile properly, > > > > > though) > > > > > > > > > > So, if we want to have supported profiles and levels in format_desc as > > > > > your idea, each supported profile should have a list of supported > > > > > levels. > > > > > I suppose it makes the structure of video_format_desc too complicated. > > > > > I'd like to avoid this complexity caused by some specific formats. > > > > > > > > > > Instead, I'd like to keep virtio_video_format_desc minimal and have > > > > > QUERY_CONTROL to query format-specific values like profiles and > > > > > levels. > > > > > This design would make it easy to extend controls when we want to > > > > > support new types of formats. We will just need to define new > > > > > VIRTIO_VIDEO_CONTROL_*. > > > > > What do you think? > > > > > > > > Yes, I think we can keep it this way, this indeed provides more > > > > flexibility and this way we won't need to modify the 'format' parsing > > > > logic when more controls are added. > > > > > > Thanks! I'm preparing the next version of the patch. > > > > > > I have one concern about terminology. > > > We have two different things called "controls" now: > > > 1) Messages passed via controlq. i.e. "ctrl" in 'enum > > > virtio_video_ctrl_type' and 'virtio_video_ctrl_hdr'. > > > 2) Commands for profiles, levels, and bitrates similar to V4L2 > > > controls. e.g.., "CONTROL" in VIRTIO_VIDEO_CONTROL_* > > > > > > I feel calling both "controls" is somewhat confusing and I am > > > wondering if we can't rename either of them. > > > > > > For example, how about renaming 2) to "command"? Then, we'll have > > > {QUERY ,GET, SET}_COMMAND. > > > It may sound strange, though. > > > Do you have any idea? > > > > "Command" sound like executing an action, but in this context we're > > thinking more about parameters or, exactly, controls. The term > > "control" is not limited to V4L2 and actually used in other subsystems > > as well, for example the Android Camera HAL. > > > > How about renaming 1) to "commandq" and then having the other symbols > > use the abbreviated "cmd"? > > First I thought about renaming 'virtio_video_ctrl_hdr' to > 'virtio_video_req_hdr', but then 'reqq' wouldn't look nice. > > So I'd be ok to go with 'commandq' and 'cmd'. > > T is for type, S is for status. I was looking at the virtio-iommu spec around > that time, I think I derived it from there. Thank you for the explanation. Since virtio-video's responses can contain information more than status (e.g. capability), let's go with "CMD" and "RESP" instead of "T" and "S" like virtio_gpu_ctrl_type. Best regards, Keiichi > > Regards, > Dmitry > > > > > Best regards, > > Tomasz > > > > > Best regards, > > > Keiichi > > > > > > > Best regards, > > > > Dmitry. > > > > > > > > > Best regards, > > > > > Keiichi > > > > > > > > > > > SET_CONTROL seems to be mandatory. If it succeeds, we can store the > > > > > > current > > > > > > value locally, so there is no need to have GET_CONTROL. > > > > > > > > > > > > The only exception is the initial (default) control value in the > > > > > > device. > > > > > > But with the removal of 'function' and with addition of 'caps' > > > > > > instead, > > > > > > the way to provide defaults is gone. So I suppose for formats we'll > > > > > > be > > > > > > just using GET on driver start to get the 'defaults'. But the thing > > > > > > is > > > > > > that for formats there are other uses for GET, but for controls GET > > > > > > apparently does not make to much sense at runtime. > > > > > > > > > > > > > > Also, shouldn't the profile come together with level? Would make > > > > > > > > sense > > > > > > > > for > > > > > > > > encoders. > > > > > > > > > > > > > > Yeah. So, in the above idea of QUERY_CONTROL, profile should be > > > > > > > required when querying supported levels. > > > > > > > > > > > > So probably should be enumerated together, not queried, as per the > > > > > > comment > > > > > > above. > > > > > > > > > > > > > > > +\item[\field{mask}] is a bitset that represents the supported > > > > > > > > > + combination of input and output format. If \textit{i}-th > > > > > > > > > bit is > > > > > > > > > set > > > > > > > > > + in \field{mask} of \textit{j}-th \field{struct > > > > > > > > > + virtio_video_format_desc} for input, the device supports > > > > > > > > > encoding > > > > > > > > > or > > > > > > > > > + decoding from the \textit{j}-th input format to > > > > > > > > > \textit{i}-th > > > > > > > > > output > > > > > > > > > + format. > > > > > > > > > +\item[\field{width, height}] represents a range of > > > > > > > > > resolutions > > > > > > > > > + supported by the device. If its \field{step} is not > > > > > > > > > applicable, > > > > > > > > > its > > > > > > > > > + \field{min} is equal to its \field{max}. > > > > > > > > > +\item[\field{num_rates}] is the length of an array > > > > > > > > > \field{frame_rates}. > > > > > > > > > In > > > > > > > > > case of decoder, the driver SHOULD ignore this value. > > > > > > > > > +\item[\field{frame_rates}] is an array of supported frame > > > > > > > > > rates. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > > > > > > > > I'd guess frame rates depend on the resolution as well. This > > > > > > > > dependency > > > > > > > > was > > > > > > > > clear in the v1 spec, but in the v2 there is no dependency > > > > > > > > anymore. I > > > > > > > > think we need to update this. > > > > > > > > > > > > > > That's a good point. I missed that dependency when updating the > > > > > > > structures. > > > > > > > So, let me update the structs like the following: > > > > > > > > > > > > > > struct virtio_video_format_frame { > > > > > > > > > > > > > > struct virtio_video_format_range width; > > > > > > > struct virtio_video_format_range height; > > > > > > > le32 num_rates; > > > > > > > u8 padding[4]; > > > > > > > /* Followed by struct virtio_video_format_range > > > > > > > frame_rates[] */ > > > > > > > > > > > > > > }; > > > > > > > > > > > > > > struct virtio_video_format_desc { > > > > > > > > > > > > > > le64 mask; > > > > > > > le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ > > > > > > > le32 planes_layout; /* See the thread [v2 0/1] */ > > > > > > > le32 num_frames; > > > > > > > u8 padding[4]; > > > > > > > /* Followed by struct virtio_video_format_frame frames[] > > > > > > > */ > > > > > > > > > > > > > > }; > > > > > > > > > > > > Yes, I do agree with this approach. > > > > > > > > > > > > Best regards, > > > > > > Dmitry. > > > > > > > > > > > > > Best regards, > > > > > > > Keiichi. > > > > > > > > > > > > > > > Best regards, > > > > > > > > Dmitry. > > > > > > > > > > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream > > > > > > > > > (context) > > > > > > > > > + within the device. > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +enum virtio_video_mem_type { > > > > > > > > > + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, > > > > > > > > > +}; > > > > > > > > > + > > > > > > > > > +struct virtio_video_stream_create { > > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > > + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* > > > > > > > > > types > > > > > > > > > */ > > > > > > > > > + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* > > > > > > > > > types > > > > > > > > > */ > > > > > > > > > + char debug_name[64]; > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{in_mem_type}] is a type of buffer management for > > > > > > > > > input > > > > > > > > > +buffers. The driver MUST set a value in \field{enum > > > > > > > > > +virtio_video_mem_type}. > > > > > > > > > +\item[\field{out_mem_type}] is a type of buffer management > > > > > > > > > for > > > > > > > > > output > > > > > > > > > +buffers. The driver MUST set a value in \field{enum > > > > > > > > > +virtio_video_mem_type}. > > > > > > > > > +\item[\field{debug_name}] is a text string for a debug > > > > > > > > > purpose. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream > > > > > > > > > (context) > > > > > > > > > + within the device. > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +struct virtio_video_stream_destroy { > > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all > > > > > > > > > the > > > > > > > > > + queued buffers through the pipeline. > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +struct virtio_video_stream_drain { > > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource > > > > > > > > > descriptor > > > > > > > > > + within the device. > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +struct virtio_video_mem_entry { > > > > > > > > > + le64 addr; > > > > > > > > > + le32 length; > > > > > > > > > + u8 padding[4]; > > > > > > > > > +}; > > > > > > > > > + > > > > > > > > > +struct virtio_video_resource_create { > > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > > + le32 resource_id; > > > > > > > > > + le32 nr_entries; > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > > > > +\item[\field{nr_entries}] number of \field{struct > > > > > > > > > + virtio_video_mem_entry} memory entries. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource > > > > > > > > > descriptor > > > > > > > > > + within the device. > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +struct virtio_video_resource_destroy { > > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > > + le32 resource_id; > > > > > > > > > + u8 padding[4]; > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the > > > > > > > > > device's > > > > > > > > > +queue. > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +#define VIRTIO_VIDEO_MAX_PLANES 8 > > > > > > > > > + > > > > > > > > > +struct virtio_video_resource_queue { > > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > > + le32 buf_type; > > > > > > > > > + le32 resource_id; > > > > > > > > > + le64 timestamp; > > > > > > > > > + le32 nr_data_size; > > > > > > > > > + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{buf_type}] buf_type of the . > > > > > > > > > +\item[\field{resource_id}] internal id of the resource. > > > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that > > > > > > > > > can be > > > > > > > > > used > > > > > > > > > + for synchronisation. > > > > > > > > > +\item[\field{nr_data_size}] number of \field{data_size} > > > > > > > > > entries. > > > > > > > > > +\item[\field{data_size}] number of data bytes within a plane. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +enum virtio_video_buffer_flag { > > > > > > > > > + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, > > > > > > > > > + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, > > > > > > > > > + /* Encoder only */ > > > > > > > > > + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, > > > > > > > > > + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, > > > > > > > > > + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, > > > > > > > > > +}; > > > > > > > > > + > > > > > > > > > +struct virtio_video_resource_queue_resp { > > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > > + le64 timestamp; > > > > > > > > > + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ > > > > > > > > > + le32 size; /* Encoded size */ > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{timestamp}] an abstract sequence counter that > > > > > > > > > can be > > > > > > > > > used > > > > > > > > > + for synchronisation. > > > > > > > > > +\item[\field{flags}] mark specific buffers in the sequence. > > > > > > > > > +\item[\field{size}] data size in the buffer (encoder only). > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +The device sends a response to the queue request > > > > > > > > > asynchronously > > > > > > > > > when > > > > > > > > > +it has finished processing the buffer. > > > > > > > > > + > > > > > > > > > +The device SHOULD mark a buffer that triggered a processing > > > > > > > > > error > > > > > > > > > with > > > > > > > > > +the VIRTIO_VIDEO_BUFFER_F_ERR flag. > > > > > > > > > + > > > > > > > > > +The device MUST mark the last buffer with the > > > > > > > > > +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the > > > > > > > > > drain > > > > > > > > > +sequence. > > > > > > > > > + > > > > > > > > > +In case of encoder, to denote a particular frame type the > > > > > > > > > devie > > > > > > > > > MUST > > > > > > > > > +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, > > > > > > > > > +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. > > > > > > > > > + > > > > > > > > > +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already > > > > > > > > > queued > > > > > > > > > + buffers back from the input or the output queue of the > > > > > > > > > device. > > > > > > > > > The > > > > > > > > > + device SHOULD return all of the buffers from the respective > > > > > > > > > queue > > > > > > > > > as > > > > > > > > > + soon as possible without pushing the buffers through the > > > > > > > > > processing > > > > > > > > > + pipeline. > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +struct virtio_video_queue_clear { > > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > > + le32 buf_type; > > > > > > > > > + u8 padding[4]; > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{buf_type}] buffer type. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input > > > > > > > > > or the > > > > > > > > > + output of a stream. > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +struct virtio_video_plane_format { > > > > > > > > > + le32 plane_size; > > > > > > > > > + le32 stride; > > > > > > > > > + u8 padding[4]; > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{plane_size}] size of the plane in bytes. > > > > > > > > > +\item[\field{stride}] stride used for the plane in bytes. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +struct virtio_video_params { > > > > > > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* > > > > > > > > > types */ > > > > > > > > > + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types > > > > > > > > > */ > > > > > > > > > + le32 frame_width; > > > > > > > > > + le32 frame_height; > > > > > > > > > + le32 min_buffers; > > > > > > > > > + le32 max_buffers; > > > > > > > > > + le32 frame_rate; > > > > > > > > > + struct virtio_video_crop { > > > > > > > > > + le32 left; > > > > > > > > > + le32 top; > > > > > > > > > + le32 width; > > > > > > > > > + le32 height; > > > > > > > > > + } crop; > > > > > > > > > + le32 num_planes; > > > > > > > > > + struct virtio_video_plane_format > > > > > > > > > plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{frame_width}] the value to get/set. > > > > > > > > > +\item[\field{frame_height}] the value to get/set. > > > > > > > > > +\item[\field{pixel_format}] the value to get/set. > > > > > > > > > +\item[\field{min_buffers}] minimum buffers required to handle > > > > > > > > > the > > > > > > > > > + format (r/o). > > > > > > > > > +\item[\field{max_buffers}] maximum buffers required to handle > > > > > > > > > the > > > > > > > > > + format (r/o). > > > > > > > > > +\item[\field{frame_rate}] the value to get/set. > > > > > > > > > +\item[\field{crop}] cropping (composing) rectangle. > > > > > > > > > +\item[\field{num_planes}] number of planes used to store > > > > > > > > > pixel data > > > > > > > > > +(r/o). > > > > > > > > > +\item[\field{plane_formats}] description of each plane. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +struct virtio_video_get_params { > > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > > + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* > > > > > > > > > types */ > > > > > > > > > +}; > > > > > > > > > + > > > > > > > > > +struct virtio_video_get_params_resp { > > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > > + struct virtio_video_params params; > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{buf_type}] buffer type. > > > > > > > > > +\item[\field{params}] parameter values. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a > > > > > > > > > stream. > > > > > > > > > + > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +struct virtio_video_set_params { > > > > > > > > > + struct virtio_video_ctrl_hdr hdr; > > > > > > > > > + struct virtio_video_params params; > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{params}] parameters to set. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +Setting stream parameters might have side effects within the > > > > > > > > > device. > > > > > > > > > +For example, the device MAY perform alignment of width and > > > > > > > > > height, > > > > > > > > > +change the number of planes it uses for the format, or do > > > > > > > > > whatever > > > > > > > > > +changes that are required to continue normal operation using > > > > > > > > > the > > > > > > > > > +updated parameters. It is up to the driver to check the > > > > > > > > > parameter > > > > > > > > > set > > > > > > > > > +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. > > > > > > > > > + > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +\subsubsection{Device Operation: eventq} > > > > > > > > > + > > > > > > > > > +The device can report events on the event queue. The driver > > > > > > > > > initially > > > > > > > > > +populates the queue with device-writeable buffers. When the > > > > > > > > > device > > > > > > > > > +needs to report an event, it fills a buffer and notifies the > > > > > > > > > driver. > > > > > > > > > +The driver consumes the report and adds a new buffer to the > > > > > > > > > virtqueue. > > > > > > > > > + > > > > > > > > > +\begin{lstlisting} > > > > > > > > > +enum virtio_video_event_type { > > > > > > > > > + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, > > > > > > > > > + /* For all functions */ > > > > > > > > > + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, > > > > > > > > > + /* For decoder only */ > > > > > > > > > + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = > > > > > > > > > 0x0200, > > > > > > > > > +}; > > > > > > > > > + > > > > > > > > > +struct virtio_video_event { > > > > > > > > > + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* > > > > > > > > > types */ > > > > > > > > > + le32 stream_id; > > > > > > > > > + u8 padding[4]; > > > > > > > > > +}; > > > > > > > > > +\end{lstlisting} > > > > > > > > > + > > > > > > > > > +\begin{description} > > > > > > > > > +\item[\field{event_type}] type of the triggered event . > > > > > > > > > +\item[\field{stream_id}] id of the source stream. > > > > > > > > > +\end{description} > > > > > > > > > + > > > > > > > > > +The device MUST send > > > > > > > > > VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED > > > > > > > > > +whenever it encounters new resolution data in the stream. > > > > > > > > > This > > > > > > > > > +includes the case of the initial device configuration after > > > > > > > > > metadata > > > > > > > > > +has been parsed and the case of dynamic resolution change. > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org >
Hi Gerd, Thank you so much for the review. I'm sorry for not replying earlier. On Thu, Dec 19, 2019 at 10:02 PM Gerd Hoffmann <kraxel@redhat.com> wrote: > > Hi, > > > > Not clearly defined in the spec: When is the decoder supposed to send > > > the response for a queue request? When it finished decoding (i.e. frame > > > is ready for playback), or when it doesn't need the buffer any more for > > > decoding (i.e. buffer can be re-queued or pages can be released)? The answer is "when it doesn't need the buffer any more for decoding". The device can access buffer contents from when a queue request is sent until the device responds it. So, the device must not responds a queue request before finishing all process that requires the buffer content. Actually, the first one "When it finished decoding (i.e. frame is ready for playback)" doesn't make much sense, as it's not necessary to have a one-to-one correspondence between an input bitstream buffer and a decoded frame. It's okay to decode one input buffer contains bitstream data for two frames. Also, a user can pass bitstream for one frame as two input buffers. I'll document it in the spec. Best regards, Keiichi > > In my eyes the both statements mean almost the same and both are valid. > > Well, no. When the device decoded a P-Frame it can notify the device, > saying "here is your decoded frame". But the device might still need > the buffer with the decoded frame to properly decode the following B/I > Frames which reference the P-Frame. > > cheers, > Gerd >
Hi Gerd, On Thu, Dec 19, 2019 at 10:12 PM Gerd Hoffmann <kraxel@redhat.com> wrote: > > Hi, > > > > However that still doesn't let the driver know which buffers will be > > > dequeued when. A simple example of this scenario is when the guest is > > > done displaying a frame and requeues the buffer back to the decoder. > > > Then the decoder will not choose it for decoding next frames into as > > > long as the frame in that buffer is still used as a reference frame, > > > even if one sends the drain request. > > It might be that I'm getting your point wrong, but do you mean some hardware > > can mark a buffer as ready to be displayed yet still using the underlying > > memory to decode other frames? > > Yes, this is how I understand Tomasz Figa. > > > This means, if you occasionally/intentionally > > write to the buffer you mess up the whole decoding pipeline. > > And to avoid this the buffer handling aspect must be clarified in the > specification. Is the device allowed to continue using the buffer after > finishing decoding and completing the queue request? If so, how do we > hand over buffer ownership back to the driver so it can free the pages? > drain request? How do we handle re-using buffers? Can the driver > simply re-queue them and expect the device figures by itself whenever it > can use the buffer or whenever it is still needed as reference frame? The device shouldn't be able to access buffers after it completes a queue request. The device can touch buffer contents from when a queue request is sent until the device responds it. In contrast, the driver must not modify buffer contents in that period. Regarding re-using, the driver can simply re-queue buffers returned by the device. If the device needs a buffer as reference frame, it must not return the buffer. I'll describe this rule in the next version of the patch. Best regards, Keiichi > > cheers, > Gerd >
Hi, > Regarding re-using, the driver can simply re-queue buffers returned by > the device. If the device needs a buffer as reference frame, it must > not return the buffer. Ok, that'll work. > I'll describe this rule in the next version of the patch. Good. You should also add a note about ordering. If the device returns the buffers as soon as they are no longer needed for decoding they might be completed out-of-order, specifically B-Frames might complete before the reference I/P frame. Is the device allowed to do that or should it complete the buffers in playback order? cheers, Gerd
On Wed, Jan 8, 2020 at 10:52 PM Keiichi Watanabe <keiichiw@chromium.org> wrote: > > Hi Gerd, > > On Thu, Dec 19, 2019 at 10:12 PM Gerd Hoffmann <kraxel@redhat.com> wrote: > > > > Hi, > > > > > > However that still doesn't let the driver know which buffers will be > > > > dequeued when. A simple example of this scenario is when the guest is > > > > done displaying a frame and requeues the buffer back to the decoder. > > > > Then the decoder will not choose it for decoding next frames into as > > > > long as the frame in that buffer is still used as a reference frame, > > > > even if one sends the drain request. > > > It might be that I'm getting your point wrong, but do you mean some hardware > > > can mark a buffer as ready to be displayed yet still using the underlying > > > memory to decode other frames? > > > > Yes, this is how I understand Tomasz Figa. > > > > > This means, if you occasionally/intentionally > > > write to the buffer you mess up the whole decoding pipeline. > > > > And to avoid this the buffer handling aspect must be clarified in the > > specification. Is the device allowed to continue using the buffer after > > finishing decoding and completing the queue request? If so, how do we > > hand over buffer ownership back to the driver so it can free the pages? > > drain request? How do we handle re-using buffers? Can the driver > > simply re-queue them and expect the device figures by itself whenever it > > can use the buffer or whenever it is still needed as reference frame? > > The device shouldn't be able to access buffers after it completes a > queue request. > The device can touch buffer contents from when a queue request is sent > until the device responds it. > In contrast, the driver must not modify buffer contents in that period. > > Regarding re-using, the driver can simply re-queue buffers returned by > the device. If the device needs a buffer as reference frame, it must > not return the buffer. I think that might not be what we expect. We want the decoder to return a decoded frame as soon as possible, but that decoded frame may be also needed for decoding next frames. In V4L2 stateful decoder, the API is defined that the client must not modify the decoded framebuffer, otherwise decoding next frames may not be correct. We may need something similar, with an explicit operation that makes the buffers not accessible to the host anymore. I think the queue flush operation could work as such. > I'll describe this rule in the next version of the patch. > > Best regards, > Keiichi > > > > > cheers, > > Gerd > >
Hi, On Thu, Jan 9, 2020 at 11:21 PM Tomasz Figa <tfiga@chromium.org> wrote: > > On Wed, Jan 8, 2020 at 10:52 PM Keiichi Watanabe <keiichiw@chromium.org> wrote: > > > > Hi Gerd, > > > > On Thu, Dec 19, 2019 at 10:12 PM Gerd Hoffmann <kraxel@redhat.com> wrote: > > > > > > Hi, > > > > > > > > However that still doesn't let the driver know which buffers will be > > > > > dequeued when. A simple example of this scenario is when the guest is > > > > > done displaying a frame and requeues the buffer back to the decoder. > > > > > Then the decoder will not choose it for decoding next frames into as > > > > > long as the frame in that buffer is still used as a reference frame, > > > > > even if one sends the drain request. > > > > It might be that I'm getting your point wrong, but do you mean some hardware > > > > can mark a buffer as ready to be displayed yet still using the underlying > > > > memory to decode other frames? > > > > > > Yes, this is how I understand Tomasz Figa. > > > > > > > This means, if you occasionally/intentionally > > > > write to the buffer you mess up the whole decoding pipeline. > > > > > > And to avoid this the buffer handling aspect must be clarified in the > > > specification. Is the device allowed to continue using the buffer after > > > finishing decoding and completing the queue request? If so, how do we > > > hand over buffer ownership back to the driver so it can free the pages? > > > drain request? How do we handle re-using buffers? Can the driver > > > simply re-queue them and expect the device figures by itself whenever it > > > can use the buffer or whenever it is still needed as reference frame? > > > > The device shouldn't be able to access buffers after it completes a > > queue request. > > The device can touch buffer contents from when a queue request is sent > > until the device responds it. > > In contrast, the driver must not modify buffer contents in that period. > > > > Regarding re-using, the driver can simply re-queue buffers returned by > > the device. If the device needs a buffer as reference frame, it must > > not return the buffer. > > I think that might not be what we expect. We want the decoder to > return a decoded frame as soon as possible, but that decoded frame may > be also needed for decoding next frames. In V4L2 stateful decoder, the > API is defined that the client must not modify the decoded > framebuffer, otherwise decoding next frames may not be correct. Thanks Tomasz! I didn't know the V4L2's convention. So, the host should be able to read a frame buffer after it is returned by responding RESOURCE_QUEUE command. > We may > need something similar, with an explicit operation that makes the > buffers not accessible to the host anymore. I think the queue flush > operation could work as such. After offline discussion with Tomasz, I suspect the queue flush operation (= VIRTIO_VIDEO_CMD_QUEUE_CLEAR) shouldn't work so, as it just forces pending QUEUE requests to be backed for video seek. So, a buffer can be readable from the device once it's queued until STREAM_DESTROY or RESOURCE_DESTROY is called. In my understanding, the life cycle of video buffers is defined as this state machine. https://drive.google.com/file/d/1c6oY5S_9bhpJlrOt4UfoQex0CanpG-kZ/view?usp=sharing ``` # The definition of the state machine in DOT language digraph G { graph [ rankdir = LR, layout = dot ]; init [shape=point] created [label="created", shape=circle] dequeued [label="dequeued", shape=circle] queued [label="queued", shape=circle] init -> created [label="RESOURCE_CREATE"] created -> queued [label="RESOURCE_QUEUE \n is sent"] dequeued -> queued [label="RESOURCE_QUEUE \n is sent"] queued -> dequeued [label="RESOURCE_QUEUE \n is returned"] created -> init [label="DESTROY"] dequeued -> init [label="DESTROY"] } ``` In each state of this figure, the accessibility of each buffer should be like the following: # Input buffers state | Guest | Host ----------------------------------- created | Read/Write | - queued | - | Read dequeued | Read/Write | - # Output buffers state | Guest | Host ---------------------------------- created | Read | - queued | - | Read/Write dequeued | Read | Read Does it make sense? If ok, I'll have this state machine and tables in the next version of spec to describe device/driver requirements by adding a subsection about buffer life cycle. By the way, regarding destruction of buffers, I suspect it doesn't make much sense to have RESOURCE_DESTROY command. Though it was suggested as a per-buffer command, it doesn't match the existing V4L2 API, as REQBUFS(0) destroys all buffers at once. I guess per-buffer destruction would require hardware or firmware supports. Even if we want to destroy buffers at once, we can destroy the stream and recreate one. So, I wonder if we can remove RESOURCE_DESTROY command from the first version of spec at least. What do you think? Best regards, Keiichi > > > I'll describe this rule in the next version of the patch. > > > > Best regards, > > Keiichi > > > > > > > > cheers, > > > Gerd > > >
Hi Keiichi, thank you for the update. On Dienstag, 14. Januar 2020 08:18:50 CET Keiichi Watanabe wrote: > Hi, > > On Thu, Jan 9, 2020 at 11:21 PM Tomasz Figa <tfiga@chromium.org> wrote: > > On Wed, Jan 8, 2020 at 10:52 PM Keiichi Watanabe <keiichiw@chromium.org> wrote: > > > Hi Gerd, > > > > > > On Thu, Dec 19, 2019 at 10:12 PM Gerd Hoffmann <kraxel@redhat.com> wrote: > > > > Hi, > > > > > > > > > > However that still doesn't let the driver know which buffers will > > > > > > be > > > > > > dequeued when. A simple example of this scenario is when the guest > > > > > > is > > > > > > done displaying a frame and requeues the buffer back to the > > > > > > decoder. > > > > > > Then the decoder will not choose it for decoding next frames into > > > > > > as > > > > > > long as the frame in that buffer is still used as a reference > > > > > > frame, > > > > > > even if one sends the drain request. > > > > > > > > > > It might be that I'm getting your point wrong, but do you mean some > > > > > hardware can mark a buffer as ready to be displayed yet still using > > > > > the underlying memory to decode other frames? > > > > > > > > Yes, this is how I understand Tomasz Figa. > > > > > > > > > This means, if you occasionally/intentionally > > > > > write to the buffer you mess up the whole decoding pipeline. > > > > > > > > And to avoid this the buffer handling aspect must be clarified in the > > > > specification. Is the device allowed to continue using the buffer > > > > after > > > > finishing decoding and completing the queue request? If so, how do we > > > > hand over buffer ownership back to the driver so it can free the > > > > pages? > > > > drain request? How do we handle re-using buffers? Can the driver > > > > simply re-queue them and expect the device figures by itself whenever > > > > it > > > > can use the buffer or whenever it is still needed as reference frame? > > > > > > The device shouldn't be able to access buffers after it completes a > > > queue request. > > > The device can touch buffer contents from when a queue request is sent > > > until the device responds it. > > > In contrast, the driver must not modify buffer contents in that period. > > > > > > Regarding re-using, the driver can simply re-queue buffers returned by > > > the device. If the device needs a buffer as reference frame, it must > > > not return the buffer. > > > > I think that might not be what we expect. We want the decoder to > > return a decoded frame as soon as possible, but that decoded frame may > > be also needed for decoding next frames. In V4L2 stateful decoder, the > > API is defined that the client must not modify the decoded > > framebuffer, otherwise decoding next frames may not be correct. > > Thanks Tomasz! I didn't know the V4L2's convention. > So, the host should be able to read a frame buffer after it is > returned by responding RESOURCE_QUEUE command. > > > We may > > need something similar, with an explicit operation that makes the > > buffers not accessible to the host anymore. I think the queue flush > > operation could work as such. > > After offline discussion with Tomasz, I suspect the queue flush > operation (= VIRTIO_VIDEO_CMD_QUEUE_CLEAR) shouldn't work so, as it > just forces pending QUEUE requests to be backed for video seek. > So, a buffer can be readable from the device once it's queued until > STREAM_DESTROY or RESOURCE_DESTROY is called. Speaking of v4l2, drivers usually get all buffers back on stop_streaming (this means seek(decoder), reset (encoder)). As per my understanding, this means that the device should not read the buffers anymore after stop_streaming(STREAMOFF) has been called. We can mention that after VIRTIO_VIDEO_CMD_QUEUE_CLEAR no device access is allowed. So: stop_streaming() = VIRTIO_VIDEO_CMD_QUEUE_CLEAR REQBUFS(0) = RESOURCE_DESTROY > > In my understanding, the life cycle of video buffers is defined as > this state machine. > https://drive.google.com/file/d/1c6oY5S_9bhpJlrOt4UfoQex0CanpG-kZ/view?usp=s > haring > > ``` > # The definition of the state machine in DOT language > digraph G { > graph [ rankdir = LR, layout = dot ]; > > init [shape=point] > created [label="created", shape=circle] > dequeued [label="dequeued", shape=circle] > queued [label="queued", shape=circle] > > init -> created [label="RESOURCE_CREATE"] > > created -> queued [label="RESOURCE_QUEUE \n is sent"] > dequeued -> queued [label="RESOURCE_QUEUE \n is sent"] > queued -> dequeued [label="RESOURCE_QUEUE \n is returned"] > > created -> init [label="DESTROY"] > dequeued -> init [label="DESTROY"] > } > ``` > > In each state of this figure, the accessibility of each buffer should > be like the following: > > # Input buffers > state | Guest | Host > ----------------------------------- > created | Read/Write | - > queued | - | Read > dequeued | Read/Write | - > > # Output buffers > state | Guest | Host > ---------------------------------- > created | Read | - > queued | - | Read/Write > dequeued | Read | Read > > Does it make sense? > If ok, I'll have this state machine and tables in the next version of > spec to describe device/driver requirements by adding a subsection > about buffer life cycle. > Yes, I think this is ok. > > By the way, regarding destruction of buffers, I suspect it doesn't > make much sense to have RESOURCE_DESTROY command. Though it was > suggested as a per-buffer command, it doesn't match the existing V4L2 > API, as REQBUFS(0) destroys all buffers at once. I guess per-buffer > destruction would require hardware or firmware supports. > Even if we want to destroy buffers at once, we can destroy the stream > and recreate one. So, I wonder if we can remove RESOURCE_DESTROY > command from the first version of spec at least. > What do you think? Stream might have a context behind it (and it in fact does), and the contents of that might depend on the device side implementation. For instance, the context can be used to keep the currently set parameters, controls and so on. So we'd avoid destroyng streams all the time for seek, resolution change, reset and so on. It still makes sense to use RESOURCE_DESTROY to destroy all resources at once for one particular queue though. We can rename it to something more meaningful. Best regards, Dmitry. > > Best regards, > Keiichi > > > > I'll describe this rule in the next version of the patch. > > > > > > Best regards, > > > Keiichi > > > > > > > cheers, > > > > > > > > Gerd
Hi, On Tue, Jan 14, 2020 at 7:35 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi Keiichi, > > thank you for the update. > > On Dienstag, 14. Januar 2020 08:18:50 CET Keiichi Watanabe wrote: > > Hi, > > > > On Thu, Jan 9, 2020 at 11:21 PM Tomasz Figa <tfiga@chromium.org> wrote: > > > On Wed, Jan 8, 2020 at 10:52 PM Keiichi Watanabe <keiichiw@chromium.org> > wrote: > > > > Hi Gerd, > > > > > > > > On Thu, Dec 19, 2019 at 10:12 PM Gerd Hoffmann <kraxel@redhat.com> wrote: > > > > > Hi, > > > > > > > > > > > > However that still doesn't let the driver know which buffers will > > > > > > > be > > > > > > > dequeued when. A simple example of this scenario is when the guest > > > > > > > is > > > > > > > done displaying a frame and requeues the buffer back to the > > > > > > > decoder. > > > > > > > Then the decoder will not choose it for decoding next frames into > > > > > > > as > > > > > > > long as the frame in that buffer is still used as a reference > > > > > > > frame, > > > > > > > even if one sends the drain request. > > > > > > > > > > > > It might be that I'm getting your point wrong, but do you mean some > > > > > > hardware can mark a buffer as ready to be displayed yet still using > > > > > > the underlying memory to decode other frames? > > > > > > > > > > Yes, this is how I understand Tomasz Figa. > > > > > > > > > > > This means, if you occasionally/intentionally > > > > > > write to the buffer you mess up the whole decoding pipeline. > > > > > > > > > > And to avoid this the buffer handling aspect must be clarified in the > > > > > specification. Is the device allowed to continue using the buffer > > > > > after > > > > > finishing decoding and completing the queue request? If so, how do we > > > > > hand over buffer ownership back to the driver so it can free the > > > > > pages? > > > > > drain request? How do we handle re-using buffers? Can the driver > > > > > simply re-queue them and expect the device figures by itself whenever > > > > > it > > > > > can use the buffer or whenever it is still needed as reference frame? > > > > > > > > The device shouldn't be able to access buffers after it completes a > > > > queue request. > > > > The device can touch buffer contents from when a queue request is sent > > > > until the device responds it. > > > > In contrast, the driver must not modify buffer contents in that period. > > > > > > > > Regarding re-using, the driver can simply re-queue buffers returned by > > > > the device. If the device needs a buffer as reference frame, it must > > > > not return the buffer. > > > > > > I think that might not be what we expect. We want the decoder to > > > return a decoded frame as soon as possible, but that decoded frame may > > > be also needed for decoding next frames. In V4L2 stateful decoder, the > > > API is defined that the client must not modify the decoded > > > framebuffer, otherwise decoding next frames may not be correct. > > > > Thanks Tomasz! I didn't know the V4L2's convention. > > So, the host should be able to read a frame buffer after it is > > returned by responding RESOURCE_QUEUE command. > > > > > We may > > > need something similar, with an explicit operation that makes the > > > buffers not accessible to the host anymore. I think the queue flush > > > operation could work as such. > > > > After offline discussion with Tomasz, I suspect the queue flush > > operation (= VIRTIO_VIDEO_CMD_QUEUE_CLEAR) shouldn't work so, as it > > just forces pending QUEUE requests to be backed for video seek. > > So, a buffer can be readable from the device once it's queued until > > STREAM_DESTROY or RESOURCE_DESTROY is called. > > Speaking of v4l2, drivers usually get all buffers back on stop_streaming (this > means seek(decoder), reset (encoder)). As per my understanding, this means > that the device should not read the buffers anymore after > stop_streaming(STREAMOFF) has been called. We can mention that after > VIRTIO_VIDEO_CMD_QUEUE_CLEAR no device access is allowed. > > So: > stop_streaming() = VIRTIO_VIDEO_CMD_QUEUE_CLEAR > REQBUFS(0) = RESOURCE_DESTROY Thank you for the explanation. It seems that the problem here is: "Should we _guarantee_ that the device won't read buffers after VIRTIO_VIDEO_CMD_QUEUE_CLEAR is called in the spec?" If we want to guarantee it and choose the way you suggested, the state machine and the table I posted were not enough to explain who can read/write which buffers. I suspect we would need to define states of streams like "on" and "off" in the spec. Then, VIRTIO_VIDEO_CMD_QUEUE_CLEAR will (1) force every pending command to being backed, and (2) change the stream state to "off". As a result, the table will become like this: state | Guest | Host ------------------------------------------- created | Read | - queued | - | Read/Write dequeued (stream: on) | Read | Read dequeued (stream: off)| Read | - However, I doubt we really want to guarantee it going so far as introducing such more complexity. If we don't need to guarantee it, "on" and "off" of streams are simply the V4L2 driver's internal state and the spec won't care about it. Moreover, even if we realize we actually want to guarantee it and define "stream on" and "stream off" in the spec, we can define a separate command that only changes the stream state. In other words, the command only does (2) above. So, I'd suggest: * we don't guarantee that the device won't read buffers after VIRTIO_VIDEO_CMD_QUEUE_CLEAR is called in the spec. * If we realize that it's worth having a way to prevent buffers from being read by the device in the future, we will define a concept of "stream_on/off" in the spec and add a new command to change the stream state. Does it make sense? > > > > > In my understanding, the life cycle of video buffers is defined as > > this state machine. > > https://drive.google.com/file/d/1c6oY5S_9bhpJlrOt4UfoQex0CanpG-kZ/view?usp=s > > haring > > > > ``` > > # The definition of the state machine in DOT language > > digraph G { > > graph [ rankdir = LR, layout = dot ]; > > > > init [shape=point] > > created [label="created", shape=circle] > > dequeued [label="dequeued", shape=circle] > > queued [label="queued", shape=circle] > > > > init -> created [label="RESOURCE_CREATE"] > > > > created -> queued [label="RESOURCE_QUEUE \n is sent"] > > dequeued -> queued [label="RESOURCE_QUEUE \n is sent"] > > queued -> dequeued [label="RESOURCE_QUEUE \n is returned"] > > > > created -> init [label="DESTROY"] > > dequeued -> init [label="DESTROY"] > > } > > ``` > > > > In each state of this figure, the accessibility of each buffer should > > be like the following: > > > > # Input buffers > > state | Guest | Host > > ----------------------------------- > > created | Read/Write | - > > queued | - | Read > > dequeued | Read/Write | - > > > > # Output buffers > > state | Guest | Host > > ---------------------------------- > > created | Read | - > > queued | - | Read/Write > > dequeued | Read | Read > > > > Does it make sense? > > If ok, I'll have this state machine and tables in the next version of > > spec to describe device/driver requirements by adding a subsection > > about buffer life cycle. > > > > Yes, I think this is ok. > > > > > By the way, regarding destruction of buffers, I suspect it doesn't > > make much sense to have RESOURCE_DESTROY command. Though it was > > suggested as a per-buffer command, it doesn't match the existing V4L2 > > API, as REQBUFS(0) destroys all buffers at once. I guess per-buffer > > destruction would require hardware or firmware supports. > > Even if we want to destroy buffers at once, we can destroy the stream > > and recreate one. So, I wonder if we can remove RESOURCE_DESTROY > > command from the first version of spec at least. > > What do you think? > > Stream might have a context behind it (and it in fact does), and the contents > of that might depend on the device side implementation. For instance, the > context can be used to keep the currently set parameters, controls and so on. > So we'd avoid destroyng streams all the time for seek, resolution change, > reset and so on. > > It still makes sense to use RESOURCE_DESTROY to destroy all resources at once > for one particular queue though. We can rename it to something more > meaningful. > Having REQBUFS(0) = RESOURCE_DESTROY makes sense. So far, every CMD_RESOURCE_* commands only affect one specified resource. To follow this naming convention, how about renaming it to something like "CMD_STREAM_RESOUCES_DESTROY"? Any other ideas are welcome. Best regards, Keiichi > Best regards, > Dmitry. > > > > > Best regards, > > Keiichi > > > > > > I'll describe this rule in the next version of the patch. > > > > > > > > Best regards, > > > > Keiichi > > > > > > > > > cheers, > > > > > > > > > > Gerd > >
On Tue, Jan 14, 2020 at 7:35 PM Dmitry Sepp <dmitry.sepp@opensynergy.com> wrote: > > Hi Keiichi, > > thank you for the update. > > On Dienstag, 14. Januar 2020 08:18:50 CET Keiichi Watanabe wrote: > > Hi, > > > > On Thu, Jan 9, 2020 at 11:21 PM Tomasz Figa <tfiga@chromium.org> wrote: > > > On Wed, Jan 8, 2020 at 10:52 PM Keiichi Watanabe <keiichiw@chromium.org> > wrote: > > > > Hi Gerd, > > > > > > > > On Thu, Dec 19, 2019 at 10:12 PM Gerd Hoffmann <kraxel@redhat.com> wrote: > > > > > Hi, > > > > > > > > > > > > However that still doesn't let the driver know which buffers will > > > > > > > be > > > > > > > dequeued when. A simple example of this scenario is when the guest > > > > > > > is > > > > > > > done displaying a frame and requeues the buffer back to the > > > > > > > decoder. > > > > > > > Then the decoder will not choose it for decoding next frames into > > > > > > > as > > > > > > > long as the frame in that buffer is still used as a reference > > > > > > > frame, > > > > > > > even if one sends the drain request. > > > > > > > > > > > > It might be that I'm getting your point wrong, but do you mean some > > > > > > hardware can mark a buffer as ready to be displayed yet still using > > > > > > the underlying memory to decode other frames? > > > > > > > > > > Yes, this is how I understand Tomasz Figa. > > > > > > > > > > > This means, if you occasionally/intentionally > > > > > > write to the buffer you mess up the whole decoding pipeline. > > > > > > > > > > And to avoid this the buffer handling aspect must be clarified in the > > > > > specification. Is the device allowed to continue using the buffer > > > > > after > > > > > finishing decoding and completing the queue request? If so, how do we > > > > > hand over buffer ownership back to the driver so it can free the > > > > > pages? > > > > > drain request? How do we handle re-using buffers? Can the driver > > > > > simply re-queue them and expect the device figures by itself whenever > > > > > it > > > > > can use the buffer or whenever it is still needed as reference frame? > > > > > > > > The device shouldn't be able to access buffers after it completes a > > > > queue request. > > > > The device can touch buffer contents from when a queue request is sent > > > > until the device responds it. > > > > In contrast, the driver must not modify buffer contents in that period. > > > > > > > > Regarding re-using, the driver can simply re-queue buffers returned by > > > > the device. If the device needs a buffer as reference frame, it must > > > > not return the buffer. > > > > > > I think that might not be what we expect. We want the decoder to > > > return a decoded frame as soon as possible, but that decoded frame may > > > be also needed for decoding next frames. In V4L2 stateful decoder, the > > > API is defined that the client must not modify the decoded > > > framebuffer, otherwise decoding next frames may not be correct. > > > > Thanks Tomasz! I didn't know the V4L2's convention. > > So, the host should be able to read a frame buffer after it is > > returned by responding RESOURCE_QUEUE command. > > > > > We may > > > need something similar, with an explicit operation that makes the > > > buffers not accessible to the host anymore. I think the queue flush > > > operation could work as such. > > > > After offline discussion with Tomasz, I suspect the queue flush > > operation (= VIRTIO_VIDEO_CMD_QUEUE_CLEAR) shouldn't work so, as it > > just forces pending QUEUE requests to be backed for video seek. > > So, a buffer can be readable from the device once it's queued until > > STREAM_DESTROY or RESOURCE_DESTROY is called. > > Speaking of v4l2, drivers usually get all buffers back on stop_streaming (this > means seek(decoder), reset (encoder)). As per my understanding, this means > that the device should not read the buffers anymore after > stop_streaming(STREAMOFF) has been called. We can mention that after > VIRTIO_VIDEO_CMD_QUEUE_CLEAR no device access is allowed. > > So: > stop_streaming() = VIRTIO_VIDEO_CMD_QUEUE_CLEAR > REQBUFS(0) = RESOURCE_DESTROY > I'm afraid this may not be enough. "drivers usually get all buffers back on stop_streaming" is more like executing DQBUF implicitly on all queued buffers. Unfortunately it doesn't necessarily mean releasing the buffers. AFAICT currently the only way to guarantee that the stateful V4L2 decoder on the host wouldn't use the buffers anymore is to call REQBUFS(CAPTURE, 0). Best regards, Tomasz > > > > In my understanding, the life cycle of video buffers is defined as > > this state machine. > > https://drive.google.com/file/d/1c6oY5S_9bhpJlrOt4UfoQex0CanpG-kZ/view?usp=s > > haring > > > > ``` > > # The definition of the state machine in DOT language > > digraph G { > > graph [ rankdir = LR, layout = dot ]; > > > > init [shape=point] > > created [label="created", shape=circle] > > dequeued [label="dequeued", shape=circle] > > queued [label="queued", shape=circle] > > > > init -> created [label="RESOURCE_CREATE"] > > > > created -> queued [label="RESOURCE_QUEUE \n is sent"] > > dequeued -> queued [label="RESOURCE_QUEUE \n is sent"] > > queued -> dequeued [label="RESOURCE_QUEUE \n is returned"] > > > > created -> init [label="DESTROY"] > > dequeued -> init [label="DESTROY"] > > } > > ``` > > > > In each state of this figure, the accessibility of each buffer should > > be like the following: > > > > # Input buffers > > state | Guest | Host > > ----------------------------------- > > created | Read/Write | - > > queued | - | Read > > dequeued | Read/Write | - > > > > # Output buffers > > state | Guest | Host > > ---------------------------------- > > created | Read | - > > queued | - | Read/Write > > dequeued | Read | Read > > > > Does it make sense? > > If ok, I'll have this state machine and tables in the next version of > > spec to describe device/driver requirements by adding a subsection > > about buffer life cycle. > > > > Yes, I think this is ok. > > > > > By the way, regarding destruction of buffers, I suspect it doesn't > > make much sense to have RESOURCE_DESTROY command. Though it was > > suggested as a per-buffer command, it doesn't match the existing V4L2 > > API, as REQBUFS(0) destroys all buffers at once. I guess per-buffer > > destruction would require hardware or firmware supports. > > Even if we want to destroy buffers at once, we can destroy the stream > > and recreate one. So, I wonder if we can remove RESOURCE_DESTROY > > command from the first version of spec at least. > > What do you think? > > Stream might have a context behind it (and it in fact does), and the contents > of that might depend on the device side implementation. For instance, the > context can be used to keep the currently set parameters, controls and so on. > So we'd avoid destroyng streams all the time for seek, resolution change, > reset and so on. > > It still makes sense to use RESOURCE_DESTROY to destroy all resources at once > for one particular queue though. We can rename it to something more > meaningful. > > Best regards, > Dmitry. > > > > > Best regards, > > Keiichi > > > > > > I'll describe this rule in the next version of the patch. > > > > > > > > Best regards, > > > > Keiichi > > > > > > > > > cheers, > > > > > > > > > > Gerd > >
diff --git a/content.tex b/content.tex index 556b373..9e56839 100644 --- a/content.tex +++ b/content.tex @@ -5743,6 +5743,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device \input{virtio-vsock.tex} \input{virtio-fs.tex} \input{virtio-rpmb.tex} +\input{virtio-video.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} diff --git a/virtio-video.tex b/virtio-video.tex new file mode 100644 index 0000000..30e728d --- /dev/null +++ b/virtio-video.tex @@ -0,0 +1,579 @@ +\section{Video Device}\label{sec:Device Types / Video Device} + +The virtio video encoder device and decoder device are virtual devices that +supports encoding and decoding respectively. Though the encoder and the decoder +are different devices, they use the same protocol. + +\subsection{Device ID}\label{sec:Device Types / Video Device / Device ID} + +\begin{description} +\item[30] encoder device +\item[31] decoder device +\end{description} + +\subsection{Virtqueues}\label{sec:Device Types / Video Device / Virtqueues} + +\begin{description} +\item[0] controlq - queue for sending control commands. +\item[1] eventq - queue for sending events happened in the device. +\end{description} + +\subsection{Feature bits}\label{sec:Device Types / Video Device / Feature bits} + +\begin{description} +\item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (0)] Guest pages can be used for video + buffers. +\end{description} + +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} + +The device MUST offer at least one of feature bits. + +\subsection{Device configuration layout}\label{sec:Device Types / Video Device / Device configuration layout} + +Video device configuration uses the following layout structure: + +\begin{lstlisting} +struct virtio_video_config { + le32 max_cap_len; +}; +\end{lstlisting} + +\begin{description} +\item[\field{max_cap_len}] defines the maximum length of a descriptor + required to call VIRTIO_VIDEO_GET_CAPABILITY in bytes. The device + MUST set this value. +\end{description} + +\subsection{Device Initialization}\label{sec:Device Types / Video Device / Device Initialization} + +\devicenormative{\subsubsection}{Device Initialization}{Device Types / Video Device / Device Initialization} + +The driver SHOULD query device capability by using the +VIRTIO_VIDEO_T_GET_CAPABILITY and use that information for the initial +setup. + +\subsection{Device Operation}\label{sec:Device Types / Video Device / Device Operation} + +The driver allocates input and output buffers and queues the buffers +to the device. The device performs operations on the buffers according +to the function in question. + +\subsubsection{Device Operation: Create stream} + +To process buffers, the device needs to associate them with a certain +video stream (essentially, a context). Streams are created by +VIRTIO_VIDEO_T_STREAM_CREATE with a default set of parameters +determined by the device. + +\subsubsection{Device Operation: Create buffers} + +Buffers are used to store the actual data as well as the relevant +metadata. Scatter lists are supported, so the buffer doesn't need to +be contiguous in guest physical memory. + +\begin{itemize*} +\item Use VIRTIO_VIDEO_T_RESOURCE_CREATE to create a virtio video + resource that is backed by a buffer allocated from the driver's + memory. +\item Use VIRTIO_VIDEO_T_RESOURCE_DESTROY to destroy a resource that + is no longer needed. +\end{itemize*} + +\subsubsection{Device Operation: Stream parameter control} + +\begin{itemize*} +\item Use VIRTIO_VIDEO_T_GET_PARAMS to get the current stream parameters for + input and output streams from the device. +\item Use VIRTIO_VIDEO_T_SET_PARAMS to provide new stream parameters to the + device. +\item After setting stream parameters, the driver may issue + VIRTIO_VIDEO_T_GET_PARAMS as some parameters of both input and output can be + changed implicitly by the device during the set operation. +\end{itemize*} + +\subsubsection{Device Operation: Process buffers} + +\begin{itemize*} +\item If the function and the buffer type require so, write data to +the buffer memory. +\item Use VIRTIO_VIDEO_T_RESOURCE_QUEUE to queue the buffer for +processing in the device. +\item The request completes asynchronously when the device has +finished with the buffer. +\end{itemize*} + +\subsubsection{Device Operation: Buffer processing control} + +\begin{itemize*} +\item Use VIRTIO_VIDEO_T_STREAM_DRAIN to ask the device to process and + return all of the already queued buffers. +\item Use VIRTIO_VIDEO_T_QUEUE_CLEAR to ask the device to return back + already queued buffers from the input or the output queue. This also + includes input or output buffers that can be currently owned by the + device's processing pipeline. +\end{itemize*} + +\subsubsection{Device Operation: Asynchronous events} + +While processing buffers, the device can send asynchronous event +notifications to the driver. The behaviour depends on the exact +stream. For example, the decoder device sends a resolution change +event when it encounters new resolution metadata in the stream. + +\subsubsection{Device Operation: Request header} + +All requests and responses on the control virt queue have a fixed +header using the following layout structure and definitions: + +\begin{lstlisting} +enum virtio_video_ctrl_type { + VIRTIO_VIDEO_CTRL_UNDEFINED = 0, + + /* request */ + VIRTIO_VIDEO_T_GET_CAPABILITY = 0x0100, + VIRTIO_VIDEO_T_STREAM_CREATE, + VIRTIO_VIDEO_T_STREAM_DESTROY, + VIRTIO_VIDEO_T_STREAM_DRAIN, + VIRTIO_VIDEO_T_RESOURCE_CREATE, + VIRTIO_VIDEO_T_RESOURCE_DESTROY, + VIRTIO_VIDEO_T_RESOURCE_QUEUE, + VIRTIO_VIDEO_T_QUEUE_CLEAR, + VIRTIO_VIDEO_T_SET_PARAMS, + VIRTIO_VIDEO_T_GET_PARAMS, + + /* response */ + VIRTIO_VIDEO_S_OK = 0x0200, + VIRTIO_VIDEO_S_OK_RESOURCE_QUEUE, + VIRTIO_VIDEO_S_OK_GET_PARAMS, + + VIRTIO_VIDEO_S_ERR_UNSPEC = 0x0300, + VIRTIO_VIDEO_S_ERR_OUT_OF_MEMORY, + VIRTIO_VIDEO_S_ERR_INVALID_RESOURCE_ID, + VIRTIO_VIDEO_S_ERR_INVALID_STREAM_ID, + VIRTIO_VIDEO_S_ERR_INVALID_PARAMETER, +}; + +struct virtio_video_ctrl_hdr { + le32 type; + le32 stream_id; + le32 len; /* Length of the structure in bytes. */ +}; +\end{lstlisting} + +\begin{description} +\item[\field{type}] is the type of the driver request or the device +response. +\item[\field{stream_id}] specifies a target stream. +\item[\field{len}] is the length of data in bytes, which includes +length of the header. +\end{description} + +\subsubsection{Device Operation: controlq} + +\begin{description} + +\item[VIRTIO_VIDEO_T_GET_CAPABILITY] Retrieve information about +supported formats. + +The driver uses \field{struct virtio_video_get_capability} to send a +query request. + +\begin{lstlisting} +enum virtio_video_buf_type { + VIRTIO_VIDEO_BUF_TYPE_INPUT, + VIRTIO_VIDEO_BUF_TYPE_OUTPUT, +}; + +struct virtio_video_get_capability { + struct virtio_video_ctrl_hdr hdr; + enum virtio_video_buf_type buf_type; +}; +\end{lstlisting} +\begin{description} +\item[\field{buf_type}] is the buffer type that the driver asks +information about. The driver MUST set either +\field{VIRTIO_VIDEO_BUF_TYPE_INPUT} or \field{VIRTIO_VIDEO_BUF_TYPE_OUTPUT}. +\end{description} + +The device responds a capability by using \field{struct +virtio_video_get_capability_resp}. +\begin{lstlisting} +enum virtio_video_format { + VIRTIO_VIDEO_FORMAT_UNDEFINED = 0, + /* Raw formats */ + VIRTIO_VIDEO_FORMAT_NV12 = 1, + VIRTIO_VIDEO_FORMAT_YUV420, + VIRTIO_VIDEO_FORMAT_YVU420, + + /* Compressed formats */ + VIRTIO_VIDEO_FORMAT_H264 = 0x1001, + VIRTIO_VIDEO_FORMAT_VP8 = 0x1002, + VIRTIO_VIDEO_FORMAT_VP9 = 0x1003, +}; + +enum virtio_video_profile { + VIRTIO_VIDEO_PROFILE_UNDEFINED = 0, + + /* H.264 */ + VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100, + VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_BASELINE, + VIRTIO_VIDEO_PROFILE_H264_MAIN, + VIRTIO_VIDEO_PROFILE_H264_EXTENDED, + VIRTIO_VIDEO_PROFILE_H264_HIGH, + VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE, + VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE, + VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE, + VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE, + VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH, + VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH, + VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + VIRTIO_VIDEO_PROFILE_H264_MAX = VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH, + + /* VP8 */ + VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x200, + VIRTIO_VIDEO_PROFILE_VP8_ANY = VIRTIO_VIDEO_PROFILE_VP8_MIN, + VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_ANY, + + /* VP9 */ + VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x300, + VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN, + VIRTIO_VIDEO_PROFILE_VP9_PROFILE1, + VIRTIO_VIDEO_PROFILE_VP9_PROFILE2, + VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, + VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3, +}; + +struct virtio_video_format_range { + le32 min; + le32 max; + le32 step; + u8 paddings[4]; +}; + +struct virtio_video_format_desc { + le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */ + le32 profile; /* One of VIRTIO_VIDEO_PROFILE_* types */ + le64 mask; + struct virtio_video_format_range width; + struct virtio_video_format_range height; + le32 num_rates; + u8 padding[4]; + /* Followed by struct virtio_video_frame_rate frame_rates[] */ +}; + +struct virtio_video_get_capability_resp { + struct virtio_video_ctrl_hdr hdr; + le32 num_descs; + /* Followed by struct virtio_video_format_desc desc[] */ +}; +\end{lstlisting} + +The format description \field{struct virtio_video_format_desc} +includes the following fields: +\begin{description} +\item[\field{format}] specifies an image format. The device MUST set one + of \field{enum virtio_video_format}. +\item[\field{profile}] specifies a profile of the compressed image format + specified in \field{format}. The driver SHOULD ignore this value if + \field{format} is a raw format. +\item[\field{mask}] is a bitset that represents the supported + combination of input and output format. If \textit{i}-th bit is set + in \field{mask} of \textit{j}-th \field{struct + virtio_video_format_desc} for input, the device supports encoding or + decoding from the \textit{j}-th input format to \textit{i}-th output + format. +\item[\field{width, height}] represents a range of resolutions + supported by the device. If its \field{step} is not applicable, its + \field{min} is equal to its \field{max}. +\item[\field{num_rates}] is the length of an array \field{frame_rates}. In case of decoder, the driver SHOULD ignore this value. +\item[\field{frame_rates}] is an array of supported frame rates. +\end{description} + +\item[VIRTIO_VIDEO_T_STREAM_CREATE] create a video stream (context) + within the device. + +\begin{lstlisting} +enum virtio_video_mem_type { + VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES, +}; + +struct virtio_video_stream_create { + struct virtio_video_ctrl_hdr hdr; + le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ + le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */ + char debug_name[64]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{in_mem_type}] is a type of buffer management for input +buffers. The driver MUST set a value in \field{enum +virtio_video_mem_type}. +\item[\field{out_mem_type}] is a type of buffer management for output +buffers. The driver MUST set a value in \field{enum +virtio_video_mem_type}. +\item[\field{debug_name}] is a text string for a debug purpose. +\end{description} + +\item[VIRTIO_VIDEO_T_STREAM_DESTROY] destroy a video stream (context) + within the device. + +\begin{lstlisting} +struct virtio_video_stream_destroy { + struct virtio_video_ctrl_hdr hdr; +}; +\end{lstlisting} + +\item[VIRTIO_VIDEO_T_STREAM_DRAIN] ask the device to push all the + queued buffers through the pipeline. + +\begin{lstlisting} +struct virtio_video_stream_drain { + struct virtio_video_ctrl_hdr hdr; +}; +\end{lstlisting} + +\item[VIRTIO_VIDEO_T_RESOURCE_CREATE] create a resource descriptor + within the device. + +\begin{lstlisting} +struct virtio_video_mem_entry { + le64 addr; + le32 length; + u8 padding[4]; +}; + +struct virtio_video_resource_create { + struct virtio_video_ctrl_hdr hdr; + le32 resource_id; + le32 nr_entries; +}; +\end{lstlisting} + +\begin{description} +\item[\field{resource_id}] internal id of the resource. +\item[\field{nr_entries}] number of \field{struct + virtio_video_mem_entry} memory entries. +\end{description} + +\item[VIRTIO_VIDEO_T_RESOURCE_DESTROY] destroy a resource descriptor + within the device. + +\begin{lstlisting} +struct virtio_video_resource_destroy { + struct virtio_video_ctrl_hdr hdr; + le32 resource_id; + u8 padding[4]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{resource_id}] internal id of the resource. +\end{description} + +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE] Add a buffer to the device's +queue. + +\begin{lstlisting} +#define VIRTIO_VIDEO_MAX_PLANES 8 + +struct virtio_video_resource_queue { + struct virtio_video_ctrl_hdr hdr; + le32 buf_type; + le32 resource_id; + le64 timestamp; + le32 nr_data_size; + le32 data_size[VIRTIO_VIDEO_MAX_PLANES]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{buf_type}] buf_type of the . +\item[\field{resource_id}] internal id of the resource. +\item[\field{timestamp}] an abstract sequence counter that can be used + for synchronisation. +\item[\field{nr_data_size}] number of \field{data_size} entries. +\item[\field{data_size}] number of data bytes within a plane. +\end{description} + +\begin{lstlisting} +enum virtio_video_buffer_flag { + VIRTIO_VIDEO_BUFFER_F_ERR = 0x0001, + VIRTIO_VIDEO_BUFFER_F_EOS = 0x0002, + /* Encoder only */ + VIRTIO_VIDEO_BUFFER_IFRAME = 0x0004, + VIRTIO_VIDEO_BUFFER_PFRAME = 0x0008, + VIRTIO_VIDEO_BUFFER_BFRAME = 0x0010, +}; + +struct virtio_video_resource_queue_resp { + struct virtio_video_ctrl_hdr hdr; + le64 timestamp; + le32 flags; /* One of VIRTIO_VIDEO_BUFFER_* flags */ + le32 size; /* Encoded size */ +}; +\end{lstlisting} + +\begin{description} +\item[\field{timestamp}] an abstract sequence counter that can be used + for synchronisation. +\item[\field{flags}] mark specific buffers in the sequence. +\item[\field{size}] data size in the buffer (encoder only). +\end{description} + +The device sends a response to the queue request asynchronously when +it has finished processing the buffer. + +The device SHOULD mark a buffer that triggered a processing error with +the VIRTIO_VIDEO_BUFFER_F_ERR flag. + +The device MUST mark the last buffer with the +VIRTIO_VIDEO_BUFFER_F_EOS flag to denote completion of the drain +sequence. + +In case of encoder, to denote a particular frame type the devie MUST +mark the respective buffer with VIRTIO_VIDEO_BUFFER_IFRAME, +VIRTIO_VIDEO_BUFFER_PFRAME, VIRTIO_VIDEO_BUFFER_BFRAME. + +\item[VIRTIO_VIDEO_T_RESOURCE_QUEUE_CLEAR] Return already queued + buffers back from the input or the output queue of the device. The + device SHOULD return all of the buffers from the respective queue as + soon as possible without pushing the buffers through the processing + pipeline. + +\begin{lstlisting} +struct virtio_video_queue_clear { + struct virtio_video_ctrl_hdr hdr; + le32 buf_type; + u8 padding[4]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{buf_type}] buffer type. +\end{description} + +\item[VIRTIO_VIDEO_T_GET_PARAMS] Get parameters of the input or the + output of a stream. + +\begin{lstlisting} +struct virtio_video_plane_format { + le32 plane_size; + le32 stride; + u8 padding[4]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{plane_size}] size of the plane in bytes. +\item[\field{stride}] stride used for the plane in bytes. +\end{description} + +\begin{lstlisting} +struct virtio_video_params { + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ + le32 fourcc; /* One of VIRTIO_VIDEO_FOURCC_* types */ + le32 frame_width; + le32 frame_height; + le32 min_buffers; + le32 max_buffers; + le32 frame_rate; + struct virtio_video_crop { + le32 left; + le32 top; + le32 width; + le32 height; + } crop; + le32 num_planes; + struct virtio_video_plane_format plane_formats[VIRTIO_VIDEO_MAX_PLANES]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{frame_width}] the value to get/set. +\item[\field{frame_height}] the value to get/set. +\item[\field{pixel_format}] the value to get/set. +\item[\field{min_buffers}] minimum buffers required to handle the + format (r/o). +\item[\field{max_buffers}] maximum buffers required to handle the + format (r/o). +\item[\field{frame_rate}] the value to get/set. +\item[\field{crop}] cropping (composing) rectangle. +\item[\field{num_planes}] number of planes used to store pixel data +(r/o). +\item[\field{plane_formats}] description of each plane. +\end{description} + +\begin{lstlisting} +struct virtio_video_get_params { + struct virtio_video_ctrl_hdr hdr; + le32 buf_type; /* One of VIRTIO_VIDEO_BUF_TYPE_* types */ +}; + +struct virtio_video_get_params_resp { + struct virtio_video_ctrl_hdr hdr; + struct virtio_video_params params; +}; +\end{lstlisting} + +\begin{description} +\item[\field{buf_type}] buffer type. +\item[\field{params}] parameter values. +\end{description} + +\item[VIRTIO_VIDEO_T_SET_PARAMS] Change parameters of a stream. + + +\begin{lstlisting} +struct virtio_video_set_params { + struct virtio_video_ctrl_hdr hdr; + struct virtio_video_params params; +}; +\end{lstlisting} + +\begin{description} +\item[\field{params}] parameters to set. +\end{description} + +Setting stream parameters might have side effects within the device. +For example, the device MAY perform alignment of width and height, +change the number of planes it uses for the format, or do whatever +changes that are required to continue normal operation using the +updated parameters. It is up to the driver to check the parameter set +after the VIRTIO_VIDEO_T_SET_PARAMS request has been issued. + +\end{description} + +\subsubsection{Device Operation: eventq} + +The device can report events on the event queue. The driver initially +populates the queue with device-writeable buffers. When the device +needs to report an event, it fills a buffer and notifies the driver. +The driver consumes the report and adds a new buffer to the virtqueue. + +\begin{lstlisting} +enum virtio_video_event_type { + VIRTIO_VIDEO_EVENT_T_UNDEFINED = 0, + /* For all functions */ + VIRTIO_VIDEO_EVENT_T_ERROR_UNSPEC = 0x0100, + /* For decoder only */ + VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED = 0x0200, +}; + +struct virtio_video_event { + le32 event_type; /* One of VIRTIO_VIDEO_EVENT_T_* types */ + le32 stream_id; + u8 padding[4]; +}; +\end{lstlisting} + +\begin{description} +\item[\field{event_type}] type of the triggered event . +\item[\field{stream_id}] id of the source stream. +\end{description} + +The device MUST send VIRTIO_VIDEO_EVENT_T_DECODER_RESOLUTION_CHANGED +whenever it encounters new resolution data in the stream. This +includes the case of the initial device configuration after metadata +has been parsed and the case of dynamic resolution change.