[v5,1/7] media: v4l2: Extend pixel formats to unify single/multi-planar handling (and more)

Commit Message

Helen Mae Koike Fornazier Aug. 4, 2020, 7:29 p.m. UTC

This is part of the multiplanar and singleplanar unification process.
v4l2_ext_pix_format is supposed to work for both cases.

We also add the concept of modifiers already employed in DRM to expose
HW-specific formats (like tiled or compressed formats) and allow
exchanging this information with the DRM subsystem in a consistent way.

Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
v4l2_ext_format, other types will be rejected if you use the
{G,S,TRY}_EXT_PIX_FMT ioctls.

New hooks have been added to v4l2_ioctl_ops to support those new ioctls
in drivers, but, in the meantime, the core takes care of converting
{S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
still work if the userspace app/lib uses the new ioctls.
The conversion is also done the other around to allow userspace
apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
_ext_ hooks.

Signed-off-by: Boris Brezillon <[e-mail]>
Signed-off-by: Helen Koike <[e-mail]>
---
Changes in v5:
- change sizes and reorder fields to avoid holes in the struct and make
  it the same for 32 and 64 bits
- removed __attribute__ ((packed)) from uapi structs
- Fix doc warning from make htmldocs
- Updated commit message with EXT_PIX prefix for the ioctls.

Changes in v4:
- Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
- Add reserved fields
- Removed num_planes from struct v4l2_ext_pix_format
- Removed flag field from struct v4l2_ext_pix_format, since the only
  defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
  where we can use modifiers, or add it back later through the reserved
  bits.
- In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
  != MOD_INVALID
- Fix type assignment in v4l_g_fmt_ext_pix()
- Rebased on top of media/master (post 5.8-rc1)

Changes in v3:
- Rebased on top of media/master (post 5.4-rc1)

Changes in v2:
- Move the modifier in v4l2_ext_format (was formerly placed in
  v4l2_ext_plane)
- Fix a few bugs in the converters and add a strict parameter to
  allow conversion of uninitialized/mis-initialized objects
---
 drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
 drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
 include/media/v4l2-ioctl.h           |  34 ++
 include/uapi/linux/videodev2.h       |  56 +++
 4 files changed, 615 insertions(+), 81 deletions(-)

Comments

Alexandre Courbot Aug. 14, 2020, 7:49 a.m. UTC | #1

On Wed, Aug 5, 2020 at 4:32 AM Helen Koike <[e-mail]> wrote:
>
> This is part of the multiplanar and singleplanar unification process.
> v4l2_ext_pix_format is supposed to work for both cases.
>
> We also add the concept of modifiers already employed in DRM to expose
> HW-specific formats (like tiled or compressed formats) and allow
> exchanging this information with the DRM subsystem in a consistent way.
>
> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
> v4l2_ext_format, other types will be rejected if you use the
> {G,S,TRY}_EXT_PIX_FMT ioctls.
>
> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
> in drivers, but, in the meantime, the core takes care of converting
> {S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
> still work if the userspace app/lib uses the new ioctls.
> The conversion is also done the other around to allow userspace
> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
> _ext_ hooks.
>
> Signed-off-by: Boris Brezillon <[e-mail]>
> Signed-off-by: Helen Koike <[e-mail]>
> ---
> Changes in v5:
> - change sizes and reorder fields to avoid holes in the struct and make
>   it the same for 32 and 64 bits
> - removed __attribute__ ((packed)) from uapi structs
> - Fix doc warning from make htmldocs
> - Updated commit message with EXT_PIX prefix for the ioctls.
>
> Changes in v4:
> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> - Add reserved fields
> - Removed num_planes from struct v4l2_ext_pix_format
> - Removed flag field from struct v4l2_ext_pix_format, since the only
>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>   where we can use modifiers, or add it back later through the reserved
>   bits.
> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>   != MOD_INVALID
> - Fix type assignment in v4l_g_fmt_ext_pix()
> - Rebased on top of media/master (post 5.8-rc1)
>
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
>
> Changes in v2:
> - Move the modifier in v4l2_ext_format (was formerly placed in
>   v4l2_ext_plane)
> - Fix a few bugs in the converters and add a strict parameter to
>   allow conversion of uninitialized/mis-initialized objects
> ---
>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>  include/media/v4l2-ioctl.h           |  34 ++
>  include/uapi/linux/videodev2.h       |  56 +++
>  4 files changed, 615 insertions(+), 81 deletions(-)
>
> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> index a593ea0598b55..e1829906bc086 100644
> --- a/drivers/media/v4l2-core/v4l2-dev.c
> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> @@ -607,25 +607,37 @@ static void determine_valid_ioctls(struct video_device *vdev)
>                         set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
>                 if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
>                                ops->vidioc_g_fmt_vid_cap_mplane ||
> +                              ops->vidioc_g_ext_pix_fmt_vid_cap ||
>                                ops->vidioc_g_fmt_vid_overlay)) ||
>                     (is_tx && (ops->vidioc_g_fmt_vid_out ||
>                                ops->vidioc_g_fmt_vid_out_mplane ||
> -                              ops->vidioc_g_fmt_vid_out_overlay)))
> +                              ops->vidioc_g_ext_pix_fmt_vid_out ||
> +                              ops->vidioc_g_fmt_vid_out_overlay))) {
>                          set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
> +                        set_bit(_IOC_NR(VIDIOC_G_EXT_PIX_FMT), valid_ioctls);
> +               }
>                 if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
>                                ops->vidioc_s_fmt_vid_cap_mplane ||
> +                              ops->vidioc_s_ext_pix_fmt_vid_cap ||
>                                ops->vidioc_s_fmt_vid_overlay)) ||
>                     (is_tx && (ops->vidioc_s_fmt_vid_out ||
>                                ops->vidioc_s_fmt_vid_out_mplane ||
> -                              ops->vidioc_s_fmt_vid_out_overlay)))
> +                              ops->vidioc_s_ext_pix_fmt_vid_out ||
> +                              ops->vidioc_s_fmt_vid_out_overlay))) {
>                          set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
> +                        set_bit(_IOC_NR(VIDIOC_S_EXT_PIX_FMT), valid_ioctls);
> +               }
>                 if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
>                                ops->vidioc_try_fmt_vid_cap_mplane ||
> +                              ops->vidioc_try_ext_pix_fmt_vid_cap ||
>                                ops->vidioc_try_fmt_vid_overlay)) ||
>                     (is_tx && (ops->vidioc_try_fmt_vid_out ||
>                                ops->vidioc_try_fmt_vid_out_mplane ||
> -                              ops->vidioc_try_fmt_vid_out_overlay)))
> +                              ops->vidioc_try_ext_pix_fmt_vid_out ||
> +                              ops->vidioc_try_fmt_vid_out_overlay))) {
>                          set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
> +                        set_bit(_IOC_NR(VIDIOC_TRY_EXT_PIX_FMT), valid_ioctls);
> +               }
>                 SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
>                 SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
>                 SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
> @@ -682,8 +694,11 @@ static void determine_valid_ioctls(struct video_device *vdev)
>                 /* touch specific ioctls */
>                 SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_vid_cap);
>                 SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_vid_cap);
> +               SET_VALID_IOCTL(ops, VIDIOC_G_EXT_PIX_FMT, vidioc_g_fmt_vid_cap);
>                 SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_vid_cap);
> +               SET_VALID_IOCTL(ops, VIDIOC_S_EXT_PIX_FMT, vidioc_s_fmt_vid_cap);
>                 SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_vid_cap);
> +               SET_VALID_IOCTL(ops, VIDIOC_TRY_EXT_PIX_FMT, vidioc_try_fmt_vid_cap);
>                 SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
>                 SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
>                 SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index a556880f225a5..14a0def50f8ea 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -17,6 +17,8 @@
>
>  #include <linux/videodev2.h>
>
> +#include <drm/drm_fourcc.h>
> +
>  #include <media/v4l2-common.h>
>  #include <media/v4l2-ioctl.h>
>  #include <media/v4l2-ctrls.h>
> @@ -378,6 +380,27 @@ static void v4l_print_format(const void *arg, bool write_only)
>         }
>  }
>
> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
> +{
> +       const struct v4l2_ext_pix_format *pix = arg;
> +       unsigned int i;
> +
> +       pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
> +               prt_names(pix->type, v4l2_type_names),
> +               pix->width, pix->height,
> +               (pix->pixelformat & 0xff),
> +               (pix->pixelformat >>  8) & 0xff,
> +               (pix->pixelformat >> 16) & 0xff,
> +               (pix->pixelformat >> 24) & 0xff,
> +               pix->modifier, prt_names(pix->field, v4l2_field_names),
> +               pix->colorspace, pix->ycbcr_enc,
> +               pix->quantization, pix->xfer_func);
> +       for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
> +               pr_debug("plane %u: bytesperline=%u sizeimage=%u\n",
> +                        i, pix->plane_fmt[i].bytesperline,
> +                        pix->plane_fmt[i].sizeimage);
> +}
> +
>  static void v4l_print_framebuffer(const void *arg, bool write_only)
>  {
>         const struct v4l2_framebuffer *p = arg;
> @@ -964,11 +987,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>         switch (type) {
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>                 if ((is_vid || is_tch) && is_rx &&
> -                   (ops->vidioc_g_fmt_vid_cap || ops->vidioc_g_fmt_vid_cap_mplane))
> +                   (ops->vidioc_g_fmt_vid_cap ||
> +                    ops->vidioc_g_ext_pix_fmt_vid_cap ||
> +                    ops->vidioc_g_fmt_vid_cap_mplane))
>                         return 0;
>                 break;
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -               if ((is_vid || is_tch) && is_rx && ops->vidioc_g_fmt_vid_cap_mplane)
> +               if ((is_vid || is_tch) && is_rx &&
> +                   (ops->vidioc_g_fmt_vid_cap_mplane ||
> +                    ops->vidioc_g_ext_pix_fmt_vid_cap))
>                         return 0;
>                 break;
>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
> @@ -977,11 +1004,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>                 break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>                 if (is_vid && is_tx &&
> -                   (ops->vidioc_g_fmt_vid_out || ops->vidioc_g_fmt_vid_out_mplane))
> +                   (ops->vidioc_g_fmt_vid_out ||
> +                    ops->vidioc_g_ext_pix_fmt_vid_out ||
> +                    ops->vidioc_g_fmt_vid_out_mplane))
>                         return 0;
>                 break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -               if (is_vid && is_tx && ops->vidioc_g_fmt_vid_out_mplane)
> +               if (is_vid && is_tx &&
> +                   (ops->vidioc_g_ext_pix_fmt_vid_out ||
> +                    ops->vidioc_g_fmt_vid_out_mplane))
>                         return 0;
>                 break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
> @@ -1061,6 +1092,134 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>                sizeof(fmt->fmt.pix) - offset);
>  }
>
> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> +                                 struct v4l2_format *f, bool mplane_cap,
> +                                 bool strict)
> +{
> +       const struct v4l2_plane_ext_pix_format *pe;
> +       struct v4l2_plane_pix_format *p;
> +       unsigned int i;
> +
> +       memset(f, 0, sizeof(*f));
> +
> +       /*
> +        * Make sure no modifier is required before doing the
> +        * conversion.
> +        */
> +       if (e->modifier && strict &&
> +           e->modifier != DRM_FORMAT_MOD_LINEAR &&
> +           e->modifier != DRM_FORMAT_MOD_INVALID)
> +               return -EINVAL;
> +
> +       if (!e->plane_fmt[0].sizeimage && strict)
> +               return -EINVAL;
> +
> +       if (e->plane_fmt[1].sizeimage && !mplane_cap && strict)
> +               return 0;
> +
> +       if (!mplane_cap) {
> +               f->fmt.pix.width = e->width;
> +               f->fmt.pix.height = e->height;
> +               f->fmt.pix.pixelformat = e->pixelformat;
> +               f->fmt.pix.field = e->field;
> +               f->fmt.pix.colorspace = e->colorspace;
> +               f->fmt.pix.ycbcr_enc = e->ycbcr_enc;
> +               f->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +               f->fmt.pix.quantization = e->quantization;
> +               pe = &e->plane_fmt[0];
> +               f->fmt.pix.bytesperline = pe->bytesperline;
> +               f->fmt.pix.sizeimage = pe->sizeimage;
> +               f->type = e->type;
> +               return 0;
> +       }
> +
> +       f->fmt.pix_mp.width = e->width;
> +       f->fmt.pix_mp.height = e->height;
> +       f->fmt.pix_mp.pixelformat = e->pixelformat;
> +       f->fmt.pix_mp.field = e->field;
> +       f->fmt.pix_mp.colorspace = e->colorspace;
> +       f->fmt.pix_mp.ycbcr_enc = e->ycbcr_enc;
> +       f->fmt.pix_mp.quantization = e->quantization;
> +       if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +               f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> +       else
> +               f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
> +
> +       for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> +               pe = &e->plane_fmt[i];
> +               p = &f->fmt.pix_mp.plane_fmt[i];
> +               p->bytesperline = pe->bytesperline;
> +               p->sizeimage = pe->sizeimage;
> +       }
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_ext_pix_format_to_format);
> +
> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
> +                                 struct v4l2_ext_pix_format *e, bool strict)
> +{
> +       const struct v4l2_plane_pix_format *p;
> +       struct v4l2_plane_ext_pix_format *pe;
> +       unsigned int i;
> +
> +       memset(e, 0, sizeof(*e));
> +
> +       switch (f->type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +               e->width = f->fmt.pix.width;
> +               e->height = f->fmt.pix.height;
> +               e->pixelformat = f->fmt.pix.pixelformat;
> +               e->field = f->fmt.pix.field;
> +               e->colorspace = f->fmt.pix.colorspace;
> +               if (f->fmt.pix.flags)
> +                       pr_warn("Ignoring pixelformat flags 0x%x\n",
> +                               f->fmt.pix.flags);
> +               e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
> +               e->quantization = f->fmt.pix.quantization;
> +               e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
> +               e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
> +               e->type = f->type;
> +               break;
> +
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +               if ((f->fmt.pix_mp.num_planes > VIDEO_MAX_PLANES ||
> +                    !f->fmt.pix_mp.num_planes) && strict)
> +                       return -EINVAL;
> +
> +               e->width = f->fmt.pix_mp.width;
> +               e->height = f->fmt.pix_mp.height;
> +               e->pixelformat = f->fmt.pix_mp.pixelformat;
> +               e->field = f->fmt.pix_mp.field;
> +               e->colorspace = f->fmt.pix_mp.colorspace;
> +               if (f->fmt.pix.flags)
> +                       pr_warn("Ignoring pixelformat flags 0x%x\n",
> +                               f->fmt.pix.flags);
> +               e->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
> +               e->quantization = f->fmt.pix_mp.quantization;
> +               if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> +                       e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +               else
> +                       e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +
> +               for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> +                       pe = &e->plane_fmt[i];
> +                       p = &f->fmt.pix_mp.plane_fmt[i];
> +                       pe->bytesperline = p->bytesperline;
> +                       pe->sizeimage = p->sizeimage;
> +               }
> +               break;
> +
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
> +
>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>                                 struct file *file, void *fh, void *arg)
>  {
> @@ -1564,6 +1723,38 @@ static void v4l_pix_format_touch(struct v4l2_pix_format *p)
>         p->xfer_func = 0;
>  }
>
> +static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +                            struct file *file, void *fh,
> +                            struct v4l2_format *f)
> +{
> +       struct v4l2_ext_pix_format ef;
> +       int ret;
> +
> +       switch (f->type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +               ef.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +               ret = ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, &ef);
> +               break;
> +
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +               ef.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +               ret = ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, &ef);
> +               break;
> +
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       if (ret)
> +               return ret;
> +
> +       return v4l2_ext_pix_format_to_format(&ef, f,
> +                                            V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +                                            true);
> +}
> +
>  static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>                                 struct file *file, void *fh, void *arg)
>  {
> @@ -1600,17 +1791,27 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>
>         switch (p->type) {
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -               if (unlikely(!ops->vidioc_g_fmt_vid_cap))
> -                       break;
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
> -               /* just in case the driver zeroed it again */
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               if (vfd->vfl_type == VFL_TYPE_TOUCH)
> -                       v4l_pix_format_touch(&p->fmt.pix);
> -               return ret;
> +               if (ops->vidioc_g_fmt_vid_cap) {
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
> +                       /* just in case the driver zeroed it again */
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +                               v4l_pix_format_touch(&p->fmt.pix);
> +                       return ret;
> +               } else if (ops->vidioc_g_ext_pix_fmt_vid_cap) {
> +                       ret = v4l_g_fmt_ext_pix(ops, file, fh, p);
> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +                               v4l_pix_format_touch(&p->fmt.pix);
> +                       return ret;
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -               return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> +               if (ops->vidioc_g_fmt_vid_cap_mplane)
> +                       return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> +               else if (ops->vidioc_g_ext_pix_fmt_vid_cap)
> +                       return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>                 return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
>         case V4L2_BUF_TYPE_VBI_CAPTURE:
> @@ -1618,15 +1819,22 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>         case V4L2_BUF_TYPE_SLICED_VBI_CAPTURE:
>                 return ops->vidioc_g_fmt_sliced_vbi_cap(file, fh, arg);
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -               if (unlikely(!ops->vidioc_g_fmt_vid_out))
> -                       break;
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
> -               /* just in case the driver zeroed it again */
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               return ret;
> +               if (ops->vidioc_g_fmt_vid_out) {
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
> +                       /* just in case the driver zeroed it again */
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       return ret;
> +               } else if (ops->vidioc_g_ext_pix_fmt_vid_out) {
> +                       return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -               return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> +               if (ops->vidioc_g_fmt_vid_out_mplane)
> +                       return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> +               else if (ops->vidioc_g_ext_pix_fmt_vid_out)
> +                       return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>                 return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
>         case V4L2_BUF_TYPE_VBI_OUTPUT:
> @@ -1645,6 +1853,76 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>         return -EINVAL;
>  }
>
> +static int v4l_g_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +                            struct file *file, void *fh, void *arg)
> +{
> +       struct v4l2_ext_pix_format *ef = arg;
> +       struct v4l2_format f = {
> +               .type = ef->type,
> +       };
> +       int ret;
> +
> +       ret = check_fmt(file, ef->type);
> +       if (ret)
> +               return ret;
> +
> +       memset(ef, 0, sizeof(*ef));
> +       ef->type = f.type;
> +
> +       switch (f.type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +               if (ops->vidioc_g_ext_pix_fmt_vid_cap)
> +                       return ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, ef);
> +               break;
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +               if (ops->vidioc_g_ext_pix_fmt_vid_out)
> +                       return ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, ef);
> +               break;
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       ret = v4l_g_fmt(ops, file, fh, &f);
> +       if (ret)
> +               return ret;
> +
> +       return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
> +static int v4l_s_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +                            struct file *file, void *fh,
> +                            struct v4l2_format *f)
> +{
> +       struct v4l2_ext_pix_format ef;
> +       int ret;
> +
> +       ret = v4l2_format_to_ext_pix_format(f, &ef, false);
> +       if (ret)
> +               return ret;
> +
> +       switch (f->type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +               ret = ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, &ef);
> +               break;
> +
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +               ret = ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, &ef);
> +               break;
> +
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       if (ret)
> +               return ret;
> +
> +       return v4l2_ext_pix_format_to_format(&ef, f,
> +                                            V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +                                            true);
> +}
> +
>  static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>                                 struct file *file, void *fh, void *arg)
>  {
> @@ -1663,23 +1941,31 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>
>         switch (p->type) {
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -               if (unlikely(!ops->vidioc_s_fmt_vid_cap))
> +               if (ops->vidioc_s_fmt_vid_cap) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
> +                       ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
> +                       /* just in case the driver zeroed it again */
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
> +                       ret = v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +               } else {
>                         break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix);
> -               ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
> -               /* just in case the driver zeroed it again */
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +               }
> +
>                 if (vfd->vfl_type == VFL_TYPE_TOUCH)
>                         v4l_pix_format_touch(&p->fmt.pix);
>                 return ret;
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -               if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -                                         bytesperline);
> -               return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
> +               if (ops->vidioc_s_fmt_vid_cap_mplane) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +                                                 bytesperline);
> +                       return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
> +                       return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>                 if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
>                         break;
> @@ -1696,21 +1982,27 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>                 CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>                 return ops->vidioc_s_fmt_sliced_vbi_cap(file, fh, arg);
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -               if (unlikely(!ops->vidioc_s_fmt_vid_out))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix);
> -               ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
> -               /* just in case the driver zeroed it again */
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               return ret;
> +               if (ops->vidioc_s_fmt_vid_out) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
> +                       ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
> +                       /* just in case the driver zeroed it again */
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       return ret;
> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
> +                       return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -               if (unlikely(!ops->vidioc_s_fmt_vid_out_mplane))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -                                         bytesperline);
> -               return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
> +               if (ops->vidioc_s_fmt_vid_out_mplane) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +                                                 bytesperline);
> +                       return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
> +                       return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>                 if (unlikely(!ops->vidioc_s_fmt_vid_out_overlay))
>                         break;
> @@ -1750,6 +2042,82 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>         return -EINVAL;
>  }
>
> +static int v4l_s_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +                            struct file *file, void *fh, void *arg)
> +{
> +       struct video_device *vfd = video_devdata(file);
> +       struct v4l2_ext_pix_format *ef = arg;
> +       struct v4l2_format f;
> +       int ret;
> +
> +       ret = check_fmt(file, ef->type);
> +       if (ret)
> +               return ret;
> +
> +       switch (ef->type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +               if (ops->vidioc_s_ext_pix_fmt_vid_cap)
> +                       return ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, ef);
> +               break;
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +               if (ops->vidioc_s_ext_pix_fmt_vid_out)
> +                       return ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, ef);
> +               break;
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       ret = v4l2_ext_pix_format_to_format(ef, &f,
> +                                           vfd->device_caps &
> +                                           (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +                                            V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +                                            V4L2_CAP_VIDEO_M2M_MPLANE),
> +                                           false);
> +       if (ret)
> +               return ret;
> +
> +       ret = v4l_s_fmt(ops, file, fh, &f);
> +       if (ret)
> +               return ret;
> +
> +       return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
> +static int v4l_try_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +                              struct file *file, void *fh,
> +                              struct v4l2_format *f)
> +{
> +       struct v4l2_ext_pix_format ef;
> +       int ret;
> +
> +       ret = v4l2_format_to_ext_pix_format(f, &ef, false);
> +       if (ret)
> +               return ret;
> +
> +       switch (f->type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +               ret = ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh, &ef);
> +               break;
> +
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +               ret = ops->vidioc_try_ext_pix_fmt_vid_out(file, fh, &ef);
> +               break;
> +
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       if (ret)
> +               return ret;
> +
> +       return v4l2_ext_pix_format_to_format(&ef, f,
> +                                            V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +                                            true);
> +}
> +
> +
>  static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>                                 struct file *file, void *fh, void *arg)
>  {
> @@ -1765,23 +2133,32 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>
>         switch (p->type) {
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -               if (unlikely(!ops->vidioc_try_fmt_vid_cap))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix);
> -               ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
> -               /* just in case the driver zeroed it again */
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               if (vfd->vfl_type == VFL_TYPE_TOUCH)
> -                       v4l_pix_format_touch(&p->fmt.pix);
> -               return ret;
> +               if (ops->vidioc_try_fmt_vid_cap) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
> +                       ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
> +                       /* just in case the driver zeroed it again */
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +                               v4l_pix_format_touch(&p->fmt.pix);
> +                       return ret;
> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +                       ret = v4l_try_fmt_ext_pix(ops, file, fh, p);
> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +                               v4l_pix_format_touch(&p->fmt.pix);
> +                       return ret;
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -               if (unlikely(!ops->vidioc_try_fmt_vid_cap_mplane))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -                                         bytesperline);
> -               return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
> +               if (ops->vidioc_try_fmt_vid_cap_mplane) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +                                                 bytesperline);
> +                       return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +                       return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>                 if (unlikely(!ops->vidioc_try_fmt_vid_overlay))
>                         break;
> @@ -1798,21 +2175,27 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>                 CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>                 return ops->vidioc_try_fmt_sliced_vbi_cap(file, fh, arg);
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -               if (unlikely(!ops->vidioc_try_fmt_vid_out))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix);
> -               ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
> -               /* just in case the driver zeroed it again */
> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -               return ret;
> +               if (ops->vidioc_try_fmt_vid_out) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
> +                       ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
> +                       /* just in case the driver zeroed it again */
> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +                       return ret;
> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +                       return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -               if (unlikely(!ops->vidioc_try_fmt_vid_out_mplane))
> -                       break;
> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -                                         bytesperline);
> -               return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
> +               if (ops->vidioc_try_fmt_vid_out_mplane) {
> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +                                                 bytesperline);
> +                       return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +                       return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +               }
> +               break;
>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>                 if (unlikely(!ops->vidioc_try_fmt_vid_out_overlay))
>                         break;
> @@ -1852,6 +2235,49 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>         return -EINVAL;
>  }
>
> +static int v4l_try_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +                              struct file *file, void *fh, void *arg)
> +{
> +       struct video_device *vfd = video_devdata(file);
> +       struct v4l2_ext_pix_format *ef = arg;
> +       struct v4l2_format f;
> +       int ret;
> +
> +       ret = check_fmt(file, ef->type);
> +       if (ret)
> +               return ret;
> +
> +       switch (ef->type) {
> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +               if (ops->vidioc_try_ext_pix_fmt_vid_cap)
> +                       return ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh,
> +                                                                  ef);
> +               break;
> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +               if (ops->vidioc_try_ext_pix_fmt_vid_out)
> +                       return ops->vidioc_try_ext_pix_fmt_vid_out(file, fh,
> +                                                                  ef);
> +               break;
> +       default:
> +               return -EINVAL;
> +       }
> +
> +       ret = v4l2_ext_pix_format_to_format(ef, &f,
> +                                           vfd->device_caps &
> +                                           (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +                                            V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +                                            V4L2_CAP_VIDEO_M2M_MPLANE),
> +                                           false);
> +       if (ret)
> +               return ret;
> +
> +       ret = v4l_try_fmt(ops, file, fh, &f);
> +       if (ret)
> +               return ret;
> +
> +       return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
>  static int v4l_streamon(const struct v4l2_ioctl_ops *ops,
>                                 struct file *file, void *fh, void *arg)
>  {
> @@ -2771,7 +3197,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>         IOCTL_INFO(VIDIOC_QUERYCAP, v4l_querycap, v4l_print_querycap, 0),
>         IOCTL_INFO(VIDIOC_ENUM_FMT, v4l_enum_fmt, v4l_print_fmtdesc, 0),
>         IOCTL_INFO(VIDIOC_G_FMT, v4l_g_fmt, v4l_print_format, 0),
> +       IOCTL_INFO(VIDIOC_G_EXT_PIX_FMT, v4l_g_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>         IOCTL_INFO(VIDIOC_S_FMT, v4l_s_fmt, v4l_print_format, INFO_FL_PRIO),
> +       IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>         IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>         IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
>         IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
> @@ -2818,6 +3246,7 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>         IOCTL_INFO(VIDIOC_S_JPEGCOMP, v4l_stub_s_jpegcomp, v4l_print_jpegcompression, INFO_FL_PRIO),
>         IOCTL_INFO(VIDIOC_QUERYSTD, v4l_querystd, v4l_print_std, 0),
>         IOCTL_INFO(VIDIOC_TRY_FMT, v4l_try_fmt, v4l_print_format, 0),
> +       IOCTL_INFO(VIDIOC_TRY_EXT_PIX_FMT, v4l_try_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>         IOCTL_INFO(VIDIOC_ENUMAUDIO, v4l_stub_enumaudio, v4l_print_audio, INFO_FL_CLEAR(v4l2_audio, index)),
>         IOCTL_INFO(VIDIOC_ENUMAUDOUT, v4l_stub_enumaudout, v4l_print_audioout, INFO_FL_CLEAR(v4l2_audioout, index)),
>         IOCTL_INFO(VIDIOC_G_PRIORITY, v4l_g_priority, v4l_print_u32, 0),
> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
> index 86878fba332b0..8bbcb74d8ee31 100644
> --- a/include/media/v4l2-ioctl.h
> +++ b/include/media/v4l2-ioctl.h
> @@ -48,11 +48,17 @@ struct v4l2_fh;
>   * @vidioc_g_fmt_vid_cap: pointer to the function that implements
>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *     in single plane mode
> + * @vidioc_g_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *     :ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *     capture
>   * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_g_fmt_vid_out: pointer to the function that implements
>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *     in single plane mode
> + * @vidioc_g_ext_pix_fmt_vid_out: pointer to the function that implements
> + *     :ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *     out
>   * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>   * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
> @@ -82,11 +88,16 @@ struct v4l2_fh;
>   * @vidioc_s_fmt_vid_cap: pointer to the function that implements
>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *     in single plane mode
> + * @vidioc_s_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *     :ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *     capture
>   * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_s_fmt_vid_out: pointer to the function that implements
>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *     in single plane mode
> + * @vidioc_s_ext_pix_fmt_vid_out: pointer to the function that implements
> + *     :ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>   * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>   * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
> @@ -116,11 +127,16 @@ struct v4l2_fh;
>   * @vidioc_try_fmt_vid_cap: pointer to the function that implements
>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *     in single plane mode
> + * @vidioc_try_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *     :ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for
> +       video capture
>   * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_try_fmt_vid_out: pointer to the function that implements
>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *     in single plane mode
> + * @vidioc_try_ext_pix_fmt_vid_out: pointer to the function that implements
> + *     :ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>   * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   *     output
> @@ -319,10 +335,14 @@ struct v4l2_ioctl_ops {
>         /* VIDIOC_G_FMT handlers */
>         int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
>                                     struct v4l2_format *f);
> +       int (*vidioc_g_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +                                           struct v4l2_ext_pix_format *f);
>         int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
>                                         struct v4l2_format *f);
>         int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
>                                     struct v4l2_format *f);
> +       int (*vidioc_g_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +                                           struct v4l2_ext_pix_format *f);
>         int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
>                                             struct v4l2_format *f);
>         int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -349,10 +369,14 @@ struct v4l2_ioctl_ops {
>         /* VIDIOC_S_FMT handlers */
>         int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
>                                     struct v4l2_format *f);
> +       int (*vidioc_s_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +                                           struct v4l2_ext_pix_format *f);
>         int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
>                                         struct v4l2_format *f);
>         int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
>                                     struct v4l2_format *f);
> +       int (*vidioc_s_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +                                           struct v4l2_ext_pix_format *f);
>         int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
>                                             struct v4l2_format *f);
>         int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -379,10 +403,14 @@ struct v4l2_ioctl_ops {
>         /* VIDIOC_TRY_FMT handlers */
>         int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
>                                       struct v4l2_format *f);
> +       int (*vidioc_try_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +                                             struct v4l2_ext_pix_format *f);
>         int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
>                                           struct v4l2_format *f);
>         int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
>                                       struct v4l2_format *f);
> +       int (*vidioc_try_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +                                             struct v4l2_ext_pix_format *f);
>         int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
>                                              struct v4l2_format *f);
>         int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -724,6 +752,12 @@ long int video_usercopy(struct file *file, unsigned int cmd,
>  long int video_ioctl2(struct file *file,
>                       unsigned int cmd, unsigned long int arg);
>
> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
> +                                 struct v4l2_ext_pix_format *e, bool strict);
> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> +                                 struct v4l2_format *f,
> +                                 bool mplane_cap, bool strict);
> +
>  /*
>   * The user space interpretation of the 'v4l2_event' differs
>   * based on the 'time_t' definition on 32-bit architectures, so
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index c7b70ff53bc1d..7123c6a4d9569 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -2254,6 +2254,58 @@ struct v4l2_pix_format_mplane {
>         __u8                            reserved[7];
>  } __attribute__ ((packed));
>
> +/**
> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
> + * @sizeimage:         maximum size in bytes required for data, for which
> + *                     this plane will be used.
> + *                     Should be set to zero for unused planes.
> + * @bytesperline:      distance in bytes between the leftmost pixels in two
> + *                     adjacent lines
> + * @reserved:          extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_plane_ext_pix_format {
> +       __u32 sizeimage;
> +       __u32 bytesperline;
> +       __u32 reserved;

Do we want to make this field a __u64 so the size of the struct is a
multiple of 64 bits?



> +};
> +
> +/**
> + * struct v4l2_ext_pix_format - extended single/multiplanar format definition
> + * @type:              type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
> + *                     V4L2_BUF_TYPE_VIDEO_OUTPUT
> + * @width:             image width in pixels
> + * @height:            image height in pixels
> + * @field:             enum v4l2_field; field order (for interlaced video)
> + * @pixelformat:       little endian four character code (fourcc)
> + * @modifier:          modifier applied to the format (used for tiled formats
> + *                     and other kind of HW-specific formats, like compressed
> + *                     formats)
> + * @colorspace:                enum v4l2_colorspace; supplemental to pixelformat
> + * @plane_fmt:         per-plane information
> + * @ycbcr_enc:         enum v4l2_ycbcr_encoding, Y'CbCr encoding
> + * @hsv_enc:           enum v4l2_hsv_encoding, HSV encoding
> + * @quantization:      enum v4l2_quantization, colorspace quantization
> + * @xfer_func:         enum v4l2_xfer_func, colorspace transfer function
> + * @reserved:          extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_ext_pix_format {
> +       __u32 type;
> +       __u32 width;
> +       __u32 height;
> +       __u32 field;
> +       __u64 modifier;
> +       __u32 pixelformat;
> +       __u32 colorspace;
> +       struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
> +       union {
> +               __u32 ycbcr_enc;
> +               __u32 hsv_enc;
> +       };
> +       __u32 quantization;
> +       __u32 xfer_func;
> +       __u32 reserved[9];
> +};
> +
>  /**
>   * struct v4l2_sdr_format - SDR format definition
>   * @pixelformat:       little endian four character code (fourcc)
> @@ -2571,6 +2623,10 @@ struct v4l2_create_buffers {
>
>  #define VIDIOC_QUERY_EXT_CTRL  _IOWR('V', 103, struct v4l2_query_ext_ctrl)
>
> +#define VIDIOC_G_EXT_PIX_FMT   _IOWR('V', 104, struct v4l2_ext_pix_format)
> +#define VIDIOC_S_EXT_PIX_FMT   _IOWR('V', 105, struct v4l2_ext_pix_format)
> +#define VIDIOC_TRY_EXT_PIX_FMT _IOWR('V', 106, struct v4l2_ext_pix_format)
> +
>  /* Reminder: when adding new ioctls please add support for them to
>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>
> --
> 2.28.0.rc2
>

Hans Verkuil Sept. 9, 2020, 11:41 a.m. UTC | #2

Hi Helen,

Some review comments, concentrating on the uAPI.

On 04/08/2020 21:29, Helen Koike wrote:
> This is part of the multiplanar and singleplanar unification process.
> v4l2_ext_pix_format is supposed to work for both cases.
> 
> We also add the concept of modifiers already employed in DRM to expose
> HW-specific formats (like tiled or compressed formats) and allow
> exchanging this information with the DRM subsystem in a consistent way.
> 
> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
> v4l2_ext_format, other types will be rejected if you use the
> {G,S,TRY}_EXT_PIX_FMT ioctls.
> 
> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
> in drivers, but, in the meantime, the core takes care of converting
> {S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
> still work if the userspace app/lib uses the new ioctls.
> The conversion is also done the other around to allow userspace
> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
> _ext_ hooks.
> 
> Signed-off-by: Boris Brezillon <[e-mail]>
> Signed-off-by: Helen Koike <[e-mail]>
> ---
> Changes in v5:
> - change sizes and reorder fields to avoid holes in the struct and make
>   it the same for 32 and 64 bits
> - removed __attribute__ ((packed)) from uapi structs
> - Fix doc warning from make htmldocs
> - Updated commit message with EXT_PIX prefix for the ioctls.
> 
> Changes in v4:
> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> - Add reserved fields
> - Removed num_planes from struct v4l2_ext_pix_format
> - Removed flag field from struct v4l2_ext_pix_format, since the only
>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>   where we can use modifiers, or add it back later through the reserved
>   bits.
> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>   != MOD_INVALID
> - Fix type assignment in v4l_g_fmt_ext_pix()
> - Rebased on top of media/master (post 5.8-rc1)
> 
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
> 
> Changes in v2:
> - Move the modifier in v4l2_ext_format (was formerly placed in
>   v4l2_ext_plane)
> - Fix a few bugs in the converters and add a strict parameter to
>   allow conversion of uninitialized/mis-initialized objects
> ---
>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>  include/media/v4l2-ioctl.h           |  34 ++
>  include/uapi/linux/videodev2.h       |  56 +++
>  4 files changed, 615 insertions(+), 81 deletions(-)
> 

<snip>

> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index c7b70ff53bc1d..7123c6a4d9569 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -2254,6 +2254,58 @@ struct v4l2_pix_format_mplane {
>  	__u8				reserved[7];
>  } __attribute__ ((packed));
>  
> +/**
> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
> + * @sizeimage:		maximum size in bytes required for data, for which
> + *			this plane will be used.
> + *			Should be set to zero for unused planes.
> + * @bytesperline:	distance in bytes between the leftmost pixels in two
> + *			adjacent lines
> + * @reserved:		extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_plane_ext_pix_format {
> +	__u32 sizeimage;
> +	__u32 bytesperline;
> +	__u32 reserved;

I'd use reserved[4] here.

> +};
> +
> +/**
> + * struct v4l2_ext_pix_format - extended single/multiplanar format definition
> + * @type:		type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
> + *			V4L2_BUF_TYPE_VIDEO_OUTPUT
> + * @width:		image width in pixels
> + * @height:		image height in pixels
> + * @field:		enum v4l2_field; field order (for interlaced video)
> + * @pixelformat:	little endian four character code (fourcc)
> + * @modifier:		modifier applied to the format (used for tiled formats
> + *			and other kind of HW-specific formats, like compressed
> + *			formats)

This should refer to the drm.h header since we're reusing the DRM modifiers.

> + * @colorspace:		enum v4l2_colorspace; supplemental to pixelformat
> + * @plane_fmt:		per-plane information
> + * @ycbcr_enc:		enum v4l2_ycbcr_encoding, Y'CbCr encoding
> + * @hsv_enc:		enum v4l2_hsv_encoding, HSV encoding
> + * @quantization:	enum v4l2_quantization, colorspace quantization
> + * @xfer_func:		enum v4l2_xfer_func, colorspace transfer function
> + * @reserved:		extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_ext_pix_format {
> +	__u32 type;
> +	__u32 width;
> +	__u32 height;
> +	__u32 field;
> +	__u64 modifier;
> +	__u32 pixelformat;
> +	__u32 colorspace;
> +	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
> +	union {
> +		__u32 ycbcr_enc;
> +		__u32 hsv_enc;
> +	};
> +	__u32 quantization;
> +	__u32 xfer_func;

I'd reorder this:

	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
	__u32 pixelformat;
	__u32 colorspace;
	__u32 xfer_func;
	union {
		__u32 ycbcr_enc;
		__u32 hsv_enc;
	};
	__u32 quantization;

The reason for reordering is that I like to keep the colorimetry fields in
that order since that is how these fields are processed mathematically: you
apply the colorspace matrix first, then the transfer function, then optionally
convert to Y'CbCr or HSV and finally you quantize the result.

There is also a __u32 flags; field missing (needed for V4L2_PIX_FMT_FLAG_PREMUL_ALPHA
and for the upcoming CSC support).

> +	__u32 reserved[9];
> +};
> +
>  /**
>   * struct v4l2_sdr_format - SDR format definition
>   * @pixelformat:	little endian four character code (fourcc)
> @@ -2571,6 +2623,10 @@ struct v4l2_create_buffers {
>  
>  #define VIDIOC_QUERY_EXT_CTRL	_IOWR('V', 103, struct v4l2_query_ext_ctrl)
>  
> +#define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
> +#define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
> +#define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
> +
>  /* Reminder: when adding new ioctls please add support for them to
>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>  
> 

Regards,

	Hans

Helen Mae Koike Fornazier Sept. 14, 2020, 2:14 a.m. UTC | #3

Hi Hans,

On 9/9/20 8:41 AM, Hans Verkuil wrote:
> Hi Helen,
> 
> Some review comments, concentrating on the uAPI.
> 
> On 04/08/2020 21:29, Helen Koike wrote:
>> This is part of the multiplanar and singleplanar unification process.
>> v4l2_ext_pix_format is supposed to work for both cases.
>>
>> We also add the concept of modifiers already employed in DRM to expose
>> HW-specific formats (like tiled or compressed formats) and allow
>> exchanging this information with the DRM subsystem in a consistent way.
>>
>> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
>> v4l2_ext_format, other types will be rejected if you use the
>> {G,S,TRY}_EXT_PIX_FMT ioctls.
>>
>> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
>> in drivers, but, in the meantime, the core takes care of converting
>> {S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
>> still work if the userspace app/lib uses the new ioctls.
>> The conversion is also done the other around to allow userspace
>> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
>> _ext_ hooks.
>>
>> Signed-off-by: Boris Brezillon <[e-mail]>
>> Signed-off-by: Helen Koike <[e-mail]>
>> ---
>> Changes in v5:
>> - change sizes and reorder fields to avoid holes in the struct and make
>>   it the same for 32 and 64 bits
>> - removed __attribute__ ((packed)) from uapi structs
>> - Fix doc warning from make htmldocs
>> - Updated commit message with EXT_PIX prefix for the ioctls.
>>
>> Changes in v4:
>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>> - Add reserved fields
>> - Removed num_planes from struct v4l2_ext_pix_format
>> - Removed flag field from struct v4l2_ext_pix_format, since the only
>>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>>   where we can use modifiers, or add it back later through the reserved
>>   bits.
>> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>>   != MOD_INVALID
>> - Fix type assignment in v4l_g_fmt_ext_pix()
>> - Rebased on top of media/master (post 5.8-rc1)
>>
>> Changes in v3:
>> - Rebased on top of media/master (post 5.4-rc1)
>>
>> Changes in v2:
>> - Move the modifier in v4l2_ext_format (was formerly placed in
>>   v4l2_ext_plane)
>> - Fix a few bugs in the converters and add a strict parameter to
>>   allow conversion of uninitialized/mis-initialized objects
>> ---
>>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>>  include/media/v4l2-ioctl.h           |  34 ++
>>  include/uapi/linux/videodev2.h       |  56 +++
>>  4 files changed, 615 insertions(+), 81 deletions(-)
>>
> 
> <snip>
> 
>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>> index c7b70ff53bc1d..7123c6a4d9569 100644
>> --- a/include/uapi/linux/videodev2.h
>> +++ b/include/uapi/linux/videodev2.h
>> @@ -2254,6 +2254,58 @@ struct v4l2_pix_format_mplane {
>>  	__u8				reserved[7];
>>  } __attribute__ ((packed));
>>  
>> +/**
>> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
>> + * @sizeimage:		maximum size in bytes required for data, for which
>> + *			this plane will be used.
>> + *			Should be set to zero for unused planes.
>> + * @bytesperline:	distance in bytes between the leftmost pixels in two
>> + *			adjacent lines
>> + * @reserved:		extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_plane_ext_pix_format {
>> +	__u32 sizeimage;
>> +	__u32 bytesperline;
>> +	__u32 reserved;
> 
> I'd use reserved[4] here.
> 
>> +};
>> +
>> +/**
>> + * struct v4l2_ext_pix_format - extended single/multiplanar format definition
>> + * @type:		type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
>> + *			V4L2_BUF_TYPE_VIDEO_OUTPUT
>> + * @width:		image width in pixels
>> + * @height:		image height in pixels
>> + * @field:		enum v4l2_field; field order (for interlaced video)
>> + * @pixelformat:	little endian four character code (fourcc)
>> + * @modifier:		modifier applied to the format (used for tiled formats
>> + *			and other kind of HW-specific formats, like compressed
>> + *			formats)
> 
> This should refer to the drm.h header since we're reusing the DRM modifiers.
> 
>> + * @colorspace:		enum v4l2_colorspace; supplemental to pixelformat
>> + * @plane_fmt:		per-plane information
>> + * @ycbcr_enc:		enum v4l2_ycbcr_encoding, Y'CbCr encoding
>> + * @hsv_enc:		enum v4l2_hsv_encoding, HSV encoding
>> + * @quantization:	enum v4l2_quantization, colorspace quantization
>> + * @xfer_func:		enum v4l2_xfer_func, colorspace transfer function
>> + * @reserved:		extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_ext_pix_format {
>> +	__u32 type;
>> +	__u32 width;
>> +	__u32 height;
>> +	__u32 field;
>> +	__u64 modifier;
>> +	__u32 pixelformat;
>> +	__u32 colorspace;
>> +	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
>> +	union {
>> +		__u32 ycbcr_enc;
>> +		__u32 hsv_enc;
>> +	};
>> +	__u32 quantization;
>> +	__u32 xfer_func;
> 
> I'd reorder this:
> 
> 	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
> 	__u32 pixelformat;
> 	__u32 colorspace;
> 	__u32 xfer_func;
> 	union {
> 		__u32 ycbcr_enc;
> 		__u32 hsv_enc;
> 	};
> 	__u32 quantization;
> 
> The reason for reordering is that I like to keep the colorimetry fields in
> that order since that is how these fields are processed mathematically: you
> apply the colorspace matrix first, then the transfer function, then optionally
> convert to Y'CbCr or HSV and finally you quantize the result.
> 
> There is also a __u32 flags; field missing (needed for V4L2_PIX_FMT_FLAG_PREMUL_ALPHA
> and for the upcoming CSC support).

We discussed this on v4 https://patchwork.linuxtv.org/project/linux-media/cover/[e-mail]/

The idea is to replace V4L2_PIX_FMT_FLAG_PREMUL_ALPHA by a modifier, and this API
won't need CSC, since the colorspace fields are read-write.

Regards,
Helen

> 
>> +	__u32 reserved[9];
>> +};
>> +
>>  /**
>>   * struct v4l2_sdr_format - SDR format definition
>>   * @pixelformat:	little endian four character code (fourcc)
>> @@ -2571,6 +2623,10 @@ struct v4l2_create_buffers {
>>  
>>  #define VIDIOC_QUERY_EXT_CTRL	_IOWR('V', 103, struct v4l2_query_ext_ctrl)
>>  
>> +#define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
>> +#define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
>> +#define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
>> +
>>  /* Reminder: when adding new ioctls please add support for them to
>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>  
>>
> 
> Regards,
> 
> 	Hans
>

Tomasz Figa Oct. 2, 2020, 7:49 p.m. UTC | #4

Hi Helen,

On Tue, Aug 04, 2020 at 04:29:33PM -0300, Helen Koike wrote:
> This is part of the multiplanar and singleplanar unification process.
> v4l2_ext_pix_format is supposed to work for both cases.
> 
> We also add the concept of modifiers already employed in DRM to expose
> HW-specific formats (like tiled or compressed formats) and allow
> exchanging this information with the DRM subsystem in a consistent way.
> 
> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
> v4l2_ext_format, other types will be rejected if you use the
> {G,S,TRY}_EXT_PIX_FMT ioctls.
> 
> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
> in drivers, but, in the meantime, the core takes care of converting
> {S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
> still work if the userspace app/lib uses the new ioctls.
> The conversion is also done the other around to allow userspace
> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
> _ext_ hooks.
> 

Thank you for the patch. Please see my comments inline.

> Signed-off-by: Boris Brezillon <[e-mail]>
> Signed-off-by: Helen Koike <[e-mail]>
> ---
> Changes in v5:
> - change sizes and reorder fields to avoid holes in the struct and make
>   it the same for 32 and 64 bits
> - removed __attribute__ ((packed)) from uapi structs
> - Fix doc warning from make htmldocs
> - Updated commit message with EXT_PIX prefix for the ioctls.
> 
> Changes in v4:
> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
> - Add reserved fields
> - Removed num_planes from struct v4l2_ext_pix_format
> - Removed flag field from struct v4l2_ext_pix_format, since the only
>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>   where we can use modifiers, or add it back later through the reserved
>   bits.
> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>   != MOD_INVALID
> - Fix type assignment in v4l_g_fmt_ext_pix()
> - Rebased on top of media/master (post 5.8-rc1)
> 
> Changes in v3:
> - Rebased on top of media/master (post 5.4-rc1)
> 
> Changes in v2:
> - Move the modifier in v4l2_ext_format (was formerly placed in
>   v4l2_ext_plane)
> - Fix a few bugs in the converters and add a strict parameter to
>   allow conversion of uninitialized/mis-initialized objects
> ---
>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>  include/media/v4l2-ioctl.h           |  34 ++
>  include/uapi/linux/videodev2.h       |  56 +++
>  4 files changed, 615 insertions(+), 81 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> index a593ea0598b55..e1829906bc086 100644
> --- a/drivers/media/v4l2-core/v4l2-dev.c
> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> @@ -607,25 +607,37 @@ static void determine_valid_ioctls(struct video_device *vdev)
>  			set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
>  		if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
>  			       ops->vidioc_g_fmt_vid_cap_mplane ||
> +			       ops->vidioc_g_ext_pix_fmt_vid_cap ||
>  			       ops->vidioc_g_fmt_vid_overlay)) ||
>  		    (is_tx && (ops->vidioc_g_fmt_vid_out ||
>  			       ops->vidioc_g_fmt_vid_out_mplane ||
> -			       ops->vidioc_g_fmt_vid_out_overlay)))
> +			       ops->vidioc_g_ext_pix_fmt_vid_out ||
> +			       ops->vidioc_g_fmt_vid_out_overlay))) {
>  			 set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
> +			 set_bit(_IOC_NR(VIDIOC_G_EXT_PIX_FMT), valid_ioctls);

Is it expected to allow the new ioctls for drivers which implement the old
vid_out_overlay callbacks?

> +		}
>  		if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
>  			       ops->vidioc_s_fmt_vid_cap_mplane ||
> +			       ops->vidioc_s_ext_pix_fmt_vid_cap ||
>  			       ops->vidioc_s_fmt_vid_overlay)) ||
>  		    (is_tx && (ops->vidioc_s_fmt_vid_out ||
>  			       ops->vidioc_s_fmt_vid_out_mplane ||
> -			       ops->vidioc_s_fmt_vid_out_overlay)))
> +			       ops->vidioc_s_ext_pix_fmt_vid_out ||
> +			       ops->vidioc_s_fmt_vid_out_overlay))) {
>  			 set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
> +			 set_bit(_IOC_NR(VIDIOC_S_EXT_PIX_FMT), valid_ioctls);
> +		}
>  		if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
>  			       ops->vidioc_try_fmt_vid_cap_mplane ||
> +			       ops->vidioc_try_ext_pix_fmt_vid_cap ||
>  			       ops->vidioc_try_fmt_vid_overlay)) ||
>  		    (is_tx && (ops->vidioc_try_fmt_vid_out ||
>  			       ops->vidioc_try_fmt_vid_out_mplane ||
> -			       ops->vidioc_try_fmt_vid_out_overlay)))
> +			       ops->vidioc_try_ext_pix_fmt_vid_out ||
> +			       ops->vidioc_try_fmt_vid_out_overlay))) {
>  			 set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
> +			 set_bit(_IOC_NR(VIDIOC_TRY_EXT_PIX_FMT), valid_ioctls);
> +		}
>  		SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
>  		SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
>  		SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
> @@ -682,8 +694,11 @@ static void determine_valid_ioctls(struct video_device *vdev)
>  		/* touch specific ioctls */
>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_vid_cap);
>  		SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_vid_cap);
> +		SET_VALID_IOCTL(ops, VIDIOC_G_EXT_PIX_FMT, vidioc_g_fmt_vid_cap);
>  		SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_vid_cap);
> +		SET_VALID_IOCTL(ops, VIDIOC_S_EXT_PIX_FMT, vidioc_s_fmt_vid_cap);
>  		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_vid_cap);
> +		SET_VALID_IOCTL(ops, VIDIOC_TRY_EXT_PIX_FMT, vidioc_try_fmt_vid_cap);
>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
>  		SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index a556880f225a5..14a0def50f8ea 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -17,6 +17,8 @@
>  
>  #include <linux/videodev2.h>
>  
> +#include <drm/drm_fourcc.h>
> +
>  #include <media/v4l2-common.h>
>  #include <media/v4l2-ioctl.h>
>  #include <media/v4l2-ctrls.h>
> @@ -378,6 +380,27 @@ static void v4l_print_format(const void *arg, bool write_only)
>  	}
>  }
>  
> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
> +{
> +	const struct v4l2_ext_pix_format *pix = arg;
> +	unsigned int i;
> +
> +	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
> +		prt_names(pix->type, v4l2_type_names),
> +		pix->width, pix->height,
> +		(pix->pixelformat & 0xff),
> +		(pix->pixelformat >>  8) & 0xff,
> +		(pix->pixelformat >> 16) & 0xff,
> +		(pix->pixelformat >> 24) & 0xff,
> +		pix->modifier, prt_names(pix->field, v4l2_field_names),
> +		pix->colorspace, pix->ycbcr_enc,
> +		pix->quantization, pix->xfer_func);
> +	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)

This is going to print 8 lines every time. Maybe we could skip 0-sized
planes or something?

> +		pr_debug("plane %u: bytesperline=%u sizeimage=%u\n",
> +			 i, pix->plane_fmt[i].bytesperline,
> +			 pix->plane_fmt[i].sizeimage);
> +}
> +
>  static void v4l_print_framebuffer(const void *arg, bool write_only)
>  {
>  	const struct v4l2_framebuffer *p = arg;
> @@ -964,11 +987,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>  	switch (type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>  		if ((is_vid || is_tch) && is_rx &&
> -		    (ops->vidioc_g_fmt_vid_cap || ops->vidioc_g_fmt_vid_cap_mplane))
> +		    (ops->vidioc_g_fmt_vid_cap ||
> +		     ops->vidioc_g_ext_pix_fmt_vid_cap ||
> +		     ops->vidioc_g_fmt_vid_cap_mplane))
>  			return 0;
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -		if ((is_vid || is_tch) && is_rx && ops->vidioc_g_fmt_vid_cap_mplane)
> +		if ((is_vid || is_tch) && is_rx &&
> +		    (ops->vidioc_g_fmt_vid_cap_mplane ||
> +		     ops->vidioc_g_ext_pix_fmt_vid_cap))
>  			return 0;
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
> @@ -977,11 +1004,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>  		if (is_vid && is_tx &&
> -		    (ops->vidioc_g_fmt_vid_out || ops->vidioc_g_fmt_vid_out_mplane))
> +		    (ops->vidioc_g_fmt_vid_out ||
> +		     ops->vidioc_g_ext_pix_fmt_vid_out ||
> +		     ops->vidioc_g_fmt_vid_out_mplane))
>  			return 0;
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -		if (is_vid && is_tx && ops->vidioc_g_fmt_vid_out_mplane)
> +		if (is_vid && is_tx &&
> +		    (ops->vidioc_g_ext_pix_fmt_vid_out ||
> +		     ops->vidioc_g_fmt_vid_out_mplane))
>  			return 0;
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
> @@ -1061,6 +1092,134 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>  	       sizeof(fmt->fmt.pix) - offset);
>  }
>  
> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> +				  struct v4l2_format *f, bool mplane_cap,
> +				  bool strict)
> +{
> +	const struct v4l2_plane_ext_pix_format *pe;
> +	struct v4l2_plane_pix_format *p;
> +	unsigned int i;
> +
> +	memset(f, 0, sizeof(*f));
> +
> +	/*
> +	 * Make sure no modifier is required before doing the
> +	 * conversion.
> +	 */
> +	if (e->modifier && strict &&

Do we need the explicit check for e->modifier != 0 if we have to check for
the 2 specific values below anyway?

> +	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
> +	    e->modifier != DRM_FORMAT_MOD_INVALID)
> +		return -EINVAL;
> +
> +	if (!e->plane_fmt[0].sizeimage && strict)
> +		return -EINVAL;

Why is this incorrect?

> +
> +	if (e->plane_fmt[1].sizeimage && !mplane_cap && strict)
> +		return 0;

Again this seems to be different from what we discussed before. In the ext
API, the planes would mean color planes and would be all filled in with the
right values. So for this conversion, if !mplane cap, then we should check
if bytesperline of planes >= 1 match the format definition and use the sum
of all planes sizeimage as the sizeimage of the legacy struct.

> +
> +	if (!mplane_cap) {
> +		f->fmt.pix.width = e->width;
> +		f->fmt.pix.height = e->height;
> +		f->fmt.pix.pixelformat = e->pixelformat;
> +		f->fmt.pix.field = e->field;
> +		f->fmt.pix.colorspace = e->colorspace;
> +		f->fmt.pix.ycbcr_enc = e->ycbcr_enc;
> +		f->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +		f->fmt.pix.quantization = e->quantization;
> +		pe = &e->plane_fmt[0];
> +		f->fmt.pix.bytesperline = pe->bytesperline;
> +		f->fmt.pix.sizeimage = pe->sizeimage;

See above for how these two should be filled in.

> +		f->type = e->type;
> +		return 0;
> +	}
> +
> +	f->fmt.pix_mp.width = e->width;
> +	f->fmt.pix_mp.height = e->height;
> +	f->fmt.pix_mp.pixelformat = e->pixelformat;
> +	f->fmt.pix_mp.field = e->field;
> +	f->fmt.pix_mp.colorspace = e->colorspace;
> +	f->fmt.pix_mp.ycbcr_enc = e->ycbcr_enc;
> +	f->fmt.pix_mp.quantization = e->quantization;
> +	if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> +		f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
> +	else
> +		f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
> +
> +	for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> +		pe = &e->plane_fmt[i];
> +		p = &f->fmt.pix_mp.plane_fmt[i];
> +		p->bytesperline = pe->bytesperline;
> +		p->sizeimage = pe->sizeimage;

This is similar to the above, but the behavior depends on whether the
pixelformat is an M or non-M variant.

> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_ext_pix_format_to_format);
> +
> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
> +				  struct v4l2_ext_pix_format *e, bool strict)

In other functions "ef" is used for the extended format. Let's make it
consistent.

> +{
> +	const struct v4l2_plane_pix_format *p;
> +	struct v4l2_plane_ext_pix_format *pe;
> +	unsigned int i;
> +
> +	memset(e, 0, sizeof(*e));
> +
> +	switch (f->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +		e->width = f->fmt.pix.width;
> +		e->height = f->fmt.pix.height;
> +		e->pixelformat = f->fmt.pix.pixelformat;
> +		e->field = f->fmt.pix.field;
> +		e->colorspace = f->fmt.pix.colorspace;
> +		if (f->fmt.pix.flags)
> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
> +				f->fmt.pix.flags);

Would it make sense to print something like video node name and/or function
name to explain where this warning comes from?

> +		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
> +		e->quantization = f->fmt.pix.quantization;

Missing xfer_func?

> +		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
> +		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;

This doesn't look right. In the ext API we expected the planes to describe
color planes, which means that bytesperline needs to be computed for planes
>= 1 and sizeimage replaced with per-plane sizes, according to the
>pixelformat.

> +		e->type = f->type;
> +		break;
> +
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +		if ((f->fmt.pix_mp.num_planes > VIDEO_MAX_PLANES ||
> +		     !f->fmt.pix_mp.num_planes) && strict)
> +			return -EINVAL;
> +
> +		e->width = f->fmt.pix_mp.width;
> +		e->height = f->fmt.pix_mp.height;
> +		e->pixelformat = f->fmt.pix_mp.pixelformat;
> +		e->field = f->fmt.pix_mp.field;
> +		e->colorspace = f->fmt.pix_mp.colorspace;
> +		if (f->fmt.pix.flags)
> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
> +				f->fmt.pix.flags);
> +		e->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
> +		e->quantization = f->fmt.pix_mp.quantization;

Missing xfer_func?

> +		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +		else
> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +
> +		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> +			pe = &e->plane_fmt[i];
> +			p = &f->fmt.pix_mp.plane_fmt[i];
> +			pe->bytesperline = p->bytesperline;
> +			pe->sizeimage = p->sizeimage;
> +		}

Same here. A blind copy is not enough. For non-M formats, the plane
parameters need to be filled according to the pixelformat.

> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
> +
>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -1564,6 +1723,38 @@ static void v4l_pix_format_touch(struct v4l2_pix_format *p)
>  	p->xfer_func = 0;
>  }
>  
> +static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +			     struct file *file, void *fh,
> +			     struct v4l2_format *f)
> +{
> +	struct v4l2_ext_pix_format ef;
> +	int ret;
> +
> +	switch (f->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +		ef.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +		ret = ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, &ef);
> +		break;
> +
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +		ef.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +		ret = ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, &ef);
> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_ext_pix_format_to_format(&ef, f,
> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +					     true);
> +}
> +
>  static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -1600,17 +1791,27 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>  
>  	switch (p->type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -		if (unlikely(!ops->vidioc_g_fmt_vid_cap))
> -			break;
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		if (vfd->vfl_type == VFL_TYPE_TOUCH)
> -			v4l_pix_format_touch(&p->fmt.pix);
> -		return ret;
> +		if (ops->vidioc_g_fmt_vid_cap) {
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +				v4l_pix_format_touch(&p->fmt.pix);
> +			return ret;
> +		} else if (ops->vidioc_g_ext_pix_fmt_vid_cap) {
> +			ret = v4l_g_fmt_ext_pix(ops, file, fh, p);
> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +				v4l_pix_format_touch(&p->fmt.pix);
> +			return ret;
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -		return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> +		if (ops->vidioc_g_fmt_vid_cap_mplane)
> +			return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
> +		else if (ops->vidioc_g_ext_pix_fmt_vid_cap)
> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>  		return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
>  	case V4L2_BUF_TYPE_VBI_CAPTURE:
> @@ -1618,15 +1819,22 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>  	case V4L2_BUF_TYPE_SLICED_VBI_CAPTURE:
>  		return ops->vidioc_g_fmt_sliced_vbi_cap(file, fh, arg);
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -		if (unlikely(!ops->vidioc_g_fmt_vid_out))
> -			break;
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		return ret;
> +		if (ops->vidioc_g_fmt_vid_out) {
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			return ret;
> +		} else if (ops->vidioc_g_ext_pix_fmt_vid_out) {
> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -		return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> +		if (ops->vidioc_g_fmt_vid_out_mplane)
> +			return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
> +		else if (ops->vidioc_g_ext_pix_fmt_vid_out)
> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>  		return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
>  	case V4L2_BUF_TYPE_VBI_OUTPUT:
> @@ -1645,6 +1853,76 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>  	return -EINVAL;
>  }
>  
> +static int v4l_g_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +			     struct file *file, void *fh, void *arg)
> +{
> +	struct v4l2_ext_pix_format *ef = arg;
> +	struct v4l2_format f = {
> +		.type = ef->type,
> +	};
> +	int ret;
> +
> +	ret = check_fmt(file, ef->type);
> +	if (ret)
> +		return ret;
> +
> +	memset(ef, 0, sizeof(*ef));
> +	ef->type = f.type;
> +
> +	switch (f.type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +		if (ops->vidioc_g_ext_pix_fmt_vid_cap)
> +			return ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, ef);
> +		break;
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +		if (ops->vidioc_g_ext_pix_fmt_vid_out)
> +			return ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, ef);
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	ret = v4l_g_fmt(ops, file, fh, &f);
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
> +static int v4l_s_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +			     struct file *file, void *fh,
> +			     struct v4l2_format *f)
> +{
> +	struct v4l2_ext_pix_format ef;
> +	int ret;
> +
> +	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
> +	if (ret)
> +		return ret;
> +
> +	switch (f->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +		ret = ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, &ef);
> +		break;
> +
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +		ret = ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, &ef);
> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_ext_pix_format_to_format(&ef, f,
> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +					     true);
> +}
> +
>  static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -1663,23 +1941,31 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>  
>  	switch (p->type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -		if (unlikely(!ops->vidioc_s_fmt_vid_cap))
> +		if (ops->vidioc_s_fmt_vid_cap) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix);
> +			ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
> +			ret = v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +		} else {
>  			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix);
> -		ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +		}
> +
>  		if (vfd->vfl_type == VFL_TYPE_TOUCH)
>  			v4l_pix_format_touch(&p->fmt.pix);
>  		return ret;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -		if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -					  bytesperline);
> -		return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
> +		if (ops->vidioc_s_fmt_vid_cap_mplane) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +						  bytesperline);
> +			return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>  		if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
>  			break;
> @@ -1696,21 +1982,27 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>  		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>  		return ops->vidioc_s_fmt_sliced_vbi_cap(file, fh, arg);
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -		if (unlikely(!ops->vidioc_s_fmt_vid_out))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix);
> -		ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		return ret;
> +		if (ops->vidioc_s_fmt_vid_out) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix);
> +			ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			return ret;
> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -		if (unlikely(!ops->vidioc_s_fmt_vid_out_mplane))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -					  bytesperline);
> -		return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
> +		if (ops->vidioc_s_fmt_vid_out_mplane) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +						  bytesperline);
> +			return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>  		if (unlikely(!ops->vidioc_s_fmt_vid_out_overlay))
>  			break;
> @@ -1750,6 +2042,82 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>  	return -EINVAL;
>  }
>  
> +static int v4l_s_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +			     struct file *file, void *fh, void *arg)
> +{
> +	struct video_device *vfd = video_devdata(file);
> +	struct v4l2_ext_pix_format *ef = arg;
> +	struct v4l2_format f;
> +	int ret;
> +
> +	ret = check_fmt(file, ef->type);
> +	if (ret)
> +		return ret;
> +

Should we zero the reserved fields as it's done with the current structs?

> +	switch (ef->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +		if (ops->vidioc_s_ext_pix_fmt_vid_cap)
> +			return ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, ef);
> +		break;
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +		if (ops->vidioc_s_ext_pix_fmt_vid_out)
> +			return ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, ef);
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	ret = v4l2_ext_pix_format_to_format(ef, &f,
> +					    vfd->device_caps &
> +					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +					     V4L2_CAP_VIDEO_M2M_MPLANE),
> +					    false);
> +	if (ret)
> +		return ret;
> +
> +	ret = v4l_s_fmt(ops, file, fh, &f);
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
> +static int v4l_try_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
> +			       struct file *file, void *fh,
> +			       struct v4l2_format *f)
> +{
> +	struct v4l2_ext_pix_format ef;
> +	int ret;
> +
> +	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
> +	if (ret)
> +		return ret;
> +
> +	switch (f->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +		ret = ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh, &ef);
> +		break;
> +
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +		ret = ops->vidioc_try_ext_pix_fmt_vid_out(file, fh, &ef);
> +		break;
> +
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_ext_pix_format_to_format(&ef, f,
> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
> +					     true);
> +}
> +
> +
>  static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -1765,23 +2133,32 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>  
>  	switch (p->type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> -		if (unlikely(!ops->vidioc_try_fmt_vid_cap))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix);
> -		ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		if (vfd->vfl_type == VFL_TYPE_TOUCH)
> -			v4l_pix_format_touch(&p->fmt.pix);
> -		return ret;
> +		if (ops->vidioc_try_fmt_vid_cap) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix);
> +			ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +				v4l_pix_format_touch(&p->fmt.pix);
> +			return ret;
> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +			ret = v4l_try_fmt_ext_pix(ops, file, fh, p);
> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
> +				v4l_pix_format_touch(&p->fmt.pix);
> +			return ret;

Should the 3 lines above be outside of the if/else block?

> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> -		if (unlikely(!ops->vidioc_try_fmt_vid_cap_mplane))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -					  bytesperline);
> -		return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
> +		if (ops->vidioc_try_fmt_vid_cap_mplane) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +						  bytesperline);
> +			return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>  		if (unlikely(!ops->vidioc_try_fmt_vid_overlay))
>  			break;
> @@ -1798,21 +2175,27 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>  		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>  		return ops->vidioc_try_fmt_sliced_vbi_cap(file, fh, arg);
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> -		if (unlikely(!ops->vidioc_try_fmt_vid_out))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix);
> -		ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
> -		/* just in case the driver zeroed it again */
> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> -		return ret;
> +		if (ops->vidioc_try_fmt_vid_out) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix);
> +			ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
> +			/* just in case the driver zeroed it again */
> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
> +			return ret;
> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> -		if (unlikely(!ops->vidioc_try_fmt_vid_out_mplane))
> -			break;
> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> -					  bytesperline);
> -		return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
> +		if (ops->vidioc_try_fmt_vid_out_mplane) {
> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
> +						  bytesperline);
> +			return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
> +		}
> +		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>  		if (unlikely(!ops->vidioc_try_fmt_vid_out_overlay))
>  			break;
> @@ -1852,6 +2235,49 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>  	return -EINVAL;
>  }
>  
> +static int v4l_try_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
> +			       struct file *file, void *fh, void *arg)
> +{
> +	struct video_device *vfd = video_devdata(file);
> +	struct v4l2_ext_pix_format *ef = arg;
> +	struct v4l2_format f;
> +	int ret;
> +
> +	ret = check_fmt(file, ef->type);
> +	if (ret)
> +		return ret;

Should we zero the reserved fields as it's done with the current structs?

> +
> +	switch (ef->type) {
> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +		if (ops->vidioc_try_ext_pix_fmt_vid_cap)
> +			return ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh,
> +								   ef);
> +		break;
> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +		if (ops->vidioc_try_ext_pix_fmt_vid_out)
> +			return ops->vidioc_try_ext_pix_fmt_vid_out(file, fh,
> +								   ef);
> +		break;
> +	default:
> +		return -EINVAL;
> +	}
> +
> +	ret = v4l2_ext_pix_format_to_format(ef, &f,
> +					    vfd->device_caps &
> +					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
> +					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
> +					     V4L2_CAP_VIDEO_M2M_MPLANE),
> +					    false);
> +	if (ret)
> +		return ret;
> +
> +	ret = v4l_try_fmt(ops, file, fh, &f);
> +	if (ret)
> +		return ret;
> +
> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
> +}
> +
>  static int v4l_streamon(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> @@ -2771,7 +3197,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>  	IOCTL_INFO(VIDIOC_QUERYCAP, v4l_querycap, v4l_print_querycap, 0),
>  	IOCTL_INFO(VIDIOC_ENUM_FMT, v4l_enum_fmt, v4l_print_fmtdesc, 0),
>  	IOCTL_INFO(VIDIOC_G_FMT, v4l_g_fmt, v4l_print_format, 0),
> +	IOCTL_INFO(VIDIOC_G_EXT_PIX_FMT, v4l_g_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>  	IOCTL_INFO(VIDIOC_S_FMT, v4l_s_fmt, v4l_print_format, INFO_FL_PRIO),
> +	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>  	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>  	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
>  	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
> @@ -2818,6 +3246,7 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>  	IOCTL_INFO(VIDIOC_S_JPEGCOMP, v4l_stub_s_jpegcomp, v4l_print_jpegcompression, INFO_FL_PRIO),
>  	IOCTL_INFO(VIDIOC_QUERYSTD, v4l_querystd, v4l_print_std, 0),
>  	IOCTL_INFO(VIDIOC_TRY_FMT, v4l_try_fmt, v4l_print_format, 0),
> +	IOCTL_INFO(VIDIOC_TRY_EXT_PIX_FMT, v4l_try_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>  	IOCTL_INFO(VIDIOC_ENUMAUDIO, v4l_stub_enumaudio, v4l_print_audio, INFO_FL_CLEAR(v4l2_audio, index)),
>  	IOCTL_INFO(VIDIOC_ENUMAUDOUT, v4l_stub_enumaudout, v4l_print_audioout, INFO_FL_CLEAR(v4l2_audioout, index)),
>  	IOCTL_INFO(VIDIOC_G_PRIORITY, v4l_g_priority, v4l_print_u32, 0),
> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
> index 86878fba332b0..8bbcb74d8ee31 100644
> --- a/include/media/v4l2-ioctl.h
> +++ b/include/media/v4l2-ioctl.h
> @@ -48,11 +48,17 @@ struct v4l2_fh;
>   * @vidioc_g_fmt_vid_cap: pointer to the function that implements
>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *	in single plane mode
> + * @vidioc_g_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *	capture
>   * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_g_fmt_vid_out: pointer to the function that implements
>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *	in single plane mode
> + * @vidioc_g_ext_pix_fmt_vid_out: pointer to the function that implements
> + *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *	out
>   * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>   * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
> @@ -82,11 +88,16 @@ struct v4l2_fh;
>   * @vidioc_s_fmt_vid_cap: pointer to the function that implements
>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *	in single plane mode
> + * @vidioc_s_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
> + *	capture
>   * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_s_fmt_vid_out: pointer to the function that implements
>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *	in single plane mode
> + * @vidioc_s_ext_pix_fmt_vid_out: pointer to the function that implements
> + *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>   * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>   * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
> @@ -116,11 +127,16 @@ struct v4l2_fh;
>   * @vidioc_try_fmt_vid_cap: pointer to the function that implements
>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
>   *	in single plane mode
> + * @vidioc_try_ext_pix_fmt_vid_cap: pointer to the function that implements
> + *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for
> +	video capture
>   * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   * @vidioc_try_fmt_vid_out: pointer to the function that implements
>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
>   *	in single plane mode
> + * @vidioc_try_ext_pix_fmt_vid_out: pointer to the function that implements
> + *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>   * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>   *	output
> @@ -319,10 +335,14 @@ struct v4l2_ioctl_ops {
>  	/* VIDIOC_G_FMT handlers */
>  	int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
>  				    struct v4l2_format *f);
> +	int (*vidioc_g_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +					    struct v4l2_ext_pix_format *f);
>  	int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
>  					struct v4l2_format *f);
>  	int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
>  				    struct v4l2_format *f);
> +	int (*vidioc_g_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +					    struct v4l2_ext_pix_format *f);
>  	int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
>  					    struct v4l2_format *f);
>  	int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -349,10 +369,14 @@ struct v4l2_ioctl_ops {
>  	/* VIDIOC_S_FMT handlers */
>  	int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
>  				    struct v4l2_format *f);
> +	int (*vidioc_s_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +					    struct v4l2_ext_pix_format *f);
>  	int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
>  					struct v4l2_format *f);
>  	int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
>  				    struct v4l2_format *f);
> +	int (*vidioc_s_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +					    struct v4l2_ext_pix_format *f);
>  	int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
>  					    struct v4l2_format *f);
>  	int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -379,10 +403,14 @@ struct v4l2_ioctl_ops {
>  	/* VIDIOC_TRY_FMT handlers */
>  	int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
>  				      struct v4l2_format *f);
> +	int (*vidioc_try_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
> +					      struct v4l2_ext_pix_format *f);
>  	int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
>  					  struct v4l2_format *f);
>  	int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
>  				      struct v4l2_format *f);
> +	int (*vidioc_try_ext_pix_fmt_vid_out)(struct file *file, void *fh,
> +					      struct v4l2_ext_pix_format *f);
>  	int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
>  					     struct v4l2_format *f);
>  	int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
> @@ -724,6 +752,12 @@ long int video_usercopy(struct file *file, unsigned int cmd,
>  long int video_ioctl2(struct file *file,
>  		      unsigned int cmd, unsigned long int arg);
>  
> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
> +				  struct v4l2_ext_pix_format *e, bool strict);
> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> +				  struct v4l2_format *f,
> +				  bool mplane_cap, bool strict);
> +
>  /*
>   * The user space interpretation of the 'v4l2_event' differs
>   * based on the 'time_t' definition on 32-bit architectures, so
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index c7b70ff53bc1d..7123c6a4d9569 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -2254,6 +2254,58 @@ struct v4l2_pix_format_mplane {
>  	__u8				reserved[7];
>  } __attribute__ ((packed));
>  
> +/**
> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
> + * @sizeimage:		maximum size in bytes required for data, for which
> + *			this plane will be used.
> + *			Should be set to zero for unused planes.
> + * @bytesperline:	distance in bytes between the leftmost pixels in two
> + *			adjacent lines
> + * @reserved:		extra space reserved for future fields, must be set to 0
> + */
> +struct v4l2_plane_ext_pix_format {

nit: Maybe v4l2_ext_plane_pix_format, since the struct describes an
ext variant of a plane_pix_format?

> +	__u32 sizeimage;
> +	__u32 bytesperline;
> +	__u32 reserved;
> +};

Actually, this seems to be exactly the same as the existing
v4l2_plane_pix_format, except actually having less reserved space. Could it
make sense to just reuse the existing struct?

Best regards,
Tomasz

Helen Mae Koike Fornazier Nov. 14, 2020, 2:21 p.m. UTC | #5

Hi Tomasz,

On 10/2/20 4:49 PM, Tomasz Figa wrote:
> Hi Helen,
> 
> On Tue, Aug 04, 2020 at 04:29:33PM -0300, Helen Koike wrote:
>> This is part of the multiplanar and singleplanar unification process.
>> v4l2_ext_pix_format is supposed to work for both cases.
>>
>> We also add the concept of modifiers already employed in DRM to expose
>> HW-specific formats (like tiled or compressed formats) and allow
>> exchanging this information with the DRM subsystem in a consistent way.
>>
>> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
>> v4l2_ext_format, other types will be rejected if you use the
>> {G,S,TRY}_EXT_PIX_FMT ioctls.
>>
>> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
>> in drivers, but, in the meantime, the core takes care of converting
>> {S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
>> still work if the userspace app/lib uses the new ioctls.
>> The conversion is also done the other around to allow userspace
>> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
>> _ext_ hooks.
>>
> 
> Thank you for the patch. Please see my comments inline.

Thanks for reviewing.

> 
>> Signed-off-by: Boris Brezillon <[e-mail]>
>> Signed-off-by: Helen Koike <[e-mail]>
>> ---
>> Changes in v5:
>> - change sizes and reorder fields to avoid holes in the struct and make
>>   it the same for 32 and 64 bits
>> - removed __attribute__ ((packed)) from uapi structs
>> - Fix doc warning from make htmldocs
>> - Updated commit message with EXT_PIX prefix for the ioctls.
>>
>> Changes in v4:
>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>> - Add reserved fields
>> - Removed num_planes from struct v4l2_ext_pix_format
>> - Removed flag field from struct v4l2_ext_pix_format, since the only
>>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>>   where we can use modifiers, or add it back later through the reserved
>>   bits.
>> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>>   != MOD_INVALID
>> - Fix type assignment in v4l_g_fmt_ext_pix()
>> - Rebased on top of media/master (post 5.8-rc1)
>>
>> Changes in v3:
>> - Rebased on top of media/master (post 5.4-rc1)
>>
>> Changes in v2:
>> - Move the modifier in v4l2_ext_format (was formerly placed in
>>   v4l2_ext_plane)
>> - Fix a few bugs in the converters and add a strict parameter to
>>   allow conversion of uninitialized/mis-initialized objects
>> ---
>>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>>  include/media/v4l2-ioctl.h           |  34 ++
>>  include/uapi/linux/videodev2.h       |  56 +++
>>  4 files changed, 615 insertions(+), 81 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
>> index a593ea0598b55..e1829906bc086 100644
>> --- a/drivers/media/v4l2-core/v4l2-dev.c
>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
>> @@ -607,25 +607,37 @@ static void determine_valid_ioctls(struct video_device *vdev)
>>  			set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
>>  		if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
>>  			       ops->vidioc_g_fmt_vid_cap_mplane ||
>> +			       ops->vidioc_g_ext_pix_fmt_vid_cap ||
>>  			       ops->vidioc_g_fmt_vid_overlay)) ||
>>  		    (is_tx && (ops->vidioc_g_fmt_vid_out ||
>>  			       ops->vidioc_g_fmt_vid_out_mplane ||
>> -			       ops->vidioc_g_fmt_vid_out_overlay)))
>> +			       ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +			       ops->vidioc_g_fmt_vid_out_overlay))) {
>>  			 set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
>> +			 set_bit(_IOC_NR(VIDIOC_G_EXT_PIX_FMT), valid_ioctls);
> 
> Is it expected to allow the new ioctls for drivers which implement the old
> vid_out_overlay callbacks?

Thanks for noticing it, I don't think so, I'll update this in next version.

> 
>> +		}
>>  		if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
>>  			       ops->vidioc_s_fmt_vid_cap_mplane ||
>> +			       ops->vidioc_s_ext_pix_fmt_vid_cap ||
>>  			       ops->vidioc_s_fmt_vid_overlay)) ||
>>  		    (is_tx && (ops->vidioc_s_fmt_vid_out ||
>>  			       ops->vidioc_s_fmt_vid_out_mplane ||
>> -			       ops->vidioc_s_fmt_vid_out_overlay)))
>> +			       ops->vidioc_s_ext_pix_fmt_vid_out ||
>> +			       ops->vidioc_s_fmt_vid_out_overlay))) {
>>  			 set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
>> +			 set_bit(_IOC_NR(VIDIOC_S_EXT_PIX_FMT), valid_ioctls);
>> +		}
>>  		if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
>>  			       ops->vidioc_try_fmt_vid_cap_mplane ||
>> +			       ops->vidioc_try_ext_pix_fmt_vid_cap ||
>>  			       ops->vidioc_try_fmt_vid_overlay)) ||
>>  		    (is_tx && (ops->vidioc_try_fmt_vid_out ||
>>  			       ops->vidioc_try_fmt_vid_out_mplane ||
>> -			       ops->vidioc_try_fmt_vid_out_overlay)))
>> +			       ops->vidioc_try_ext_pix_fmt_vid_out ||
>> +			       ops->vidioc_try_fmt_vid_out_overlay))) {
>>  			 set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
>> +			 set_bit(_IOC_NR(VIDIOC_TRY_EXT_PIX_FMT), valid_ioctls);
>> +		}
>>  		SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
>>  		SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
>>  		SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
>> @@ -682,8 +694,11 @@ static void determine_valid_ioctls(struct video_device *vdev)
>>  		/* touch specific ioctls */
>>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_vid_cap);
>>  		SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_vid_cap);
>> +		SET_VALID_IOCTL(ops, VIDIOC_G_EXT_PIX_FMT, vidioc_g_fmt_vid_cap);
>>  		SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_vid_cap);
>> +		SET_VALID_IOCTL(ops, VIDIOC_S_EXT_PIX_FMT, vidioc_s_fmt_vid_cap);
>>  		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_vid_cap);
>> +		SET_VALID_IOCTL(ops, VIDIOC_TRY_EXT_PIX_FMT, vidioc_try_fmt_vid_cap);
>>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
>>  		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
>>  		SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index a556880f225a5..14a0def50f8ea 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -17,6 +17,8 @@
>>  
>>  #include <linux/videodev2.h>
>>  
>> +#include <drm/drm_fourcc.h>
>> +
>>  #include <media/v4l2-common.h>
>>  #include <media/v4l2-ioctl.h>
>>  #include <media/v4l2-ctrls.h>
>> @@ -378,6 +380,27 @@ static void v4l_print_format(const void *arg, bool write_only)
>>  	}
>>  }
>>  
>> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
>> +{
>> +	const struct v4l2_ext_pix_format *pix = arg;
>> +	unsigned int i;
>> +
>> +	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
>> +		prt_names(pix->type, v4l2_type_names),
>> +		pix->width, pix->height,
>> +		(pix->pixelformat & 0xff),
>> +		(pix->pixelformat >>  8) & 0xff,
>> +		(pix->pixelformat >> 16) & 0xff,
>> +		(pix->pixelformat >> 24) & 0xff,
>> +		pix->modifier, prt_names(pix->field, v4l2_field_names),
>> +		pix->colorspace, pix->ycbcr_enc,
>> +		pix->quantization, pix->xfer_func);
>> +	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
> 
> This is going to print 8 lines every time. Maybe we could skip 0-sized
> planes or something?

I'm already checking pix->plane_fmt[i].sizeimage in the loop, it shouldn't
print 8 lines every time.

> 
>> +		pr_debug("plane %u: bytesperline=%u sizeimage=%u\n",
>> +			 i, pix->plane_fmt[i].bytesperline,
>> +			 pix->plane_fmt[i].sizeimage);
>> +}
>> +
>>  static void v4l_print_framebuffer(const void *arg, bool write_only)
>>  {
>>  	const struct v4l2_framebuffer *p = arg;
>> @@ -964,11 +987,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>>  	switch (type) {
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>>  		if ((is_vid || is_tch) && is_rx &&
>> -		    (ops->vidioc_g_fmt_vid_cap || ops->vidioc_g_fmt_vid_cap_mplane))
>> +		    (ops->vidioc_g_fmt_vid_cap ||
>> +		     ops->vidioc_g_ext_pix_fmt_vid_cap ||
>> +		     ops->vidioc_g_fmt_vid_cap_mplane))
>>  			return 0;
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -		if ((is_vid || is_tch) && is_rx && ops->vidioc_g_fmt_vid_cap_mplane)
>> +		if ((is_vid || is_tch) && is_rx &&
>> +		    (ops->vidioc_g_fmt_vid_cap_mplane ||
>> +		     ops->vidioc_g_ext_pix_fmt_vid_cap))
>>  			return 0;
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>> @@ -977,11 +1004,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>>  		if (is_vid && is_tx &&
>> -		    (ops->vidioc_g_fmt_vid_out || ops->vidioc_g_fmt_vid_out_mplane))
>> +		    (ops->vidioc_g_fmt_vid_out ||
>> +		     ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +		     ops->vidioc_g_fmt_vid_out_mplane))
>>  			return 0;
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -		if (is_vid && is_tx && ops->vidioc_g_fmt_vid_out_mplane)
>> +		if (is_vid && is_tx &&
>> +		    (ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +		     ops->vidioc_g_fmt_vid_out_mplane))
>>  			return 0;
>>  		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>> @@ -1061,6 +1092,134 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>  	       sizeof(fmt->fmt.pix) - offset);
>>  }
>>  
>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>> +				  struct v4l2_format *f, bool mplane_cap,
>> +				  bool strict)
>> +{
>> +	const struct v4l2_plane_ext_pix_format *pe;
>> +	struct v4l2_plane_pix_format *p;
>> +	unsigned int i;
>> +
>> +	memset(f, 0, sizeof(*f));
>> +
>> +	/*
>> +	 * Make sure no modifier is required before doing the
>> +	 * conversion.
>> +	 */
>> +	if (e->modifier && strict &&
> 
> Do we need the explicit check for e->modifier != 0 if we have to check for
> the 2 specific values below anyway?

We don't, since DRM_FORMAT_MOD_LINEAR is zero.

But I wanted to make it explicit we don't support modifiers in this conversion.
But I can remove this check, no problem.

> 
>> +	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
>> +	    e->modifier != DRM_FORMAT_MOD_INVALID)
>> +		return -EINVAL;
>> +
>> +	if (!e->plane_fmt[0].sizeimage && strict)
>> +		return -EINVAL;
> 
> Why is this incorrect?

!sizeimage for the first plane means that there are no planes in ef.
strict is true if the result for the conversion should be returned to userspace
and it is not some internal handling.

So if there are no planes, we would return an incomplete v4l2_format struct
to userspace.

But this is not very clear, I'll improve this for the next version.

> 
>> +
>> +	if (e->plane_fmt[1].sizeimage && !mplane_cap && strict)
>> +		return 0;
> 
> Again this seems to be different from what we discussed before. In the ext
> API, the planes would mean color planes and would be all filled in with the
> right values. So for this conversion, if !mplane cap, then we should check
> if bytesperline of planes >= 1 match the format definition and use the sum
> of all planes sizeimage as the sizeimage of the legacy struct.

Agreed, I'll update for next version.

> 
>> +
>> +	if (!mplane_cap) {
>> +		f->fmt.pix.width = e->width;
>> +		f->fmt.pix.height = e->height;
>> +		f->fmt.pix.pixelformat = e->pixelformat;
>> +		f->fmt.pix.field = e->field;
>> +		f->fmt.pix.colorspace = e->colorspace;
>> +		f->fmt.pix.ycbcr_enc = e->ycbcr_enc;
>> +		f->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +		f->fmt.pix.quantization = e->quantization;
>> +		pe = &e->plane_fmt[0];
>> +		f->fmt.pix.bytesperline = pe->bytesperline;
>> +		f->fmt.pix.sizeimage = pe->sizeimage;
> 
> See above for how these two should be filled in.

Ack.

> 
>> +		f->type = e->type;
>> +		return 0;
>> +	}
>> +
>> +	f->fmt.pix_mp.width = e->width;
>> +	f->fmt.pix_mp.height = e->height;
>> +	f->fmt.pix_mp.pixelformat = e->pixelformat;
>> +	f->fmt.pix_mp.field = e->field;
>> +	f->fmt.pix_mp.colorspace = e->colorspace;
>> +	f->fmt.pix_mp.ycbcr_enc = e->ycbcr_enc;
>> +	f->fmt.pix_mp.quantization = e->quantization;
>> +	if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
>> +		f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>> +	else
>> +		f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
>> +
>> +	for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>> +		pe = &e->plane_fmt[i];
>> +		p = &f->fmt.pix_mp.plane_fmt[i];
>> +		p->bytesperline = pe->bytesperline;
>> +		p->sizeimage = pe->sizeimage;
> 
> This is similar to the above, but the behavior depends on whether the
> pixelformat is an M or non-M variant.

Ack.

> 
>> +	}
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_ext_pix_format_to_format);
>> +
>> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>> +				  struct v4l2_ext_pix_format *e, bool strict)
> 
> In other functions "ef" is used for the extended format. Let's make it
> consistent.

Ack.

> 
>> +{
>> +	const struct v4l2_plane_pix_format *p;
>> +	struct v4l2_plane_ext_pix_format *pe;
>> +	unsigned int i;
>> +
>> +	memset(e, 0, sizeof(*e));
>> +
>> +	switch (f->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +		e->width = f->fmt.pix.width;
>> +		e->height = f->fmt.pix.height;
>> +		e->pixelformat = f->fmt.pix.pixelformat;
>> +		e->field = f->fmt.pix.field;
>> +		e->colorspace = f->fmt.pix.colorspace;
>> +		if (f->fmt.pix.flags)
>> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
>> +				f->fmt.pix.flags);
> 
> Would it make sense to print something like video node name and/or function
> name to explain where this warning comes from?

I would need to update the function to receive this information, I can try but
I'm not sure if it is worthy.

> 
>> +		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
>> +		e->quantization = f->fmt.pix.quantization;
> 
> Missing xfer_func?

Yes, thanks for catching this.

> 
>> +		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
>> +		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
> 
> This doesn't look right. In the ext API we expected the planes to describe
> color planes, which means that bytesperline needs to be computed for planes
>> = 1 and sizeimage replaced with per-plane sizes, according to the
>> pixelformat.

Ack.

Just to be clear, even if we are using a planar format that isn't a V4L2_PIX_FMT_*M
variant, we should describe every plane separatly.

For instance, if V4L2_PIX_FMT_YVU420 is being used, then f->fmt.pix.bytesperline
will have data, and we need to calculate bytesperline for all 3 planes, so we'll fill
out:

f->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
f->plane_fmt[1].bytesperline = f->fmt.pix.bytesperline / hdiv;
f->plane_fmt[2].bytesperline = f->fmt.pix.bytesperline / hdiv;

I'll update this for the next version.

> 
>> +		e->type = f->type;
>> +		break;
>> +
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +		if ((f->fmt.pix_mp.num_planes > VIDEO_MAX_PLANES ||
>> +		     !f->fmt.pix_mp.num_planes) && strict)
>> +			return -EINVAL;
>> +
>> +		e->width = f->fmt.pix_mp.width;
>> +		e->height = f->fmt.pix_mp.height;
>> +		e->pixelformat = f->fmt.pix_mp.pixelformat;
>> +		e->field = f->fmt.pix_mp.field;
>> +		e->colorspace = f->fmt.pix_mp.colorspace;
>> +		if (f->fmt.pix.flags)
>> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
>> +				f->fmt.pix.flags);
>> +		e->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
>> +		e->quantization = f->fmt.pix_mp.quantization;
> 
> Missing xfer_func?

Yes, thanks for catching this.

> 
>> +		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
>> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +		else
>> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +
>> +		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>> +			pe = &e->plane_fmt[i];
>> +			p = &f->fmt.pix_mp.plane_fmt[i];
>> +			pe->bytesperline = p->bytesperline;
>> +			pe->sizeimage = p->sizeimage;
>> +		}
> 
> Same here. A blind copy is not enough. For non-M formats, the plane
> parameters need to be filled according to the pixelformat.


Right, following the idea above, we need a different handling if we
aren't using a M-variant of the pixelformat, and we also need to
convert the pixelformat from the M-variant to non-M-variant.

I'll also need to save that the original format was a
M-variant or not, so I can convert it back as expected.

I'll change this and submit for review.


> 
>> +		break;
>> +
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
>> +
>>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -1564,6 +1723,38 @@ static void v4l_pix_format_touch(struct v4l2_pix_format *p)
>>  	p->xfer_func = 0;
>>  }
>>  
>> +static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +			     struct file *file, void *fh,
>> +			     struct v4l2_format *f)
>> +{
>> +	struct v4l2_ext_pix_format ef;
>> +	int ret;
>> +
>> +	switch (f->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +		ef.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +		ret = ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +		break;
>> +
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +		ef.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +		ret = ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, &ef);
>> +		break;
>> +
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_ext_pix_format_to_format(&ef, f,
>> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +					     true);
>> +}
>> +
>>  static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -1600,17 +1791,27 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>  
>>  	switch (p->type) {
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -		if (unlikely(!ops->vidioc_g_fmt_vid_cap))
>> -			break;
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> -			v4l_pix_format_touch(&p->fmt.pix);
>> -		return ret;
>> +		if (ops->vidioc_g_fmt_vid_cap) {
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +				v4l_pix_format_touch(&p->fmt.pix);
>> +			return ret;
>> +		} else if (ops->vidioc_g_ext_pix_fmt_vid_cap) {
>> +			ret = v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +				v4l_pix_format_touch(&p->fmt.pix);
>> +			return ret;
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -		return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
>> +		if (ops->vidioc_g_fmt_vid_cap_mplane)
>> +			return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
>> +		else if (ops->vidioc_g_ext_pix_fmt_vid_cap)
>> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>  		return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VBI_CAPTURE:
>> @@ -1618,15 +1819,22 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>  	case V4L2_BUF_TYPE_SLICED_VBI_CAPTURE:
>>  		return ops->vidioc_g_fmt_sliced_vbi_cap(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -		if (unlikely(!ops->vidioc_g_fmt_vid_out))
>> -			break;
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		return ret;
>> +		if (ops->vidioc_g_fmt_vid_out) {
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			return ret;
>> +		} else if (ops->vidioc_g_ext_pix_fmt_vid_out) {
>> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -		return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
>> +		if (ops->vidioc_g_fmt_vid_out_mplane)
>> +			return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
>> +		else if (ops->vidioc_g_ext_pix_fmt_vid_out)
>> +			return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>  		return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VBI_OUTPUT:
>> @@ -1645,6 +1853,76 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>  	return -EINVAL;
>>  }
>>  
>> +static int v4l_g_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +			     struct file *file, void *fh, void *arg)
>> +{
>> +	struct v4l2_ext_pix_format *ef = arg;
>> +	struct v4l2_format f = {
>> +		.type = ef->type,
>> +	};
>> +	int ret;
>> +
>> +	ret = check_fmt(file, ef->type);
>> +	if (ret)
>> +		return ret;
>> +
>> +	memset(ef, 0, sizeof(*ef));
>> +	ef->type = f.type;
>> +
>> +	switch (f.type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +		if (ops->vidioc_g_ext_pix_fmt_vid_cap)
>> +			return ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, ef);
>> +		break;
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +		if (ops->vidioc_g_ext_pix_fmt_vid_out)
>> +			return ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, ef);
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	ret = v4l_g_fmt(ops, file, fh, &f);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>> +static int v4l_s_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +			     struct file *file, void *fh,
>> +			     struct v4l2_format *f)
>> +{
>> +	struct v4l2_ext_pix_format ef;
>> +	int ret;
>> +
>> +	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
>> +	if (ret)
>> +		return ret;
>> +
>> +	switch (f->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +		ret = ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +		break;
>> +
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +		ret = ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, &ef);
>> +		break;
>> +
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_ext_pix_format_to_format(&ef, f,
>> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +					     true);
>> +}
>> +
>>  static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -1663,23 +1941,31 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>  
>>  	switch (p->type) {
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -		if (unlikely(!ops->vidioc_s_fmt_vid_cap))
>> +		if (ops->vidioc_s_fmt_vid_cap) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix);
>> +			ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
>> +			ret = v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +		} else {
>>  			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix);
>> -		ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +		}
>> +
>>  		if (vfd->vfl_type == VFL_TYPE_TOUCH)
>>  			v4l_pix_format_touch(&p->fmt.pix);
>>  		return ret;
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -		if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -					  bytesperline);
>> -		return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
>> +		if (ops->vidioc_s_fmt_vid_cap_mplane) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +						  bytesperline);
>> +			return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
>> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
>> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>  		if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
>>  			break;
>> @@ -1696,21 +1982,27 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>  		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>>  		return ops->vidioc_s_fmt_sliced_vbi_cap(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -		if (unlikely(!ops->vidioc_s_fmt_vid_out))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix);
>> -		ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		return ret;
>> +		if (ops->vidioc_s_fmt_vid_out) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix);
>> +			ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			return ret;
>> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
>> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -		if (unlikely(!ops->vidioc_s_fmt_vid_out_mplane))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -					  bytesperline);
>> -		return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
>> +		if (ops->vidioc_s_fmt_vid_out_mplane) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +						  bytesperline);
>> +			return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
>> +		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
>> +			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>  		if (unlikely(!ops->vidioc_s_fmt_vid_out_overlay))
>>  			break;
>> @@ -1750,6 +2042,82 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>  	return -EINVAL;
>>  }
>>  
>> +static int v4l_s_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +			     struct file *file, void *fh, void *arg)
>> +{
>> +	struct video_device *vfd = video_devdata(file);
>> +	struct v4l2_ext_pix_format *ef = arg;
>> +	struct v4l2_format f;
>> +	int ret;
>> +
>> +	ret = check_fmt(file, ef->type);
>> +	if (ret)
>> +		return ret;
>> +
> 
> Should we zero the reserved fields as it's done with the current structs?

Yes, I'll update this.

> 
>> +	switch (ef->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +		if (ops->vidioc_s_ext_pix_fmt_vid_cap)
>> +			return ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, ef);
>> +		break;
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +		if (ops->vidioc_s_ext_pix_fmt_vid_out)
>> +			return ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, ef);
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	ret = v4l2_ext_pix_format_to_format(ef, &f,
>> +					    vfd->device_caps &
>> +					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +					     V4L2_CAP_VIDEO_M2M_MPLANE),
>> +					    false);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = v4l_s_fmt(ops, file, fh, &f);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>> +static int v4l_try_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +			       struct file *file, void *fh,
>> +			       struct v4l2_format *f)
>> +{
>> +	struct v4l2_ext_pix_format ef;
>> +	int ret;
>> +
>> +	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
>> +	if (ret)
>> +		return ret;
>> +
>> +	switch (f->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +		ret = ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +		break;
>> +
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +		ret = ops->vidioc_try_ext_pix_fmt_vid_out(file, fh, &ef);
>> +		break;
>> +
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_ext_pix_format_to_format(&ef, f,
>> +					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +					     true);
>> +}
>> +
>> +
>>  static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -1765,23 +2133,32 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>  
>>  	switch (p->type) {
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -		if (unlikely(!ops->vidioc_try_fmt_vid_cap))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix);
>> -		ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> -			v4l_pix_format_touch(&p->fmt.pix);
>> -		return ret;
>> +		if (ops->vidioc_try_fmt_vid_cap) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix);
>> +			ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +				v4l_pix_format_touch(&p->fmt.pix);
>> +			return ret;
>> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +			ret = v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +			if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +				v4l_pix_format_touch(&p->fmt.pix);
>> +			return ret;
> 
> Should the 3 lines above be outside of the if/else block?

I can move them out, I just need to add an else condition for the break below.
I'll update this in the next version.

> 
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -		if (unlikely(!ops->vidioc_try_fmt_vid_cap_mplane))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -					  bytesperline);
>> -		return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
>> +		if (ops->vidioc_try_fmt_vid_cap_mplane) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +						  bytesperline);
>> +			return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
>> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>  		if (unlikely(!ops->vidioc_try_fmt_vid_overlay))
>>  			break;
>> @@ -1798,21 +2175,27 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>  		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>>  		return ops->vidioc_try_fmt_sliced_vbi_cap(file, fh, arg);
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -		if (unlikely(!ops->vidioc_try_fmt_vid_out))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix);
>> -		ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
>> -		/* just in case the driver zeroed it again */
>> -		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -		return ret;
>> +		if (ops->vidioc_try_fmt_vid_out) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix);
>> +			ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
>> +			/* just in case the driver zeroed it again */
>> +			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +			return ret;
>> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -		if (unlikely(!ops->vidioc_try_fmt_vid_out_mplane))
>> -			break;
>> -		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -					  bytesperline);
>> -		return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
>> +		if (ops->vidioc_try_fmt_vid_out_mplane) {
>> +			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +						  bytesperline);
>> +			return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
>> +		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +			return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +		}
>> +		break;
>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>  		if (unlikely(!ops->vidioc_try_fmt_vid_out_overlay))
>>  			break;
>> @@ -1852,6 +2235,49 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>  	return -EINVAL;
>>  }
>>  
>> +static int v4l_try_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +			       struct file *file, void *fh, void *arg)
>> +{
>> +	struct video_device *vfd = video_devdata(file);
>> +	struct v4l2_ext_pix_format *ef = arg;
>> +	struct v4l2_format f;
>> +	int ret;
>> +
>> +	ret = check_fmt(file, ef->type);
>> +	if (ret)
>> +		return ret;
> 
> Should we zero the reserved fields as it's done with the current structs?

Yes, I'll update this in the next version.

> 
>> +
>> +	switch (ef->type) {
>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +		if (ops->vidioc_try_ext_pix_fmt_vid_cap)
>> +			return ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh,
>> +								   ef);
>> +		break;
>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +		if (ops->vidioc_try_ext_pix_fmt_vid_out)
>> +			return ops->vidioc_try_ext_pix_fmt_vid_out(file, fh,
>> +								   ef);
>> +		break;
>> +	default:
>> +		return -EINVAL;
>> +	}
>> +
>> +	ret = v4l2_ext_pix_format_to_format(ef, &f,
>> +					    vfd->device_caps &
>> +					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +					     V4L2_CAP_VIDEO_M2M_MPLANE),
>> +					    false);
>> +	if (ret)
>> +		return ret;
>> +
>> +	ret = v4l_try_fmt(ops, file, fh, &f);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>>  static int v4l_streamon(const struct v4l2_ioctl_ops *ops,
>>  				struct file *file, void *fh, void *arg)
>>  {
>> @@ -2771,7 +3197,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>  	IOCTL_INFO(VIDIOC_QUERYCAP, v4l_querycap, v4l_print_querycap, 0),
>>  	IOCTL_INFO(VIDIOC_ENUM_FMT, v4l_enum_fmt, v4l_print_fmtdesc, 0),
>>  	IOCTL_INFO(VIDIOC_G_FMT, v4l_g_fmt, v4l_print_format, 0),
>> +	IOCTL_INFO(VIDIOC_G_EXT_PIX_FMT, v4l_g_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>>  	IOCTL_INFO(VIDIOC_S_FMT, v4l_s_fmt, v4l_print_format, INFO_FL_PRIO),
>> +	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>>  	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>>  	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
>>  	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
>> @@ -2818,6 +3246,7 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>  	IOCTL_INFO(VIDIOC_S_JPEGCOMP, v4l_stub_s_jpegcomp, v4l_print_jpegcompression, INFO_FL_PRIO),
>>  	IOCTL_INFO(VIDIOC_QUERYSTD, v4l_querystd, v4l_print_std, 0),
>>  	IOCTL_INFO(VIDIOC_TRY_FMT, v4l_try_fmt, v4l_print_format, 0),
>> +	IOCTL_INFO(VIDIOC_TRY_EXT_PIX_FMT, v4l_try_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>>  	IOCTL_INFO(VIDIOC_ENUMAUDIO, v4l_stub_enumaudio, v4l_print_audio, INFO_FL_CLEAR(v4l2_audio, index)),
>>  	IOCTL_INFO(VIDIOC_ENUMAUDOUT, v4l_stub_enumaudout, v4l_print_audioout, INFO_FL_CLEAR(v4l2_audioout, index)),
>>  	IOCTL_INFO(VIDIOC_G_PRIORITY, v4l_g_priority, v4l_print_u32, 0),
>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
>> index 86878fba332b0..8bbcb74d8ee31 100644
>> --- a/include/media/v4l2-ioctl.h
>> +++ b/include/media/v4l2-ioctl.h
>> @@ -48,11 +48,17 @@ struct v4l2_fh;
>>   * @vidioc_g_fmt_vid_cap: pointer to the function that implements
>>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *	in single plane mode
>> + * @vidioc_g_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *	capture
>>   * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_g_fmt_vid_out: pointer to the function that implements
>>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *	in single plane mode
>> + * @vidioc_g_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *	out
>>   * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>>   * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
>> @@ -82,11 +88,16 @@ struct v4l2_fh;
>>   * @vidioc_s_fmt_vid_cap: pointer to the function that implements
>>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *	in single plane mode
>> + * @vidioc_s_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *	capture
>>   * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_s_fmt_vid_out: pointer to the function that implements
>>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *	in single plane mode
>> + * @vidioc_s_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>>   * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
>> @@ -116,11 +127,16 @@ struct v4l2_fh;
>>   * @vidioc_try_fmt_vid_cap: pointer to the function that implements
>>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *	in single plane mode
>> + * @vidioc_try_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for
>> +	video capture
>>   * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_try_fmt_vid_out: pointer to the function that implements
>>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *	in single plane mode
>> + * @vidioc_try_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
>>   *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   *	output
>> @@ -319,10 +335,14 @@ struct v4l2_ioctl_ops {
>>  	/* VIDIOC_G_FMT handlers */
>>  	int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
>>  				    struct v4l2_format *f);
>> +	int (*vidioc_g_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +					    struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
>>  					struct v4l2_format *f);
>>  	int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
>>  				    struct v4l2_format *f);
>> +	int (*vidioc_g_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +					    struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
>>  					    struct v4l2_format *f);
>>  	int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -349,10 +369,14 @@ struct v4l2_ioctl_ops {
>>  	/* VIDIOC_S_FMT handlers */
>>  	int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
>>  				    struct v4l2_format *f);
>> +	int (*vidioc_s_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +					    struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
>>  					struct v4l2_format *f);
>>  	int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
>>  				    struct v4l2_format *f);
>> +	int (*vidioc_s_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +					    struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
>>  					    struct v4l2_format *f);
>>  	int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -379,10 +403,14 @@ struct v4l2_ioctl_ops {
>>  	/* VIDIOC_TRY_FMT handlers */
>>  	int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
>>  				      struct v4l2_format *f);
>> +	int (*vidioc_try_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +					      struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
>>  					  struct v4l2_format *f);
>>  	int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
>>  				      struct v4l2_format *f);
>> +	int (*vidioc_try_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +					      struct v4l2_ext_pix_format *f);
>>  	int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
>>  					     struct v4l2_format *f);
>>  	int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -724,6 +752,12 @@ long int video_usercopy(struct file *file, unsigned int cmd,
>>  long int video_ioctl2(struct file *file,
>>  		      unsigned int cmd, unsigned long int arg);
>>  
>> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>> +				  struct v4l2_ext_pix_format *e, bool strict);
>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>> +				  struct v4l2_format *f,
>> +				  bool mplane_cap, bool strict);
>> +
>>  /*
>>   * The user space interpretation of the 'v4l2_event' differs
>>   * based on the 'time_t' definition on 32-bit architectures, so
>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>> index c7b70ff53bc1d..7123c6a4d9569 100644
>> --- a/include/uapi/linux/videodev2.h
>> +++ b/include/uapi/linux/videodev2.h
>> @@ -2254,6 +2254,58 @@ struct v4l2_pix_format_mplane {
>>  	__u8				reserved[7];
>>  } __attribute__ ((packed));
>>  
>> +/**
>> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
>> + * @sizeimage:		maximum size in bytes required for data, for which
>> + *			this plane will be used.
>> + *			Should be set to zero for unused planes.
>> + * @bytesperline:	distance in bytes between the leftmost pixels in two
>> + *			adjacent lines
>> + * @reserved:		extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_plane_ext_pix_format {
> 
> nit: Maybe v4l2_ext_plane_pix_format, since the struct describes an
> ext variant of a plane_pix_format?

I'm removing this struct as mentioned below.

> 
>> +	__u32 sizeimage;
>> +	__u32 bytesperline;
>> +	__u32 reserved;
>> +};
> 
> Actually, this seems to be exactly the same as the existing
> v4l2_plane_pix_format, except actually having less reserved space. Could it
> make sense to just reuse the existing struct?

I think so, I'm removing this struct for the next version.

Regards,
Helen

> 
> Best regards,
> Tomasz
>

Tomasz Figa Nov. 19, 2020, 5:45 a.m. UTC | #6

On Sat, Nov 14, 2020 at 11:21:26AM -0300, Helen Koike wrote:
> Hi Tomasz,
> 
> On 10/2/20 4:49 PM, Tomasz Figa wrote:
> > Hi Helen,
> > 
> > On Tue, Aug 04, 2020 at 04:29:33PM -0300, Helen Koike wrote:
[snip]
> >> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
> >> +{
> >> +	const struct v4l2_ext_pix_format *pix = arg;
> >> +	unsigned int i;
> >> +
> >> +	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
> >> +		prt_names(pix->type, v4l2_type_names),
> >> +		pix->width, pix->height,
> >> +		(pix->pixelformat & 0xff),
> >> +		(pix->pixelformat >>  8) & 0xff,
> >> +		(pix->pixelformat >> 16) & 0xff,
> >> +		(pix->pixelformat >> 24) & 0xff,
> >> +		pix->modifier, prt_names(pix->field, v4l2_field_names),
> >> +		pix->colorspace, pix->ycbcr_enc,
> >> +		pix->quantization, pix->xfer_func);
> >> +	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
> > 
> > This is going to print 8 lines every time. Maybe we could skip 0-sized
> > planes or something?
> 
> I'm already checking pix->plane_fmt[i].sizeimage in the loop, it shouldn't
> print 8 lines every time.
> 

Oops, how could I not notice it. Sorry for the noise.

[snip]
> >> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> >> +				  struct v4l2_format *f, bool mplane_cap,
> >> +				  bool strict)
> >> +{
> >> +	const struct v4l2_plane_ext_pix_format *pe;
> >> +	struct v4l2_plane_pix_format *p;
> >> +	unsigned int i;
> >> +
> >> +	memset(f, 0, sizeof(*f));
> >> +
> >> +	/*
> >> +	 * Make sure no modifier is required before doing the
> >> +	 * conversion.
> >> +	 */
> >> +	if (e->modifier && strict &&
> > 
> > Do we need the explicit check for e->modifier != 0 if we have to check for
> > the 2 specific values below anyway?
> 
> We don't, since DRM_FORMAT_MOD_LINEAR is zero.
> 
> But I wanted to make it explicit we don't support modifiers in this conversion.
> But I can remove this check, no problem.
> 

Yes, please. I think the double checking is confusing for the reader.

> > 
> >> +	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
> >> +	    e->modifier != DRM_FORMAT_MOD_INVALID)
> >> +		return -EINVAL;
> >> +
> >> +	if (!e->plane_fmt[0].sizeimage && strict)
> >> +		return -EINVAL;
> > 
> > Why is this incorrect?
> 
> !sizeimage for the first plane means that there are no planes in ef.
> strict is true if the result for the conversion should be returned to userspace
> and it is not some internal handling.
> 
> So if there are no planes, we would return an incomplete v4l2_format struct
> to userspace.
> 
> But this is not very clear, I'll improve this for the next version.
> 

So I can see 2 cases here:

1) Userspace gives ext struct and driver accepts legacy.

In this case, the kernel needs to adjust the structure to be correct.
-EINVAL is only valid if

"The struct v4l2_format type field is invalid or the requested buffer type not supported."

as per the current uAPI documentation.

2) Driver gives ext struct and userspace accepts legacy.

If at this point we get a struct with no planes, that sounds like a
driver bug, so rather than signaling -EINVAL to the userspace, we should
probably WARN()?

Or am I getting something wrong? :)

[snip]
> >> +{
> >> +	const struct v4l2_plane_pix_format *p;
> >> +	struct v4l2_plane_ext_pix_format *pe;
> >> +	unsigned int i;
> >> +
> >> +	memset(e, 0, sizeof(*e));
> >> +
> >> +	switch (f->type) {
> >> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> >> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> >> +		e->width = f->fmt.pix.width;
> >> +		e->height = f->fmt.pix.height;
> >> +		e->pixelformat = f->fmt.pix.pixelformat;
> >> +		e->field = f->fmt.pix.field;
> >> +		e->colorspace = f->fmt.pix.colorspace;
> >> +		if (f->fmt.pix.flags)
> >> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
> >> +				f->fmt.pix.flags);
> > 
> > Would it make sense to print something like video node name and/or function
> > name to explain where this warning comes from?
> 
> I would need to update the function to receive this information, I can try but
> I'm not sure if it is worthy.
> 

I don't have a strong opinion on this, so maybe let's see if others have
any comments.

> > 
> >> +		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
> >> +		e->quantization = f->fmt.pix.quantization;
> > 
> > Missing xfer_func?
> 
> Yes, thanks for catching this.
> 
> > 
> >> +		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
> >> +		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
> > 
> > This doesn't look right. In the ext API we expected the planes to describe
> > color planes, which means that bytesperline needs to be computed for planes
> >> = 1 and sizeimage replaced with per-plane sizes, according to the
> >> pixelformat.
> 
> Ack.
> 
> Just to be clear, even if we are using a planar format that isn't a V4L2_PIX_FMT_*M
> variant, we should describe every plane separatly.
> 
> For instance, if V4L2_PIX_FMT_YVU420 is being used, then f->fmt.pix.bytesperline
> will have data, and we need to calculate bytesperline for all 3 planes, so we'll fill
> out:
> 
> f->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
> f->plane_fmt[1].bytesperline = f->fmt.pix.bytesperline / hdiv;
> f->plane_fmt[2].bytesperline = f->fmt.pix.bytesperline / hdiv;
> 
> I'll update this for the next version.
> 

Yes. This basically gives us a unified representation across all
pixelformats and allows userspace to handle them in a uniform way, as
opposed to current uAPI.

[snip]
> >> +		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> >> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> >> +		else
> >> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> >> +
> >> +		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> >> +			pe = &e->plane_fmt[i];
> >> +			p = &f->fmt.pix_mp.plane_fmt[i];
> >> +			pe->bytesperline = p->bytesperline;
> >> +			pe->sizeimage = p->sizeimage;
> >> +		}
> > 
> > Same here. A blind copy is not enough. For non-M formats, the plane
> > parameters need to be filled according to the pixelformat.
> 
> 
> Right, following the idea above, we need a different handling if we
> aren't using a M-variant of the pixelformat, and we also need to
> convert the pixelformat from the M-variant to non-M-variant.
> 
> I'll also need to save that the original format was a
> M-variant or not, so I can convert it back as expected.

I'm still reading the rest of the series, so it might be answered
already, but did we decide to do anything about the pixelformat codes
themselves? If both M and non-M variants would be allowed with the new
API, then I guess there isn't anything to save, because the original
format would be preserved?

> 
> I'll change this and submit for review.
> 

Cool, thanks.

Best regards,
Tomasz

Helen Mae Koike Fornazier Nov. 19, 2020, 10:08 a.m. UTC | #7

Hi Tomasz,

On 11/19/20 2:45 AM, Tomasz Figa wrote:
> On Sat, Nov 14, 2020 at 11:21:26AM -0300, Helen Koike wrote:
>> Hi Tomasz,
>>
>> On 10/2/20 4:49 PM, Tomasz Figa wrote:
>>> Hi Helen,
>>>
>>> On Tue, Aug 04, 2020 at 04:29:33PM -0300, Helen Koike wrote:
> [snip]
>>>> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
>>>> +{
>>>> +	const struct v4l2_ext_pix_format *pix = arg;
>>>> +	unsigned int i;
>>>> +
>>>> +	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
>>>> +		prt_names(pix->type, v4l2_type_names),
>>>> +		pix->width, pix->height,
>>>> +		(pix->pixelformat & 0xff),
>>>> +		(pix->pixelformat >>  8) & 0xff,
>>>> +		(pix->pixelformat >> 16) & 0xff,
>>>> +		(pix->pixelformat >> 24) & 0xff,
>>>> +		pix->modifier, prt_names(pix->field, v4l2_field_names),
>>>> +		pix->colorspace, pix->ycbcr_enc,
>>>> +		pix->quantization, pix->xfer_func);
>>>> +	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
>>>
>>> This is going to print 8 lines every time. Maybe we could skip 0-sized
>>> planes or something?
>>
>> I'm already checking pix->plane_fmt[i].sizeimage in the loop, it shouldn't
>> print 8 lines every time.
>>
> 
> Oops, how could I not notice it. Sorry for the noise.
> 
> [snip]
>>>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>>>> +				  struct v4l2_format *f, bool mplane_cap,
>>>> +				  bool strict)
>>>> +{
>>>> +	const struct v4l2_plane_ext_pix_format *pe;
>>>> +	struct v4l2_plane_pix_format *p;
>>>> +	unsigned int i;
>>>> +
>>>> +	memset(f, 0, sizeof(*f));
>>>> +
>>>> +	/*
>>>> +	 * Make sure no modifier is required before doing the
>>>> +	 * conversion.
>>>> +	 */
>>>> +	if (e->modifier && strict &&
>>>
>>> Do we need the explicit check for e->modifier != 0 if we have to check for
>>> the 2 specific values below anyway?
>>
>> We don't, since DRM_FORMAT_MOD_LINEAR is zero.
>>
>> But I wanted to make it explicit we don't support modifiers in this conversion.
>> But I can remove this check, no problem.
>>
> 
> Yes, please. I think the double checking is confusing for the reader.

ok.

> 
>>>
>>>> +	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
>>>> +	    e->modifier != DRM_FORMAT_MOD_INVALID)
>>>> +		return -EINVAL;
>>>> +
>>>> +	if (!e->plane_fmt[0].sizeimage && strict)
>>>> +		return -EINVAL;
>>>
>>> Why is this incorrect?
>>
>> !sizeimage for the first plane means that there are no planes in ef.
>> strict is true if the result for the conversion should be returned to userspace
>> and it is not some internal handling.
>>
>> So if there are no planes, we would return an incomplete v4l2_format struct
>> to userspace.
>>
>> But this is not very clear, I'll improve this for the next version.
>>
> 
> So I can see 2 cases here:
> 
> 1) Userspace gives ext struct and driver accepts legacy.
> 
> In this case, the kernel needs to adjust the structure to be correct.
> -EINVAL is only valid if
> 
> "The struct v4l2_format type field is invalid or the requested buffer type not supported."
> 
> as per the current uAPI documentation.
> 
> 2) Driver gives ext struct and userspace accepts legacy.
> 
> If at this point we get a struct with no planes, that sounds like a
> driver bug, so rather than signaling -EINVAL to the userspace, we should
> probably WARN()?
> 
> Or am I getting something wrong? :)

Make sense, I'll restructure this for the next version.

> 
> [snip]
>>>> +{
>>>> +	const struct v4l2_plane_pix_format *p;
>>>> +	struct v4l2_plane_ext_pix_format *pe;
>>>> +	unsigned int i;
>>>> +
>>>> +	memset(e, 0, sizeof(*e));
>>>> +
>>>> +	switch (f->type) {
>>>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>>>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>>>> +		e->width = f->fmt.pix.width;
>>>> +		e->height = f->fmt.pix.height;
>>>> +		e->pixelformat = f->fmt.pix.pixelformat;
>>>> +		e->field = f->fmt.pix.field;
>>>> +		e->colorspace = f->fmt.pix.colorspace;
>>>> +		if (f->fmt.pix.flags)
>>>> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
>>>> +				f->fmt.pix.flags);
>>>
>>> Would it make sense to print something like video node name and/or function
>>> name to explain where this warning comes from?
>>
>> I would need to update the function to receive this information, I can try but
>> I'm not sure if it is worthy.
>>
> 
> I don't have a strong opinion on this, so maybe let's see if others have
> any comments.
> 
>>>
>>>> +		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
>>>> +		e->quantization = f->fmt.pix.quantization;
>>>
>>> Missing xfer_func?
>>
>> Yes, thanks for catching this.
>>
>>>
>>>> +		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
>>>> +		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
>>>
>>> This doesn't look right. In the ext API we expected the planes to describe
>>> color planes, which means that bytesperline needs to be computed for planes
>>>> = 1 and sizeimage replaced with per-plane sizes, according to the
>>>> pixelformat.
>>
>> Ack.
>>
>> Just to be clear, even if we are using a planar format that isn't a V4L2_PIX_FMT_*M
>> variant, we should describe every plane separatly.
>>
>> For instance, if V4L2_PIX_FMT_YVU420 is being used, then f->fmt.pix.bytesperline
>> will have data, and we need to calculate bytesperline for all 3 planes, so we'll fill
>> out:
>>
>> f->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
>> f->plane_fmt[1].bytesperline = f->fmt.pix.bytesperline / hdiv;
>> f->plane_fmt[2].bytesperline = f->fmt.pix.bytesperline / hdiv;
>>
>> I'll update this for the next version.
>>
> 
> Yes. This basically gives us a unified representation across all
> pixelformats and allows userspace to handle them in a uniform way, as
> opposed to current uAPI.

Right, I already updated this in my wip branch for next version.

> 
> [snip]
>>>> +		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
>>>> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>>>> +		else
>>>> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>>>> +
>>>> +		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>>>> +			pe = &e->plane_fmt[i];
>>>> +			p = &f->fmt.pix_mp.plane_fmt[i];
>>>> +			pe->bytesperline = p->bytesperline;
>>>> +			pe->sizeimage = p->sizeimage;
>>>> +		}
>>>
>>> Same here. A blind copy is not enough. For non-M formats, the plane
>>> parameters need to be filled according to the pixelformat.
>>
>>
>> Right, following the idea above, we need a different handling if we
>> aren't using a M-variant of the pixelformat, and we also need to
>> convert the pixelformat from the M-variant to non-M-variant.
>>
>> I'll also need to save that the original format was a
>> M-variant or not, so I can convert it back as expected.
> 
> I'm still reading the rest of the series, so it might be answered
> already, but did we decide to do anything about the pixelformat codes
> themselves? If both M and non-M variants would be allowed with the new
> API, then I guess there isn't anything to save, because the original
> format would be preserved?

I was working with the idea that M-variants wouldn't be allowed.
But then, we have cases where non-M-variant don't exist, such as:

V4L2_PIX_FMT_YVU422M
V4L2_PIX_FMT_YVU444M

(at least, I couldn't find non-M-variant equivalent for those)

But actually, I don't think we formally decided this (and it seems
easier to implement if both are allowed).

Should we allow both variants in the Ext API ?

Thanks
Helen

> 
>>
>> I'll change this and submit for review.
>>
> 
> Cool, thanks.
> 
> Best regards,
> Tomasz
>

Helen Mae Koike Fornazier Nov. 19, 2020, 1:43 p.m. UTC | #8

On 11/19/20 7:08 AM, Helen Koike wrote:
> Hi Tomasz,
> 
> On 11/19/20 2:45 AM, Tomasz Figa wrote:
>> On Sat, Nov 14, 2020 at 11:21:26AM -0300, Helen Koike wrote:
>>> Hi Tomasz,
>>>
>>> On 10/2/20 4:49 PM, Tomasz Figa wrote:
>>>> Hi Helen,
>>>>
>>>> On Tue, Aug 04, 2020 at 04:29:33PM -0300, Helen Koike wrote:
>> [snip]
>>>>> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
>>>>> +{
>>>>> +	const struct v4l2_ext_pix_format *pix = arg;
>>>>> +	unsigned int i;
>>>>> +
>>>>> +	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
>>>>> +		prt_names(pix->type, v4l2_type_names),
>>>>> +		pix->width, pix->height,
>>>>> +		(pix->pixelformat & 0xff),
>>>>> +		(pix->pixelformat >>  8) & 0xff,
>>>>> +		(pix->pixelformat >> 16) & 0xff,
>>>>> +		(pix->pixelformat >> 24) & 0xff,
>>>>> +		pix->modifier, prt_names(pix->field, v4l2_field_names),
>>>>> +		pix->colorspace, pix->ycbcr_enc,
>>>>> +		pix->quantization, pix->xfer_func);
>>>>> +	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
>>>>
>>>> This is going to print 8 lines every time. Maybe we could skip 0-sized
>>>> planes or something?
>>>
>>> I'm already checking pix->plane_fmt[i].sizeimage in the loop, it shouldn't
>>> print 8 lines every time.
>>>
>>
>> Oops, how could I not notice it. Sorry for the noise.
>>
>> [snip]
>>>>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>>>>> +				  struct v4l2_format *f, bool mplane_cap,
>>>>> +				  bool strict)
>>>>> +{
>>>>> +	const struct v4l2_plane_ext_pix_format *pe;
>>>>> +	struct v4l2_plane_pix_format *p;
>>>>> +	unsigned int i;
>>>>> +
>>>>> +	memset(f, 0, sizeof(*f));
>>>>> +
>>>>> +	/*
>>>>> +	 * Make sure no modifier is required before doing the
>>>>> +	 * conversion.
>>>>> +	 */
>>>>> +	if (e->modifier && strict &&
>>>>
>>>> Do we need the explicit check for e->modifier != 0 if we have to check for
>>>> the 2 specific values below anyway?
>>>
>>> We don't, since DRM_FORMAT_MOD_LINEAR is zero.
>>>
>>> But I wanted to make it explicit we don't support modifiers in this conversion.
>>> But I can remove this check, no problem.
>>>
>>
>> Yes, please. I think the double checking is confusing for the reader.
> 
> ok.
> 
>>
>>>>
>>>>> +	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
>>>>> +	    e->modifier != DRM_FORMAT_MOD_INVALID)
>>>>> +		return -EINVAL;
>>>>> +
>>>>> +	if (!e->plane_fmt[0].sizeimage && strict)
>>>>> +		return -EINVAL;
>>>>
>>>> Why is this incorrect?
>>>
>>> !sizeimage for the first plane means that there are no planes in ef.
>>> strict is true if the result for the conversion should be returned to userspace
>>> and it is not some internal handling.
>>>
>>> So if there are no planes, we would return an incomplete v4l2_format struct
>>> to userspace.
>>>
>>> But this is not very clear, I'll improve this for the next version.
>>>
>>
>> So I can see 2 cases here:
>>
>> 1) Userspace gives ext struct and driver accepts legacy.
>>
>> In this case, the kernel needs to adjust the structure to be correct.
>> -EINVAL is only valid if
>>
>> "The struct v4l2_format type field is invalid or the requested buffer type not supported."
>>
>> as per the current uAPI documentation.
>>
>> 2) Driver gives ext struct and userspace accepts legacy.
>>
>> If at this point we get a struct with no planes, that sounds like a
>> driver bug, so rather than signaling -EINVAL to the userspace, we should
>> probably WARN()?
>>
>> Or am I getting something wrong? :)
> 
> Make sense, I'll restructure this for the next version.
> 
>>
>> [snip]
>>>>> +{
>>>>> +	const struct v4l2_plane_pix_format *p;
>>>>> +	struct v4l2_plane_ext_pix_format *pe;
>>>>> +	unsigned int i;
>>>>> +
>>>>> +	memset(e, 0, sizeof(*e));
>>>>> +
>>>>> +	switch (f->type) {
>>>>> +	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>>>>> +	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>>>>> +		e->width = f->fmt.pix.width;
>>>>> +		e->height = f->fmt.pix.height;
>>>>> +		e->pixelformat = f->fmt.pix.pixelformat;
>>>>> +		e->field = f->fmt.pix.field;
>>>>> +		e->colorspace = f->fmt.pix.colorspace;
>>>>> +		if (f->fmt.pix.flags)
>>>>> +			pr_warn("Ignoring pixelformat flags 0x%x\n",
>>>>> +				f->fmt.pix.flags);
>>>>
>>>> Would it make sense to print something like video node name and/or function
>>>> name to explain where this warning comes from?
>>>
>>> I would need to update the function to receive this information, I can try but
>>> I'm not sure if it is worthy.
>>>
>>
>> I don't have a strong opinion on this, so maybe let's see if others have
>> any comments.
>>
>>>>
>>>>> +		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
>>>>> +		e->quantization = f->fmt.pix.quantization;
>>>>
>>>> Missing xfer_func?
>>>
>>> Yes, thanks for catching this.
>>>
>>>>
>>>>> +		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
>>>>> +		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
>>>>
>>>> This doesn't look right. In the ext API we expected the planes to describe
>>>> color planes, which means that bytesperline needs to be computed for planes
>>>>> = 1 and sizeimage replaced with per-plane sizes, according to the
>>>>> pixelformat.
>>>
>>> Ack.
>>>
>>> Just to be clear, even if we are using a planar format that isn't a V4L2_PIX_FMT_*M
>>> variant, we should describe every plane separatly.
>>>
>>> For instance, if V4L2_PIX_FMT_YVU420 is being used, then f->fmt.pix.bytesperline
>>> will have data, and we need to calculate bytesperline for all 3 planes, so we'll fill
>>> out:
>>>
>>> f->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
>>> f->plane_fmt[1].bytesperline = f->fmt.pix.bytesperline / hdiv;
>>> f->plane_fmt[2].bytesperline = f->fmt.pix.bytesperline / hdiv;
>>>
>>> I'll update this for the next version.
>>>
>>
>> Yes. This basically gives us a unified representation across all
>> pixelformats and allows userspace to handle them in a uniform way, as
>> opposed to current uAPI.
> 
> Right, I already updated this in my wip branch for next version.
> 
>>
>> [snip]
>>>>> +		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
>>>>> +			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>>>>> +		else
>>>>> +			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>>>>> +
>>>>> +		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>>>>> +			pe = &e->plane_fmt[i];
>>>>> +			p = &f->fmt.pix_mp.plane_fmt[i];
>>>>> +			pe->bytesperline = p->bytesperline;
>>>>> +			pe->sizeimage = p->sizeimage;
>>>>> +		}
>>>>
>>>> Same here. A blind copy is not enough. For non-M formats, the plane
>>>> parameters need to be filled according to the pixelformat.
>>>
>>>
>>> Right, following the idea above, we need a different handling if we
>>> aren't using a M-variant of the pixelformat, and we also need to
>>> convert the pixelformat from the M-variant to non-M-variant.
>>>
>>> I'll also need to save that the original format was a
>>> M-variant or not, so I can convert it back as expected.
>>
>> I'm still reading the rest of the series, so it might be answered
>> already, but did we decide to do anything about the pixelformat codes
>> themselves? If both M and non-M variants would be allowed with the new
>> API, then I guess there isn't anything to save, because the original
>> format would be preserved?
> 
> I was working with the idea that M-variants wouldn't be allowed.
> But then, we have cases where non-M-variant don't exist, such as:
> 
> V4L2_PIX_FMT_YVU422M
> V4L2_PIX_FMT_YVU444M
> 
> (at least, I couldn't find non-M-variant equivalent for those)
> 
> But actually, I don't think we formally decided this (and it seems
> easier to implement if both are allowed).
> 
> Should we allow both variants in the Ext API ?

I see 3 options:

1) Ext API doesn't accept M-variants and return -EINVAL.

    But this doesn't seem to be the v4l2 way, where we avoid returning
    errors and try to adjust to what we think it is better.

    At the same time, this could allow us, in a very remote hypothetical
    future situation, to remove the M-variants from the kernel when/if
    the old API gets obsolete.

    Future ENUM_EXT_FMT won't enumerate M-variants in this option.

2) Ext API accept M-variants without normalization.

    The driver can receive both variants, and need to handle both as
    equivalents, i.e. if (V4L2_PIX_FMT_YUV420M || V4L2_PIX_FMT_YUV420)

    Both can be returned to userspace.

    Future ENUM_EXT_FMT can enumerate M-variants in this option.

3) Ext API accept M-variants with normalization.

    If userspace uses V4L2_PIX_FMT_YUV420M, the framework converts
    to V4L2_PIX_FMT_YUV420 before passing it to the driver.

    Only V4L2_PIX_FMT_YUV420 is sent back to userspace (even if userspace
    used the M variant, the kernel normalizes it, which is a similar behavior
    when userspace try to use a non-supported resolution and the kernel
    adjusts it).

    (we could also leave this pixelformat normalization to the driver,
    but I don't see a reason to that)

    Future ENUM_EXT_FMT won't enumerate M-variants in this option.



I'm leaning towards option 3, please let me know your thoughts.

Thanks
Helen


> 
> Thanks
> Helen
> 
>>
>>>
>>> I'll change this and submit for review.
>>>
>>
>> Cool, thanks.
>>
>> Best regards,
>> Tomasz
>>

Helen Mae Koike Fornazier Nov. 19, 2020, 4:23 p.m. UTC | #9

Hi Alexandre,

Thanks for your review.

On 8/14/20 4:49 AM, Alexandre Courbot wrote:
> On Wed, Aug 5, 2020 at 4:32 AM Helen Koike <[e-mail]> wrote:
>>
>> This is part of the multiplanar and singleplanar unification process.
>> v4l2_ext_pix_format is supposed to work for both cases.
>>
>> We also add the concept of modifiers already employed in DRM to expose
>> HW-specific formats (like tiled or compressed formats) and allow
>> exchanging this information with the DRM subsystem in a consistent way.
>>
>> Note that only V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] are accepted in
>> v4l2_ext_format, other types will be rejected if you use the
>> {G,S,TRY}_EXT_PIX_FMT ioctls.
>>
>> New hooks have been added to v4l2_ioctl_ops to support those new ioctls
>> in drivers, but, in the meantime, the core takes care of converting
>> {S,G,TRY}_EXT_PIX_FMT requests into {S,G,TRY}_FMT so that old drivers can
>> still work if the userspace app/lib uses the new ioctls.
>> The conversion is also done the other around to allow userspace
>> apps/libs using {S,G,TRY}_FMT to work with drivers implementing the
>> _ext_ hooks.
>>
>> Signed-off-by: Boris Brezillon <[e-mail]>
>> Signed-off-by: Helen Koike <[e-mail]>
>> ---
>> Changes in v5:
>> - change sizes and reorder fields to avoid holes in the struct and make
>>   it the same for 32 and 64 bits
>> - removed __attribute__ ((packed)) from uapi structs
>> - Fix doc warning from make htmldocs
>> - Updated commit message with EXT_PIX prefix for the ioctls.
>>
>> Changes in v4:
>> - Use v4l2_ext_pix_format directly in the ioctl, drop v4l2_ext_format,
>> making V4L2_BUF_TYPE_VIDEO_[OUTPUT,CAPTURE] the only valid types.
>> - Add reserved fields
>> - Removed num_planes from struct v4l2_ext_pix_format
>> - Removed flag field from struct v4l2_ext_pix_format, since the only
>>   defined value is V4L2_PIX_FMT_FLAG_PREMUL_ALPHA only used by vsp1,
>>   where we can use modifiers, or add it back later through the reserved
>>   bits.
>> - In v4l2_ext_format_to_format(), check if modifier is != MOD_LINEAR &&
>>   != MOD_INVALID
>> - Fix type assignment in v4l_g_fmt_ext_pix()
>> - Rebased on top of media/master (post 5.8-rc1)
>>
>> Changes in v3:
>> - Rebased on top of media/master (post 5.4-rc1)
>>
>> Changes in v2:
>> - Move the modifier in v4l2_ext_format (was formerly placed in
>>   v4l2_ext_plane)
>> - Fix a few bugs in the converters and add a strict parameter to
>>   allow conversion of uninitialized/mis-initialized objects
>> ---
>>  drivers/media/v4l2-core/v4l2-dev.c   |  21 +-
>>  drivers/media/v4l2-core/v4l2-ioctl.c | 585 +++++++++++++++++++++++----
>>  include/media/v4l2-ioctl.h           |  34 ++
>>  include/uapi/linux/videodev2.h       |  56 +++
>>  4 files changed, 615 insertions(+), 81 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
>> index a593ea0598b55..e1829906bc086 100644
>> --- a/drivers/media/v4l2-core/v4l2-dev.c
>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
>> @@ -607,25 +607,37 @@ static void determine_valid_ioctls(struct video_device *vdev)
>>                         set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
>>                 if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
>>                                ops->vidioc_g_fmt_vid_cap_mplane ||
>> +                              ops->vidioc_g_ext_pix_fmt_vid_cap ||
>>                                ops->vidioc_g_fmt_vid_overlay)) ||
>>                     (is_tx && (ops->vidioc_g_fmt_vid_out ||
>>                                ops->vidioc_g_fmt_vid_out_mplane ||
>> -                              ops->vidioc_g_fmt_vid_out_overlay)))
>> +                              ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +                              ops->vidioc_g_fmt_vid_out_overlay))) {
>>                          set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
>> +                        set_bit(_IOC_NR(VIDIOC_G_EXT_PIX_FMT), valid_ioctls);
>> +               }
>>                 if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
>>                                ops->vidioc_s_fmt_vid_cap_mplane ||
>> +                              ops->vidioc_s_ext_pix_fmt_vid_cap ||
>>                                ops->vidioc_s_fmt_vid_overlay)) ||
>>                     (is_tx && (ops->vidioc_s_fmt_vid_out ||
>>                                ops->vidioc_s_fmt_vid_out_mplane ||
>> -                              ops->vidioc_s_fmt_vid_out_overlay)))
>> +                              ops->vidioc_s_ext_pix_fmt_vid_out ||
>> +                              ops->vidioc_s_fmt_vid_out_overlay))) {
>>                          set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
>> +                        set_bit(_IOC_NR(VIDIOC_S_EXT_PIX_FMT), valid_ioctls);
>> +               }
>>                 if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
>>                                ops->vidioc_try_fmt_vid_cap_mplane ||
>> +                              ops->vidioc_try_ext_pix_fmt_vid_cap ||
>>                                ops->vidioc_try_fmt_vid_overlay)) ||
>>                     (is_tx && (ops->vidioc_try_fmt_vid_out ||
>>                                ops->vidioc_try_fmt_vid_out_mplane ||
>> -                              ops->vidioc_try_fmt_vid_out_overlay)))
>> +                              ops->vidioc_try_ext_pix_fmt_vid_out ||
>> +                              ops->vidioc_try_fmt_vid_out_overlay))) {
>>                          set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
>> +                        set_bit(_IOC_NR(VIDIOC_TRY_EXT_PIX_FMT), valid_ioctls);
>> +               }
>>                 SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
>>                 SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
>>                 SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
>> @@ -682,8 +694,11 @@ static void determine_valid_ioctls(struct video_device *vdev)
>>                 /* touch specific ioctls */
>>                 SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_vid_cap);
>>                 SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_vid_cap);
>> +               SET_VALID_IOCTL(ops, VIDIOC_G_EXT_PIX_FMT, vidioc_g_fmt_vid_cap);
>>                 SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_vid_cap);
>> +               SET_VALID_IOCTL(ops, VIDIOC_S_EXT_PIX_FMT, vidioc_s_fmt_vid_cap);
>>                 SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_vid_cap);
>> +               SET_VALID_IOCTL(ops, VIDIOC_TRY_EXT_PIX_FMT, vidioc_try_fmt_vid_cap);
>>                 SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
>>                 SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
>>                 SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index a556880f225a5..14a0def50f8ea 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -17,6 +17,8 @@
>>
>>  #include <linux/videodev2.h>
>>
>> +#include <drm/drm_fourcc.h>
>> +
>>  #include <media/v4l2-common.h>
>>  #include <media/v4l2-ioctl.h>
>>  #include <media/v4l2-ctrls.h>
>> @@ -378,6 +380,27 @@ static void v4l_print_format(const void *arg, bool write_only)
>>         }
>>  }
>>
>> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
>> +{
>> +       const struct v4l2_ext_pix_format *pix = arg;
>> +       unsigned int i;
>> +
>> +       pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
>> +               prt_names(pix->type, v4l2_type_names),
>> +               pix->width, pix->height,
>> +               (pix->pixelformat & 0xff),
>> +               (pix->pixelformat >>  8) & 0xff,
>> +               (pix->pixelformat >> 16) & 0xff,
>> +               (pix->pixelformat >> 24) & 0xff,
>> +               pix->modifier, prt_names(pix->field, v4l2_field_names),
>> +               pix->colorspace, pix->ycbcr_enc,
>> +               pix->quantization, pix->xfer_func);
>> +       for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
>> +               pr_debug("plane %u: bytesperline=%u sizeimage=%u\n",
>> +                        i, pix->plane_fmt[i].bytesperline,
>> +                        pix->plane_fmt[i].sizeimage);
>> +}
>> +
>>  static void v4l_print_framebuffer(const void *arg, bool write_only)
>>  {
>>         const struct v4l2_framebuffer *p = arg;
>> @@ -964,11 +987,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>>         switch (type) {
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>>                 if ((is_vid || is_tch) && is_rx &&
>> -                   (ops->vidioc_g_fmt_vid_cap || ops->vidioc_g_fmt_vid_cap_mplane))
>> +                   (ops->vidioc_g_fmt_vid_cap ||
>> +                    ops->vidioc_g_ext_pix_fmt_vid_cap ||
>> +                    ops->vidioc_g_fmt_vid_cap_mplane))
>>                         return 0;
>>                 break;
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -               if ((is_vid || is_tch) && is_rx && ops->vidioc_g_fmt_vid_cap_mplane)
>> +               if ((is_vid || is_tch) && is_rx &&
>> +                   (ops->vidioc_g_fmt_vid_cap_mplane ||
>> +                    ops->vidioc_g_ext_pix_fmt_vid_cap))
>>                         return 0;
>>                 break;
>>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>> @@ -977,11 +1004,15 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>>                 break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>>                 if (is_vid && is_tx &&
>> -                   (ops->vidioc_g_fmt_vid_out || ops->vidioc_g_fmt_vid_out_mplane))
>> +                   (ops->vidioc_g_fmt_vid_out ||
>> +                    ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +                    ops->vidioc_g_fmt_vid_out_mplane))
>>                         return 0;
>>                 break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -               if (is_vid && is_tx && ops->vidioc_g_fmt_vid_out_mplane)
>> +               if (is_vid && is_tx &&
>> +                   (ops->vidioc_g_ext_pix_fmt_vid_out ||
>> +                    ops->vidioc_g_fmt_vid_out_mplane))
>>                         return 0;
>>                 break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>> @@ -1061,6 +1092,134 @@ static void v4l_sanitize_format(struct v4l2_format *fmt)
>>                sizeof(fmt->fmt.pix) - offset);
>>  }
>>
>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>> +                                 struct v4l2_format *f, bool mplane_cap,
>> +                                 bool strict)
>> +{
>> +       const struct v4l2_plane_ext_pix_format *pe;
>> +       struct v4l2_plane_pix_format *p;
>> +       unsigned int i;
>> +
>> +       memset(f, 0, sizeof(*f));
>> +
>> +       /*
>> +        * Make sure no modifier is required before doing the
>> +        * conversion.
>> +        */
>> +       if (e->modifier && strict &&
>> +           e->modifier != DRM_FORMAT_MOD_LINEAR &&
>> +           e->modifier != DRM_FORMAT_MOD_INVALID)
>> +               return -EINVAL;
>> +
>> +       if (!e->plane_fmt[0].sizeimage && strict)
>> +               return -EINVAL;
>> +
>> +       if (e->plane_fmt[1].sizeimage && !mplane_cap && strict)
>> +               return 0;
>> +
>> +       if (!mplane_cap) {
>> +               f->fmt.pix.width = e->width;
>> +               f->fmt.pix.height = e->height;
>> +               f->fmt.pix.pixelformat = e->pixelformat;
>> +               f->fmt.pix.field = e->field;
>> +               f->fmt.pix.colorspace = e->colorspace;
>> +               f->fmt.pix.ycbcr_enc = e->ycbcr_enc;
>> +               f->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +               f->fmt.pix.quantization = e->quantization;
>> +               pe = &e->plane_fmt[0];
>> +               f->fmt.pix.bytesperline = pe->bytesperline;
>> +               f->fmt.pix.sizeimage = pe->sizeimage;
>> +               f->type = e->type;
>> +               return 0;
>> +       }
>> +
>> +       f->fmt.pix_mp.width = e->width;
>> +       f->fmt.pix_mp.height = e->height;
>> +       f->fmt.pix_mp.pixelformat = e->pixelformat;
>> +       f->fmt.pix_mp.field = e->field;
>> +       f->fmt.pix_mp.colorspace = e->colorspace;
>> +       f->fmt.pix_mp.ycbcr_enc = e->ycbcr_enc;
>> +       f->fmt.pix_mp.quantization = e->quantization;
>> +       if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
>> +               f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
>> +       else
>> +               f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
>> +
>> +       for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>> +               pe = &e->plane_fmt[i];
>> +               p = &f->fmt.pix_mp.plane_fmt[i];
>> +               p->bytesperline = pe->bytesperline;
>> +               p->sizeimage = pe->sizeimage;
>> +       }
>> +
>> +       return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_ext_pix_format_to_format);
>> +
>> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>> +                                 struct v4l2_ext_pix_format *e, bool strict)
>> +{
>> +       const struct v4l2_plane_pix_format *p;
>> +       struct v4l2_plane_ext_pix_format *pe;
>> +       unsigned int i;
>> +
>> +       memset(e, 0, sizeof(*e));
>> +
>> +       switch (f->type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +               e->width = f->fmt.pix.width;
>> +               e->height = f->fmt.pix.height;
>> +               e->pixelformat = f->fmt.pix.pixelformat;
>> +               e->field = f->fmt.pix.field;
>> +               e->colorspace = f->fmt.pix.colorspace;
>> +               if (f->fmt.pix.flags)
>> +                       pr_warn("Ignoring pixelformat flags 0x%x\n",
>> +                               f->fmt.pix.flags);
>> +               e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
>> +               e->quantization = f->fmt.pix.quantization;
>> +               e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
>> +               e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
>> +               e->type = f->type;
>> +               break;
>> +
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +               if ((f->fmt.pix_mp.num_planes > VIDEO_MAX_PLANES ||
>> +                    !f->fmt.pix_mp.num_planes) && strict)
>> +                       return -EINVAL;
>> +
>> +               e->width = f->fmt.pix_mp.width;
>> +               e->height = f->fmt.pix_mp.height;
>> +               e->pixelformat = f->fmt.pix_mp.pixelformat;
>> +               e->field = f->fmt.pix_mp.field;
>> +               e->colorspace = f->fmt.pix_mp.colorspace;
>> +               if (f->fmt.pix.flags)
>> +                       pr_warn("Ignoring pixelformat flags 0x%x\n",
>> +                               f->fmt.pix.flags);
>> +               e->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
>> +               e->quantization = f->fmt.pix_mp.quantization;
>> +               if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
>> +                       e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +               else
>> +                       e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +
>> +               for (i = 0; i < VIDEO_MAX_PLANES; i++) {
>> +                       pe = &e->plane_fmt[i];
>> +                       p = &f->fmt.pix_mp.plane_fmt[i];
>> +                       pe->bytesperline = p->bytesperline;
>> +                       pe->sizeimage = p->sizeimage;
>> +               }
>> +               break;
>> +
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
>> +
>>  static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>>                                 struct file *file, void *fh, void *arg)
>>  {
>> @@ -1564,6 +1723,38 @@ static void v4l_pix_format_touch(struct v4l2_pix_format *p)
>>         p->xfer_func = 0;
>>  }
>>
>> +static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +                            struct file *file, void *fh,
>> +                            struct v4l2_format *f)
>> +{
>> +       struct v4l2_ext_pix_format ef;
>> +       int ret;
>> +
>> +       switch (f->type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +               ef.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +               ret = ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +               break;
>> +
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +               ef.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +               ret = ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, &ef);
>> +               break;
>> +
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       if (ret)
>> +               return ret;
>> +
>> +       return v4l2_ext_pix_format_to_format(&ef, f,
>> +                                            V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +                                            true);
>> +}
>> +
>>  static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>                                 struct file *file, void *fh, void *arg)
>>  {
>> @@ -1600,17 +1791,27 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>
>>         switch (p->type) {
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -               if (unlikely(!ops->vidioc_g_fmt_vid_cap))
>> -                       break;
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
>> -               /* just in case the driver zeroed it again */
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> -                       v4l_pix_format_touch(&p->fmt.pix);
>> -               return ret;
>> +               if (ops->vidioc_g_fmt_vid_cap) {
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
>> +                       /* just in case the driver zeroed it again */
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +                               v4l_pix_format_touch(&p->fmt.pix);
>> +                       return ret;
>> +               } else if (ops->vidioc_g_ext_pix_fmt_vid_cap) {
>> +                       ret = v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +                               v4l_pix_format_touch(&p->fmt.pix);
>> +                       return ret;
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -               return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
>> +               if (ops->vidioc_g_fmt_vid_cap_mplane)
>> +                       return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
>> +               else if (ops->vidioc_g_ext_pix_fmt_vid_cap)
>> +                       return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>                 return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
>>         case V4L2_BUF_TYPE_VBI_CAPTURE:
>> @@ -1618,15 +1819,22 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>         case V4L2_BUF_TYPE_SLICED_VBI_CAPTURE:
>>                 return ops->vidioc_g_fmt_sliced_vbi_cap(file, fh, arg);
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -               if (unlikely(!ops->vidioc_g_fmt_vid_out))
>> -                       break;
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
>> -               /* just in case the driver zeroed it again */
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               return ret;
>> +               if (ops->vidioc_g_fmt_vid_out) {
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
>> +                       /* just in case the driver zeroed it again */
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       return ret;
>> +               } else if (ops->vidioc_g_ext_pix_fmt_vid_out) {
>> +                       return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -               return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
>> +               if (ops->vidioc_g_fmt_vid_out_mplane)
>> +                       return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
>> +               else if (ops->vidioc_g_ext_pix_fmt_vid_out)
>> +                       return v4l_g_fmt_ext_pix(ops, file, fh, p);
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>                 return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
>>         case V4L2_BUF_TYPE_VBI_OUTPUT:
>> @@ -1645,6 +1853,76 @@ static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
>>         return -EINVAL;
>>  }
>>
>> +static int v4l_g_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +                            struct file *file, void *fh, void *arg)
>> +{
>> +       struct v4l2_ext_pix_format *ef = arg;
>> +       struct v4l2_format f = {
>> +               .type = ef->type,
>> +       };
>> +       int ret;
>> +
>> +       ret = check_fmt(file, ef->type);
>> +       if (ret)
>> +               return ret;
>> +
>> +       memset(ef, 0, sizeof(*ef));
>> +       ef->type = f.type;
>> +
>> +       switch (f.type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +               if (ops->vidioc_g_ext_pix_fmt_vid_cap)
>> +                       return ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, ef);
>> +               break;
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +               if (ops->vidioc_g_ext_pix_fmt_vid_out)
>> +                       return ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, ef);
>> +               break;
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       ret = v4l_g_fmt(ops, file, fh, &f);
>> +       if (ret)
>> +               return ret;
>> +
>> +       return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>> +static int v4l_s_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +                            struct file *file, void *fh,
>> +                            struct v4l2_format *f)
>> +{
>> +       struct v4l2_ext_pix_format ef;
>> +       int ret;
>> +
>> +       ret = v4l2_format_to_ext_pix_format(f, &ef, false);
>> +       if (ret)
>> +               return ret;
>> +
>> +       switch (f->type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +               ret = ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +               break;
>> +
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +               ret = ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, &ef);
>> +               break;
>> +
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       if (ret)
>> +               return ret;
>> +
>> +       return v4l2_ext_pix_format_to_format(&ef, f,
>> +                                            V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +                                            true);
>> +}
>> +
>>  static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>                                 struct file *file, void *fh, void *arg)
>>  {
>> @@ -1663,23 +1941,31 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>
>>         switch (p->type) {
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -               if (unlikely(!ops->vidioc_s_fmt_vid_cap))
>> +               if (ops->vidioc_s_fmt_vid_cap) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
>> +                       ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
>> +                       /* just in case the driver zeroed it again */
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
>> +                       ret = v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +               } else {
>>                         break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix);
>> -               ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
>> -               /* just in case the driver zeroed it again */
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +               }
>> +
>>                 if (vfd->vfl_type == VFL_TYPE_TOUCH)
>>                         v4l_pix_format_touch(&p->fmt.pix);
>>                 return ret;
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -               if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -                                         bytesperline);
>> -               return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
>> +               if (ops->vidioc_s_fmt_vid_cap_mplane) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +                                                 bytesperline);
>> +                       return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
>> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
>> +                       return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>                 if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
>>                         break;
>> @@ -1696,21 +1982,27 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>                 CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>>                 return ops->vidioc_s_fmt_sliced_vbi_cap(file, fh, arg);
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -               if (unlikely(!ops->vidioc_s_fmt_vid_out))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix);
>> -               ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
>> -               /* just in case the driver zeroed it again */
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               return ret;
>> +               if (ops->vidioc_s_fmt_vid_out) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
>> +                       ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
>> +                       /* just in case the driver zeroed it again */
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       return ret;
>> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
>> +                       return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -               if (unlikely(!ops->vidioc_s_fmt_vid_out_mplane))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -                                         bytesperline);
>> -               return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
>> +               if (ops->vidioc_s_fmt_vid_out_mplane) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +                                                 bytesperline);
>> +                       return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
>> +               } else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
>> +                       return v4l_s_fmt_ext_pix(ops, file, fh, arg);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>                 if (unlikely(!ops->vidioc_s_fmt_vid_out_overlay))
>>                         break;
>> @@ -1750,6 +2042,82 @@ static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
>>         return -EINVAL;
>>  }
>>
>> +static int v4l_s_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +                            struct file *file, void *fh, void *arg)
>> +{
>> +       struct video_device *vfd = video_devdata(file);
>> +       struct v4l2_ext_pix_format *ef = arg;
>> +       struct v4l2_format f;
>> +       int ret;
>> +
>> +       ret = check_fmt(file, ef->type);
>> +       if (ret)
>> +               return ret;
>> +
>> +       switch (ef->type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +               if (ops->vidioc_s_ext_pix_fmt_vid_cap)
>> +                       return ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, ef);
>> +               break;
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +               if (ops->vidioc_s_ext_pix_fmt_vid_out)
>> +                       return ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, ef);
>> +               break;
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       ret = v4l2_ext_pix_format_to_format(ef, &f,
>> +                                           vfd->device_caps &
>> +                                           (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +                                            V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +                                            V4L2_CAP_VIDEO_M2M_MPLANE),
>> +                                           false);
>> +       if (ret)
>> +               return ret;
>> +
>> +       ret = v4l_s_fmt(ops, file, fh, &f);
>> +       if (ret)
>> +               return ret;
>> +
>> +       return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>> +static int v4l_try_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
>> +                              struct file *file, void *fh,
>> +                              struct v4l2_format *f)
>> +{
>> +       struct v4l2_ext_pix_format ef;
>> +       int ret;
>> +
>> +       ret = v4l2_format_to_ext_pix_format(f, &ef, false);
>> +       if (ret)
>> +               return ret;
>> +
>> +       switch (f->type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> +               ret = ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh, &ef);
>> +               break;
>> +
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> +               ret = ops->vidioc_try_ext_pix_fmt_vid_out(file, fh, &ef);
>> +               break;
>> +
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       if (ret)
>> +               return ret;
>> +
>> +       return v4l2_ext_pix_format_to_format(&ef, f,
>> +                                            V4L2_TYPE_IS_MULTIPLANAR(f->type),
>> +                                            true);
>> +}
>> +
>> +
>>  static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>                                 struct file *file, void *fh, void *arg)
>>  {
>> @@ -1765,23 +2133,32 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>
>>         switch (p->type) {
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> -               if (unlikely(!ops->vidioc_try_fmt_vid_cap))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix);
>> -               ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
>> -               /* just in case the driver zeroed it again */
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> -                       v4l_pix_format_touch(&p->fmt.pix);
>> -               return ret;
>> +               if (ops->vidioc_try_fmt_vid_cap) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
>> +                       ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
>> +                       /* just in case the driver zeroed it again */
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +                               v4l_pix_format_touch(&p->fmt.pix);
>> +                       return ret;
>> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +                       ret = v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +                       if (vfd->vfl_type == VFL_TYPE_TOUCH)
>> +                               v4l_pix_format_touch(&p->fmt.pix);
>> +                       return ret;
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>> -               if (unlikely(!ops->vidioc_try_fmt_vid_cap_mplane))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -                                         bytesperline);
>> -               return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
>> +               if (ops->vidioc_try_fmt_vid_cap_mplane) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +                                                 bytesperline);
>> +                       return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
>> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +                       return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>                 if (unlikely(!ops->vidioc_try_fmt_vid_overlay))
>>                         break;
>> @@ -1798,21 +2175,27 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>                 CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
>>                 return ops->vidioc_try_fmt_sliced_vbi_cap(file, fh, arg);
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> -               if (unlikely(!ops->vidioc_try_fmt_vid_out))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix);
>> -               ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
>> -               /* just in case the driver zeroed it again */
>> -               p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> -               return ret;
>> +               if (ops->vidioc_try_fmt_vid_out) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix);
>> +                       ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
>> +                       /* just in case the driver zeroed it again */
>> +                       p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
>> +                       return ret;
>> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +                       return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>> -               if (unlikely(!ops->vidioc_try_fmt_vid_out_mplane))
>> -                       break;
>> -               CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> -               for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> -                       CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> -                                         bytesperline);
>> -               return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
>> +               if (ops->vidioc_try_fmt_vid_out_mplane) {
>> +                       CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
>> +                       for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
>> +                               CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
>> +                                                 bytesperline);
>> +                       return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
>> +               } else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
>> +                       return v4l_try_fmt_ext_pix(ops, file, fh, p);
>> +               }
>> +               break;
>>         case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
>>                 if (unlikely(!ops->vidioc_try_fmt_vid_out_overlay))
>>                         break;
>> @@ -1852,6 +2235,49 @@ static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
>>         return -EINVAL;
>>  }
>>
>> +static int v4l_try_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
>> +                              struct file *file, void *fh, void *arg)
>> +{
>> +       struct video_device *vfd = video_devdata(file);
>> +       struct v4l2_ext_pix_format *ef = arg;
>> +       struct v4l2_format f;
>> +       int ret;
>> +
>> +       ret = check_fmt(file, ef->type);
>> +       if (ret)
>> +               return ret;
>> +
>> +       switch (ef->type) {
>> +       case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>> +               if (ops->vidioc_try_ext_pix_fmt_vid_cap)
>> +                       return ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh,
>> +                                                                  ef);
>> +               break;
>> +       case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>> +               if (ops->vidioc_try_ext_pix_fmt_vid_out)
>> +                       return ops->vidioc_try_ext_pix_fmt_vid_out(file, fh,
>> +                                                                  ef);
>> +               break;
>> +       default:
>> +               return -EINVAL;
>> +       }
>> +
>> +       ret = v4l2_ext_pix_format_to_format(ef, &f,
>> +                                           vfd->device_caps &
>> +                                           (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
>> +                                            V4L2_CAP_VIDEO_OUTPUT_MPLANE |
>> +                                            V4L2_CAP_VIDEO_M2M_MPLANE),
>> +                                           false);
>> +       if (ret)
>> +               return ret;
>> +
>> +       ret = v4l_try_fmt(ops, file, fh, &f);
>> +       if (ret)
>> +               return ret;
>> +
>> +       return v4l2_format_to_ext_pix_format(&f, ef, true);
>> +}
>> +
>>  static int v4l_streamon(const struct v4l2_ioctl_ops *ops,
>>                                 struct file *file, void *fh, void *arg)
>>  {
>> @@ -2771,7 +3197,9 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>         IOCTL_INFO(VIDIOC_QUERYCAP, v4l_querycap, v4l_print_querycap, 0),
>>         IOCTL_INFO(VIDIOC_ENUM_FMT, v4l_enum_fmt, v4l_print_fmtdesc, 0),
>>         IOCTL_INFO(VIDIOC_G_FMT, v4l_g_fmt, v4l_print_format, 0),
>> +       IOCTL_INFO(VIDIOC_G_EXT_PIX_FMT, v4l_g_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>>         IOCTL_INFO(VIDIOC_S_FMT, v4l_s_fmt, v4l_print_format, INFO_FL_PRIO),
>> +       IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
>>         IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
>>         IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
>>         IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
>> @@ -2818,6 +3246,7 @@ static const struct v4l2_ioctl_info v4l2_ioctls[] = {
>>         IOCTL_INFO(VIDIOC_S_JPEGCOMP, v4l_stub_s_jpegcomp, v4l_print_jpegcompression, INFO_FL_PRIO),
>>         IOCTL_INFO(VIDIOC_QUERYSTD, v4l_querystd, v4l_print_std, 0),
>>         IOCTL_INFO(VIDIOC_TRY_FMT, v4l_try_fmt, v4l_print_format, 0),
>> +       IOCTL_INFO(VIDIOC_TRY_EXT_PIX_FMT, v4l_try_ext_pix_fmt, v4l_print_ext_pix_format, 0),
>>         IOCTL_INFO(VIDIOC_ENUMAUDIO, v4l_stub_enumaudio, v4l_print_audio, INFO_FL_CLEAR(v4l2_audio, index)),
>>         IOCTL_INFO(VIDIOC_ENUMAUDOUT, v4l_stub_enumaudout, v4l_print_audioout, INFO_FL_CLEAR(v4l2_audioout, index)),
>>         IOCTL_INFO(VIDIOC_G_PRIORITY, v4l_g_priority, v4l_print_u32, 0),
>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
>> index 86878fba332b0..8bbcb74d8ee31 100644
>> --- a/include/media/v4l2-ioctl.h
>> +++ b/include/media/v4l2-ioctl.h
>> @@ -48,11 +48,17 @@ struct v4l2_fh;
>>   * @vidioc_g_fmt_vid_cap: pointer to the function that implements
>>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *     in single plane mode
>> + * @vidioc_g_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *     :ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *     capture
>>   * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
>>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_g_fmt_vid_out: pointer to the function that implements
>>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *     in single plane mode
>> + * @vidioc_g_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *     :ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *     out
>>   * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
>>   *     :ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>>   * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
>> @@ -82,11 +88,16 @@ struct v4l2_fh;
>>   * @vidioc_s_fmt_vid_cap: pointer to the function that implements
>>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *     in single plane mode
>> + * @vidioc_s_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *     :ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
>> + *     capture
>>   * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
>>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_s_fmt_vid_out: pointer to the function that implements
>>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *     in single plane mode
>> + * @vidioc_s_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *     :ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
>>   *     :ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
>>   * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
>> @@ -116,11 +127,16 @@ struct v4l2_fh;
>>   * @vidioc_try_fmt_vid_cap: pointer to the function that implements
>>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
>>   *     in single plane mode
>> + * @vidioc_try_ext_pix_fmt_vid_cap: pointer to the function that implements
>> + *     :ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for
>> +       video capture
>>   * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
>>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   * @vidioc_try_fmt_vid_out: pointer to the function that implements
>>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   *     in single plane mode
>> + * @vidioc_try_ext_pix_fmt_vid_out: pointer to the function that implements
>> + *     :ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
>>   * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
>>   *     :ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
>>   *     output
>> @@ -319,10 +335,14 @@ struct v4l2_ioctl_ops {
>>         /* VIDIOC_G_FMT handlers */
>>         int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
>>                                     struct v4l2_format *f);
>> +       int (*vidioc_g_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +                                           struct v4l2_ext_pix_format *f);
>>         int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
>>                                         struct v4l2_format *f);
>>         int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
>>                                     struct v4l2_format *f);
>> +       int (*vidioc_g_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +                                           struct v4l2_ext_pix_format *f);
>>         int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
>>                                             struct v4l2_format *f);
>>         int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -349,10 +369,14 @@ struct v4l2_ioctl_ops {
>>         /* VIDIOC_S_FMT handlers */
>>         int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
>>                                     struct v4l2_format *f);
>> +       int (*vidioc_s_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +                                           struct v4l2_ext_pix_format *f);
>>         int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
>>                                         struct v4l2_format *f);
>>         int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
>>                                     struct v4l2_format *f);
>> +       int (*vidioc_s_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +                                           struct v4l2_ext_pix_format *f);
>>         int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
>>                                             struct v4l2_format *f);
>>         int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -379,10 +403,14 @@ struct v4l2_ioctl_ops {
>>         /* VIDIOC_TRY_FMT handlers */
>>         int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
>>                                       struct v4l2_format *f);
>> +       int (*vidioc_try_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
>> +                                             struct v4l2_ext_pix_format *f);
>>         int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
>>                                           struct v4l2_format *f);
>>         int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
>>                                       struct v4l2_format *f);
>> +       int (*vidioc_try_ext_pix_fmt_vid_out)(struct file *file, void *fh,
>> +                                             struct v4l2_ext_pix_format *f);
>>         int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
>>                                              struct v4l2_format *f);
>>         int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
>> @@ -724,6 +752,12 @@ long int video_usercopy(struct file *file, unsigned int cmd,
>>  long int video_ioctl2(struct file *file,
>>                       unsigned int cmd, unsigned long int arg);
>>
>> +int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
>> +                                 struct v4l2_ext_pix_format *e, bool strict);
>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
>> +                                 struct v4l2_format *f,
>> +                                 bool mplane_cap, bool strict);
>> +
>>  /*
>>   * The user space interpretation of the 'v4l2_event' differs
>>   * based on the 'time_t' definition on 32-bit architectures, so
>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>> index c7b70ff53bc1d..7123c6a4d9569 100644
>> --- a/include/uapi/linux/videodev2.h
>> +++ b/include/uapi/linux/videodev2.h
>> @@ -2254,6 +2254,58 @@ struct v4l2_pix_format_mplane {
>>         __u8                            reserved[7];
>>  } __attribute__ ((packed));
>>
>> +/**
>> + * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
>> + * @sizeimage:         maximum size in bytes required for data, for which
>> + *                     this plane will be used.
>> + *                     Should be set to zero for unused planes.
>> + * @bytesperline:      distance in bytes between the leftmost pixels in two
>> + *                     adjacent lines
>> + * @reserved:          extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_plane_ext_pix_format {
>> +       __u32 sizeimage;
>> +       __u32 bytesperline;
>> +       __u32 reserved;
> 
> Do we want to make this field a __u64 so the size of the struct is a
> multiple of 64 bits?

For next version, I'm reusing struct v4l2_plane_pix_format as suggested by Tomasz,
so this adjustment won't be required.

Thanks,
Helen

> 
> 
> 
>> +};
>> +
>> +/**
>> + * struct v4l2_ext_pix_format - extended single/multiplanar format definition
>> + * @type:              type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
>> + *                     V4L2_BUF_TYPE_VIDEO_OUTPUT
>> + * @width:             image width in pixels
>> + * @height:            image height in pixels
>> + * @field:             enum v4l2_field; field order (for interlaced video)
>> + * @pixelformat:       little endian four character code (fourcc)
>> + * @modifier:          modifier applied to the format (used for tiled formats
>> + *                     and other kind of HW-specific formats, like compressed
>> + *                     formats)
>> + * @colorspace:                enum v4l2_colorspace; supplemental to pixelformat
>> + * @plane_fmt:         per-plane information
>> + * @ycbcr_enc:         enum v4l2_ycbcr_encoding, Y'CbCr encoding
>> + * @hsv_enc:           enum v4l2_hsv_encoding, HSV encoding
>> + * @quantization:      enum v4l2_quantization, colorspace quantization
>> + * @xfer_func:         enum v4l2_xfer_func, colorspace transfer function
>> + * @reserved:          extra space reserved for future fields, must be set to 0
>> + */
>> +struct v4l2_ext_pix_format {
>> +       __u32 type;
>> +       __u32 width;
>> +       __u32 height;
>> +       __u32 field;
>> +       __u64 modifier;
>> +       __u32 pixelformat;
>> +       __u32 colorspace;
>> +       struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
>> +       union {
>> +               __u32 ycbcr_enc;
>> +               __u32 hsv_enc;
>> +       };
>> +       __u32 quantization;
>> +       __u32 xfer_func;
>> +       __u32 reserved[9];
>> +};
>> +
>>  /**
>>   * struct v4l2_sdr_format - SDR format definition
>>   * @pixelformat:       little endian four character code (fourcc)
>> @@ -2571,6 +2623,10 @@ struct v4l2_create_buffers {
>>
>>  #define VIDIOC_QUERY_EXT_CTRL  _IOWR('V', 103, struct v4l2_query_ext_ctrl)
>>
>> +#define VIDIOC_G_EXT_PIX_FMT   _IOWR('V', 104, struct v4l2_ext_pix_format)
>> +#define VIDIOC_S_EXT_PIX_FMT   _IOWR('V', 105, struct v4l2_ext_pix_format)
>> +#define VIDIOC_TRY_EXT_PIX_FMT _IOWR('V', 106, struct v4l2_ext_pix_format)
>> +
>>  /* Reminder: when adding new ioctls please add support for them to
>>     drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */
>>
>> --
>> 2.28.0.rc2
>>

Tomasz Figa Dec. 14, 2020, 10:19 a.m. UTC | #10

On Thu, Nov 19, 2020 at 10:43 PM Helen Koike <[e-mail]> wrote:
>
>
>
> On 11/19/20 7:08 AM, Helen Koike wrote:
> > Hi Tomasz,
> >
> > On 11/19/20 2:45 AM, Tomasz Figa wrote:
> >> On Sat, Nov 14, 2020 at 11:21:26AM -0300, Helen Koike wrote:
> >>> Hi Tomasz,
> >>>
> >>> On 10/2/20 4:49 PM, Tomasz Figa wrote:
> >>>> Hi Helen,
> >>>>
> >>>> On Tue, Aug 04, 2020 at 04:29:33PM -0300, Helen Koike wrote:
> >> [snip]
> >>>>> +static void v4l_print_ext_pix_format(const void *arg, bool write_only)
> >>>>> +{
> >>>>> + const struct v4l2_ext_pix_format *pix = arg;
> >>>>> + unsigned int i;
> >>>>> +
> >>>>> + pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
> >>>>> +         prt_names(pix->type, v4l2_type_names),
> >>>>> +         pix->width, pix->height,
> >>>>> +         (pix->pixelformat & 0xff),
> >>>>> +         (pix->pixelformat >>  8) & 0xff,
> >>>>> +         (pix->pixelformat >> 16) & 0xff,
> >>>>> +         (pix->pixelformat >> 24) & 0xff,
> >>>>> +         pix->modifier, prt_names(pix->field, v4l2_field_names),
> >>>>> +         pix->colorspace, pix->ycbcr_enc,
> >>>>> +         pix->quantization, pix->xfer_func);
> >>>>> + for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
> >>>>
> >>>> This is going to print 8 lines every time. Maybe we could skip 0-sized
> >>>> planes or something?
> >>>
> >>> I'm already checking pix->plane_fmt[i].sizeimage in the loop, it shouldn't
> >>> print 8 lines every time.
> >>>
> >>
> >> Oops, how could I not notice it. Sorry for the noise.
> >>
> >> [snip]
> >>>>> +int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
> >>>>> +                           struct v4l2_format *f, bool mplane_cap,
> >>>>> +                           bool strict)
> >>>>> +{
> >>>>> + const struct v4l2_plane_ext_pix_format *pe;
> >>>>> + struct v4l2_plane_pix_format *p;
> >>>>> + unsigned int i;
> >>>>> +
> >>>>> + memset(f, 0, sizeof(*f));
> >>>>> +
> >>>>> + /*
> >>>>> +  * Make sure no modifier is required before doing the
> >>>>> +  * conversion.
> >>>>> +  */
> >>>>> + if (e->modifier && strict &&
> >>>>
> >>>> Do we need the explicit check for e->modifier != 0 if we have to check for
> >>>> the 2 specific values below anyway?
> >>>
> >>> We don't, since DRM_FORMAT_MOD_LINEAR is zero.
> >>>
> >>> But I wanted to make it explicit we don't support modifiers in this conversion.
> >>> But I can remove this check, no problem.
> >>>
> >>
> >> Yes, please. I think the double checking is confusing for the reader.
> >
> > ok.
> >
> >>
> >>>>
> >>>>> +     e->modifier != DRM_FORMAT_MOD_LINEAR &&
> >>>>> +     e->modifier != DRM_FORMAT_MOD_INVALID)
> >>>>> +         return -EINVAL;
> >>>>> +
> >>>>> + if (!e->plane_fmt[0].sizeimage && strict)
> >>>>> +         return -EINVAL;
> >>>>
> >>>> Why is this incorrect?
> >>>
> >>> !sizeimage for the first plane means that there are no planes in ef.
> >>> strict is true if the result for the conversion should be returned to userspace
> >>> and it is not some internal handling.
> >>>
> >>> So if there are no planes, we would return an incomplete v4l2_format struct
> >>> to userspace.
> >>>
> >>> But this is not very clear, I'll improve this for the next version.
> >>>
> >>
> >> So I can see 2 cases here:
> >>
> >> 1) Userspace gives ext struct and driver accepts legacy.
> >>
> >> In this case, the kernel needs to adjust the structure to be correct.
> >> -EINVAL is only valid if
> >>
> >> "The struct v4l2_format type field is invalid or the requested buffer type not supported."
> >>
> >> as per the current uAPI documentation.
> >>
> >> 2) Driver gives ext struct and userspace accepts legacy.
> >>
> >> If at this point we get a struct with no planes, that sounds like a
> >> driver bug, so rather than signaling -EINVAL to the userspace, we should
> >> probably WARN()?
> >>
> >> Or am I getting something wrong? :)
> >
> > Make sense, I'll restructure this for the next version.
> >
> >>
> >> [snip]
> >>>>> +{
> >>>>> + const struct v4l2_plane_pix_format *p;
> >>>>> + struct v4l2_plane_ext_pix_format *pe;
> >>>>> + unsigned int i;
> >>>>> +
> >>>>> + memset(e, 0, sizeof(*e));
> >>>>> +
> >>>>> + switch (f->type) {
> >>>>> + case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> >>>>> + case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> >>>>> +         e->width = f->fmt.pix.width;
> >>>>> +         e->height = f->fmt.pix.height;
> >>>>> +         e->pixelformat = f->fmt.pix.pixelformat;
> >>>>> +         e->field = f->fmt.pix.field;
> >>>>> +         e->colorspace = f->fmt.pix.colorspace;
> >>>>> +         if (f->fmt.pix.flags)
> >>>>> +                 pr_warn("Ignoring pixelformat flags 0x%x\n",
> >>>>> +                         f->fmt.pix.flags);
> >>>>
> >>>> Would it make sense to print something like video node name and/or function
> >>>> name to explain where this warning comes from?
> >>>
> >>> I would need to update the function to receive this information, I can try but
> >>> I'm not sure if it is worthy.
> >>>
> >>
> >> I don't have a strong opinion on this, so maybe let's see if others have
> >> any comments.
> >>
> >>>>
> >>>>> +         e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
> >>>>> +         e->quantization = f->fmt.pix.quantization;
> >>>>
> >>>> Missing xfer_func?
> >>>
> >>> Yes, thanks for catching this.
> >>>
> >>>>
> >>>>> +         e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
> >>>>> +         e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
> >>>>
> >>>> This doesn't look right. In the ext API we expected the planes to describe
> >>>> color planes, which means that bytesperline needs to be computed for planes
> >>>>> = 1 and sizeimage replaced with per-plane sizes, according to the
> >>>>> pixelformat.
> >>>
> >>> Ack.
> >>>
> >>> Just to be clear, even if we are using a planar format that isn't a V4L2_PIX_FMT_*M
> >>> variant, we should describe every plane separatly.
> >>>
> >>> For instance, if V4L2_PIX_FMT_YVU420 is being used, then f->fmt.pix.bytesperline
> >>> will have data, and we need to calculate bytesperline for all 3 planes, so we'll fill
> >>> out:
> >>>
> >>> f->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
> >>> f->plane_fmt[1].bytesperline = f->fmt.pix.bytesperline / hdiv;
> >>> f->plane_fmt[2].bytesperline = f->fmt.pix.bytesperline / hdiv;
> >>>
> >>> I'll update this for the next version.
> >>>
> >>
> >> Yes. This basically gives us a unified representation across all
> >> pixelformats and allows userspace to handle them in a uniform way, as
> >> opposed to current uAPI.
> >
> > Right, I already updated this in my wip branch for next version.
> >
> >>
> >> [snip]
> >>>>> +         if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
> >>>>> +                 e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> >>>>> +         else
> >>>>> +                 e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> >>>>> +
> >>>>> +         for (i = 0; i < VIDEO_MAX_PLANES; i++) {
> >>>>> +                 pe = &e->plane_fmt[i];
> >>>>> +                 p = &f->fmt.pix_mp.plane_fmt[i];
> >>>>> +                 pe->bytesperline = p->bytesperline;
> >>>>> +                 pe->sizeimage = p->sizeimage;
> >>>>> +         }
> >>>>
> >>>> Same here. A blind copy is not enough. For non-M formats, the plane
> >>>> parameters need to be filled according to the pixelformat.
> >>>
> >>>
> >>> Right, following the idea above, we need a different handling if we
> >>> aren't using a M-variant of the pixelformat, and we also need to
> >>> convert the pixelformat from the M-variant to non-M-variant.
> >>>
> >>> I'll also need to save that the original format was a
> >>> M-variant or not, so I can convert it back as expected.
> >>
> >> I'm still reading the rest of the series, so it might be answered
> >> already, but did we decide to do anything about the pixelformat codes
> >> themselves? If both M and non-M variants would be allowed with the new
> >> API, then I guess there isn't anything to save, because the original
> >> format would be preserved?
> >
> > I was working with the idea that M-variants wouldn't be allowed.
> > But then, we have cases where non-M-variant don't exist, such as:
> >
> > V4L2_PIX_FMT_YVU422M
> > V4L2_PIX_FMT_YVU444M
> >
> > (at least, I couldn't find non-M-variant equivalent for those)
> >
> > But actually, I don't think we formally decided this (and it seems
> > easier to implement if both are allowed).
> >
> > Should we allow both variants in the Ext API ?
>
> I see 3 options:
>
> 1) Ext API doesn't accept M-variants and return -EINVAL.
>
>     But this doesn't seem to be the v4l2 way, where we avoid returning
>     errors and try to adjust to what we think it is better.
>
>     At the same time, this could allow us, in a very remote hypothetical
>     future situation, to remove the M-variants from the kernel when/if
>     the old API gets obsolete.
>
>     Future ENUM_EXT_FMT won't enumerate M-variants in this option.
>
> 2) Ext API accept M-variants without normalization.
>
>     The driver can receive both variants, and need to handle both as
>     equivalents, i.e. if (V4L2_PIX_FMT_YUV420M || V4L2_PIX_FMT_YUV420)
>
>     Both can be returned to userspace.
>
>     Future ENUM_EXT_FMT can enumerate M-variants in this option.
>
> 3) Ext API accept M-variants with normalization.
>
>     If userspace uses V4L2_PIX_FMT_YUV420M, the framework converts
>     to V4L2_PIX_FMT_YUV420 before passing it to the driver.
>
>     Only V4L2_PIX_FMT_YUV420 is sent back to userspace (even if userspace
>     used the M variant, the kernel normalizes it, which is a similar behavior
>     when userspace try to use a non-supported resolution and the kernel
>     adjusts it).
>
>     (we could also leave this pixelformat normalization to the driver,
>     but I don't see a reason to that)
>
>     Future ENUM_EXT_FMT won't enumerate M-variants in this option.
>
>
>
> I'm leaning towards option 3, please let me know your thoughts.

Same here. I think it would make it easier to support both APIs at the
same time and also allows having M-only formats as is.

The drawback of option 3 is that it would basically carve the non-M
and M fourcc duality in stone and we couldn't clean them up even if we
eventually deprecate the old API after many years. On the other hand,
in practice we might not be able to deprecate the old API anyway, like
what happened with the current single-planar API, so maybe that's
fine? I'd like others' opinions on this.

Best regards,
Tomasz

Patch

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index a593ea0598b55..e1829906bc086 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -607,25 +607,37 @@  static void determine_valid_ioctls(struct video_device *vdev)
 			set_bit(_IOC_NR(VIDIOC_ENUM_FMT), valid_ioctls);
 		if ((is_rx && (ops->vidioc_g_fmt_vid_cap ||
 			       ops->vidioc_g_fmt_vid_cap_mplane ||
+			       ops->vidioc_g_ext_pix_fmt_vid_cap ||
 			       ops->vidioc_g_fmt_vid_overlay)) ||
 		    (is_tx && (ops->vidioc_g_fmt_vid_out ||
 			       ops->vidioc_g_fmt_vid_out_mplane ||
-			       ops->vidioc_g_fmt_vid_out_overlay)))
+			       ops->vidioc_g_ext_pix_fmt_vid_out ||
+			       ops->vidioc_g_fmt_vid_out_overlay))) {
 			 set_bit(_IOC_NR(VIDIOC_G_FMT), valid_ioctls);
+			 set_bit(_IOC_NR(VIDIOC_G_EXT_PIX_FMT), valid_ioctls);
+		}
 		if ((is_rx && (ops->vidioc_s_fmt_vid_cap ||
 			       ops->vidioc_s_fmt_vid_cap_mplane ||
+			       ops->vidioc_s_ext_pix_fmt_vid_cap ||
 			       ops->vidioc_s_fmt_vid_overlay)) ||
 		    (is_tx && (ops->vidioc_s_fmt_vid_out ||
 			       ops->vidioc_s_fmt_vid_out_mplane ||
-			       ops->vidioc_s_fmt_vid_out_overlay)))
+			       ops->vidioc_s_ext_pix_fmt_vid_out ||
+			       ops->vidioc_s_fmt_vid_out_overlay))) {
 			 set_bit(_IOC_NR(VIDIOC_S_FMT), valid_ioctls);
+			 set_bit(_IOC_NR(VIDIOC_S_EXT_PIX_FMT), valid_ioctls);
+		}
 		if ((is_rx && (ops->vidioc_try_fmt_vid_cap ||
 			       ops->vidioc_try_fmt_vid_cap_mplane ||
+			       ops->vidioc_try_ext_pix_fmt_vid_cap ||
 			       ops->vidioc_try_fmt_vid_overlay)) ||
 		    (is_tx && (ops->vidioc_try_fmt_vid_out ||
 			       ops->vidioc_try_fmt_vid_out_mplane ||
-			       ops->vidioc_try_fmt_vid_out_overlay)))
+			       ops->vidioc_try_ext_pix_fmt_vid_out ||
+			       ops->vidioc_try_fmt_vid_out_overlay))) {
 			 set_bit(_IOC_NR(VIDIOC_TRY_FMT), valid_ioctls);
+			 set_bit(_IOC_NR(VIDIOC_TRY_EXT_PIX_FMT), valid_ioctls);
+		}
 		SET_VALID_IOCTL(ops, VIDIOC_OVERLAY, vidioc_overlay);
 		SET_VALID_IOCTL(ops, VIDIOC_G_FBUF, vidioc_g_fbuf);
 		SET_VALID_IOCTL(ops, VIDIOC_S_FBUF, vidioc_s_fbuf);
@@ -682,8 +694,11 @@  static void determine_valid_ioctls(struct video_device *vdev)
 		/* touch specific ioctls */
 		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FMT, vidioc_enum_fmt_vid_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_G_FMT, vidioc_g_fmt_vid_cap);
+		SET_VALID_IOCTL(ops, VIDIOC_G_EXT_PIX_FMT, vidioc_g_fmt_vid_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_S_FMT, vidioc_s_fmt_vid_cap);
+		SET_VALID_IOCTL(ops, VIDIOC_S_EXT_PIX_FMT, vidioc_s_fmt_vid_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_TRY_FMT, vidioc_try_fmt_vid_cap);
+		SET_VALID_IOCTL(ops, VIDIOC_TRY_EXT_PIX_FMT, vidioc_try_fmt_vid_cap);
 		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMESIZES, vidioc_enum_framesizes);
 		SET_VALID_IOCTL(ops, VIDIOC_ENUM_FRAMEINTERVALS, vidioc_enum_frameintervals);
 		SET_VALID_IOCTL(ops, VIDIOC_ENUMINPUT, vidioc_enum_input);
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index a556880f225a5..14a0def50f8ea 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -17,6 +17,8 @@ 
 
 #include <linux/videodev2.h>
 
+#include <drm/drm_fourcc.h>
+
 #include <media/v4l2-common.h>
 #include <media/v4l2-ioctl.h>
 #include <media/v4l2-ctrls.h>
@@ -378,6 +380,27 @@  static void v4l_print_format(const void *arg, bool write_only)
 	}
 }
 
+static void v4l_print_ext_pix_format(const void *arg, bool write_only)
+{
+	const struct v4l2_ext_pix_format *pix = arg;
+	unsigned int i;
+
+	pr_cont("type=%s, width=%u, height=%u, format=%c%c%c%c, modifier %llx, field=%s, colorspace=%d, ycbcr_enc=%u, quantization=%u, xfer_func=%u\n",
+		prt_names(pix->type, v4l2_type_names),
+		pix->width, pix->height,
+		(pix->pixelformat & 0xff),
+		(pix->pixelformat >>  8) & 0xff,
+		(pix->pixelformat >> 16) & 0xff,
+		(pix->pixelformat >> 24) & 0xff,
+		pix->modifier, prt_names(pix->field, v4l2_field_names),
+		pix->colorspace, pix->ycbcr_enc,
+		pix->quantization, pix->xfer_func);
+	for (i = 0; i < VIDEO_MAX_PLANES && pix->plane_fmt[i].sizeimage; i++)
+		pr_debug("plane %u: bytesperline=%u sizeimage=%u\n",
+			 i, pix->plane_fmt[i].bytesperline,
+			 pix->plane_fmt[i].sizeimage);
+}
+
 static void v4l_print_framebuffer(const void *arg, bool write_only)
 {
 	const struct v4l2_framebuffer *p = arg;
@@ -964,11 +987,15 @@  static int check_fmt(struct file *file, enum v4l2_buf_type type)
 	switch (type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
 		if ((is_vid || is_tch) && is_rx &&
-		    (ops->vidioc_g_fmt_vid_cap || ops->vidioc_g_fmt_vid_cap_mplane))
+		    (ops->vidioc_g_fmt_vid_cap ||
+		     ops->vidioc_g_ext_pix_fmt_vid_cap ||
+		     ops->vidioc_g_fmt_vid_cap_mplane))
 			return 0;
 		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
-		if ((is_vid || is_tch) && is_rx && ops->vidioc_g_fmt_vid_cap_mplane)
+		if ((is_vid || is_tch) && is_rx &&
+		    (ops->vidioc_g_fmt_vid_cap_mplane ||
+		     ops->vidioc_g_ext_pix_fmt_vid_cap))
 			return 0;
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
@@ -977,11 +1004,15 @@  static int check_fmt(struct file *file, enum v4l2_buf_type type)
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
 		if (is_vid && is_tx &&
-		    (ops->vidioc_g_fmt_vid_out || ops->vidioc_g_fmt_vid_out_mplane))
+		    (ops->vidioc_g_fmt_vid_out ||
+		     ops->vidioc_g_ext_pix_fmt_vid_out ||
+		     ops->vidioc_g_fmt_vid_out_mplane))
 			return 0;
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		if (is_vid && is_tx && ops->vidioc_g_fmt_vid_out_mplane)
+		if (is_vid && is_tx &&
+		    (ops->vidioc_g_ext_pix_fmt_vid_out ||
+		     ops->vidioc_g_fmt_vid_out_mplane))
 			return 0;
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
@@ -1061,6 +1092,134 @@  static void v4l_sanitize_format(struct v4l2_format *fmt)
 	       sizeof(fmt->fmt.pix) - offset);
 }
 
+int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
+				  struct v4l2_format *f, bool mplane_cap,
+				  bool strict)
+{
+	const struct v4l2_plane_ext_pix_format *pe;
+	struct v4l2_plane_pix_format *p;
+	unsigned int i;
+
+	memset(f, 0, sizeof(*f));
+
+	/*
+	 * Make sure no modifier is required before doing the
+	 * conversion.
+	 */
+	if (e->modifier && strict &&
+	    e->modifier != DRM_FORMAT_MOD_LINEAR &&
+	    e->modifier != DRM_FORMAT_MOD_INVALID)
+		return -EINVAL;
+
+	if (!e->plane_fmt[0].sizeimage && strict)
+		return -EINVAL;
+
+	if (e->plane_fmt[1].sizeimage && !mplane_cap && strict)
+		return 0;
+
+	if (!mplane_cap) {
+		f->fmt.pix.width = e->width;
+		f->fmt.pix.height = e->height;
+		f->fmt.pix.pixelformat = e->pixelformat;
+		f->fmt.pix.field = e->field;
+		f->fmt.pix.colorspace = e->colorspace;
+		f->fmt.pix.ycbcr_enc = e->ycbcr_enc;
+		f->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+		f->fmt.pix.quantization = e->quantization;
+		pe = &e->plane_fmt[0];
+		f->fmt.pix.bytesperline = pe->bytesperline;
+		f->fmt.pix.sizeimage = pe->sizeimage;
+		f->type = e->type;
+		return 0;
+	}
+
+	f->fmt.pix_mp.width = e->width;
+	f->fmt.pix_mp.height = e->height;
+	f->fmt.pix_mp.pixelformat = e->pixelformat;
+	f->fmt.pix_mp.field = e->field;
+	f->fmt.pix_mp.colorspace = e->colorspace;
+	f->fmt.pix_mp.ycbcr_enc = e->ycbcr_enc;
+	f->fmt.pix_mp.quantization = e->quantization;
+	if (e->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
+		f->type = V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE;
+	else
+		f->type = V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+
+	for (i = 0; i < VIDEO_MAX_PLANES; i++) {
+		pe = &e->plane_fmt[i];
+		p = &f->fmt.pix_mp.plane_fmt[i];
+		p->bytesperline = pe->bytesperline;
+		p->sizeimage = pe->sizeimage;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_ext_pix_format_to_format);
+
+int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
+				  struct v4l2_ext_pix_format *e, bool strict)
+{
+	const struct v4l2_plane_pix_format *p;
+	struct v4l2_plane_ext_pix_format *pe;
+	unsigned int i;
+
+	memset(e, 0, sizeof(*e));
+
+	switch (f->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+		e->width = f->fmt.pix.width;
+		e->height = f->fmt.pix.height;
+		e->pixelformat = f->fmt.pix.pixelformat;
+		e->field = f->fmt.pix.field;
+		e->colorspace = f->fmt.pix.colorspace;
+		if (f->fmt.pix.flags)
+			pr_warn("Ignoring pixelformat flags 0x%x\n",
+				f->fmt.pix.flags);
+		e->ycbcr_enc = f->fmt.pix.ycbcr_enc;
+		e->quantization = f->fmt.pix.quantization;
+		e->plane_fmt[0].bytesperline = f->fmt.pix.bytesperline;
+		e->plane_fmt[0].sizeimage = f->fmt.pix.sizeimage;
+		e->type = f->type;
+		break;
+
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		if ((f->fmt.pix_mp.num_planes > VIDEO_MAX_PLANES ||
+		     !f->fmt.pix_mp.num_planes) && strict)
+			return -EINVAL;
+
+		e->width = f->fmt.pix_mp.width;
+		e->height = f->fmt.pix_mp.height;
+		e->pixelformat = f->fmt.pix_mp.pixelformat;
+		e->field = f->fmt.pix_mp.field;
+		e->colorspace = f->fmt.pix_mp.colorspace;
+		if (f->fmt.pix.flags)
+			pr_warn("Ignoring pixelformat flags 0x%x\n",
+				f->fmt.pix.flags);
+		e->ycbcr_enc = f->fmt.pix_mp.ycbcr_enc;
+		e->quantization = f->fmt.pix_mp.quantization;
+		if (f->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
+			e->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+		else
+			e->type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
+
+		for (i = 0; i < VIDEO_MAX_PLANES; i++) {
+			pe = &e->plane_fmt[i];
+			p = &f->fmt.pix_mp.plane_fmt[i];
+			pe->bytesperline = p->bytesperline;
+			pe->sizeimage = p->sizeimage;
+		}
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_format_to_ext_pix_format);
+
 static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -1564,6 +1723,38 @@  static void v4l_pix_format_touch(struct v4l2_pix_format *p)
 	p->xfer_func = 0;
 }
 
+static int v4l_g_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
+			     struct file *file, void *fh,
+			     struct v4l2_format *f)
+{
+	struct v4l2_ext_pix_format ef;
+	int ret;
+
+	switch (f->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+		ef.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+		ret = ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, &ef);
+		break;
+
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		ef.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
+		ret = ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, &ef);
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	if (ret)
+		return ret;
+
+	return v4l2_ext_pix_format_to_format(&ef, f,
+					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
+					     true);
+}
+
 static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -1600,17 +1791,27 @@  static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
 
 	switch (p->type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
-		if (unlikely(!ops->vidioc_g_fmt_vid_cap))
-			break;
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		if (vfd->vfl_type == VFL_TYPE_TOUCH)
-			v4l_pix_format_touch(&p->fmt.pix);
-		return ret;
+		if (ops->vidioc_g_fmt_vid_cap) {
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			ret = ops->vidioc_g_fmt_vid_cap(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			if (vfd->vfl_type == VFL_TYPE_TOUCH)
+				v4l_pix_format_touch(&p->fmt.pix);
+			return ret;
+		} else if (ops->vidioc_g_ext_pix_fmt_vid_cap) {
+			ret = v4l_g_fmt_ext_pix(ops, file, fh, p);
+			if (vfd->vfl_type == VFL_TYPE_TOUCH)
+				v4l_pix_format_touch(&p->fmt.pix);
+			return ret;
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
-		return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
+		if (ops->vidioc_g_fmt_vid_cap_mplane)
+			return ops->vidioc_g_fmt_vid_cap_mplane(file, fh, arg);
+		else if (ops->vidioc_g_ext_pix_fmt_vid_cap)
+			return v4l_g_fmt_ext_pix(ops, file, fh, p);
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
 		return ops->vidioc_g_fmt_vid_overlay(file, fh, arg);
 	case V4L2_BUF_TYPE_VBI_CAPTURE:
@@ -1618,15 +1819,22 @@  static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
 	case V4L2_BUF_TYPE_SLICED_VBI_CAPTURE:
 		return ops->vidioc_g_fmt_sliced_vbi_cap(file, fh, arg);
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
-		if (unlikely(!ops->vidioc_g_fmt_vid_out))
-			break;
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		return ret;
+		if (ops->vidioc_g_fmt_vid_out) {
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			ret = ops->vidioc_g_fmt_vid_out(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			return ret;
+		} else if (ops->vidioc_g_ext_pix_fmt_vid_out) {
+			return v4l_g_fmt_ext_pix(ops, file, fh, p);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
+		if (ops->vidioc_g_fmt_vid_out_mplane)
+			return ops->vidioc_g_fmt_vid_out_mplane(file, fh, arg);
+		else if (ops->vidioc_g_ext_pix_fmt_vid_out)
+			return v4l_g_fmt_ext_pix(ops, file, fh, p);
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
 		return ops->vidioc_g_fmt_vid_out_overlay(file, fh, arg);
 	case V4L2_BUF_TYPE_VBI_OUTPUT:
@@ -1645,6 +1853,76 @@  static int v4l_g_fmt(const struct v4l2_ioctl_ops *ops,
 	return -EINVAL;
 }
 
+static int v4l_g_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
+			     struct file *file, void *fh, void *arg)
+{
+	struct v4l2_ext_pix_format *ef = arg;
+	struct v4l2_format f = {
+		.type = ef->type,
+	};
+	int ret;
+
+	ret = check_fmt(file, ef->type);
+	if (ret)
+		return ret;
+
+	memset(ef, 0, sizeof(*ef));
+	ef->type = f.type;
+
+	switch (f.type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+		if (ops->vidioc_g_ext_pix_fmt_vid_cap)
+			return ops->vidioc_g_ext_pix_fmt_vid_cap(file, fh, ef);
+		break;
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+		if (ops->vidioc_g_ext_pix_fmt_vid_out)
+			return ops->vidioc_g_ext_pix_fmt_vid_out(file, fh, ef);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	ret = v4l_g_fmt(ops, file, fh, &f);
+	if (ret)
+		return ret;
+
+	return v4l2_format_to_ext_pix_format(&f, ef, true);
+}
+
+static int v4l_s_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
+			     struct file *file, void *fh,
+			     struct v4l2_format *f)
+{
+	struct v4l2_ext_pix_format ef;
+	int ret;
+
+	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
+	if (ret)
+		return ret;
+
+	switch (f->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+		ret = ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, &ef);
+		break;
+
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		ret = ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, &ef);
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	if (ret)
+		return ret;
+
+	return v4l2_ext_pix_format_to_format(&ef, f,
+					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
+					     true);
+}
+
 static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -1663,23 +1941,31 @@  static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
 
 	switch (p->type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
-		if (unlikely(!ops->vidioc_s_fmt_vid_cap))
+		if (ops->vidioc_s_fmt_vid_cap) {
+			CLEAR_AFTER_FIELD(p, fmt.pix);
+			ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
+			ret = v4l_s_fmt_ext_pix(ops, file, fh, arg);
+		} else {
 			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix);
-		ret = ops->vidioc_s_fmt_vid_cap(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+		}
+
 		if (vfd->vfl_type == VFL_TYPE_TOUCH)
 			v4l_pix_format_touch(&p->fmt.pix);
 		return ret;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
-		if (unlikely(!ops->vidioc_s_fmt_vid_cap_mplane))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
-		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
-			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
-					  bytesperline);
-		return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
+		if (ops->vidioc_s_fmt_vid_cap_mplane) {
+			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
+			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
+				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
+						  bytesperline);
+			return ops->vidioc_s_fmt_vid_cap_mplane(file, fh, arg);
+		} else if (ops->vidioc_s_ext_pix_fmt_vid_cap) {
+			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
 		if (unlikely(!ops->vidioc_s_fmt_vid_overlay))
 			break;
@@ -1696,21 +1982,27 @@  static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
 		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
 		return ops->vidioc_s_fmt_sliced_vbi_cap(file, fh, arg);
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
-		if (unlikely(!ops->vidioc_s_fmt_vid_out))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix);
-		ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		return ret;
+		if (ops->vidioc_s_fmt_vid_out) {
+			CLEAR_AFTER_FIELD(p, fmt.pix);
+			ret = ops->vidioc_s_fmt_vid_out(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			return ret;
+		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
+			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		if (unlikely(!ops->vidioc_s_fmt_vid_out_mplane))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
-		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
-			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
-					  bytesperline);
-		return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
+		if (ops->vidioc_s_fmt_vid_out_mplane) {
+			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
+			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
+				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
+						  bytesperline);
+			return ops->vidioc_s_fmt_vid_out_mplane(file, fh, arg);
+		} else if (ops->vidioc_s_ext_pix_fmt_vid_out) {
+			return v4l_s_fmt_ext_pix(ops, file, fh, arg);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
 		if (unlikely(!ops->vidioc_s_fmt_vid_out_overlay))
 			break;
@@ -1750,6 +2042,82 @@  static int v4l_s_fmt(const struct v4l2_ioctl_ops *ops,
 	return -EINVAL;
 }
 
+static int v4l_s_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
+			     struct file *file, void *fh, void *arg)
+{
+	struct video_device *vfd = video_devdata(file);
+	struct v4l2_ext_pix_format *ef = arg;
+	struct v4l2_format f;
+	int ret;
+
+	ret = check_fmt(file, ef->type);
+	if (ret)
+		return ret;
+
+	switch (ef->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+		if (ops->vidioc_s_ext_pix_fmt_vid_cap)
+			return ops->vidioc_s_ext_pix_fmt_vid_cap(file, fh, ef);
+		break;
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+		if (ops->vidioc_s_ext_pix_fmt_vid_out)
+			return ops->vidioc_s_ext_pix_fmt_vid_out(file, fh, ef);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	ret = v4l2_ext_pix_format_to_format(ef, &f,
+					    vfd->device_caps &
+					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+					     V4L2_CAP_VIDEO_M2M_MPLANE),
+					    false);
+	if (ret)
+		return ret;
+
+	ret = v4l_s_fmt(ops, file, fh, &f);
+	if (ret)
+		return ret;
+
+	return v4l2_format_to_ext_pix_format(&f, ef, true);
+}
+
+static int v4l_try_fmt_ext_pix(const struct v4l2_ioctl_ops *ops,
+			       struct file *file, void *fh,
+			       struct v4l2_format *f)
+{
+	struct v4l2_ext_pix_format ef;
+	int ret;
+
+	ret = v4l2_format_to_ext_pix_format(f, &ef, false);
+	if (ret)
+		return ret;
+
+	switch (f->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+		ret = ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh, &ef);
+		break;
+
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+		ret = ops->vidioc_try_ext_pix_fmt_vid_out(file, fh, &ef);
+		break;
+
+	default:
+		return -EINVAL;
+	}
+
+	if (ret)
+		return ret;
+
+	return v4l2_ext_pix_format_to_format(&ef, f,
+					     V4L2_TYPE_IS_MULTIPLANAR(f->type),
+					     true);
+}
+
+
 static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -1765,23 +2133,32 @@  static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
 
 	switch (p->type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
-		if (unlikely(!ops->vidioc_try_fmt_vid_cap))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix);
-		ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		if (vfd->vfl_type == VFL_TYPE_TOUCH)
-			v4l_pix_format_touch(&p->fmt.pix);
-		return ret;
+		if (ops->vidioc_try_fmt_vid_cap) {
+			CLEAR_AFTER_FIELD(p, fmt.pix);
+			ret = ops->vidioc_try_fmt_vid_cap(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			if (vfd->vfl_type == VFL_TYPE_TOUCH)
+				v4l_pix_format_touch(&p->fmt.pix);
+			return ret;
+		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
+			ret = v4l_try_fmt_ext_pix(ops, file, fh, p);
+			if (vfd->vfl_type == VFL_TYPE_TOUCH)
+				v4l_pix_format_touch(&p->fmt.pix);
+			return ret;
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
-		if (unlikely(!ops->vidioc_try_fmt_vid_cap_mplane))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
-		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
-			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
-					  bytesperline);
-		return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
+		if (ops->vidioc_try_fmt_vid_cap_mplane) {
+			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
+			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
+				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
+						  bytesperline);
+			return ops->vidioc_try_fmt_vid_cap_mplane(file, fh, arg);
+		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
+			return v4l_try_fmt_ext_pix(ops, file, fh, p);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
 		if (unlikely(!ops->vidioc_try_fmt_vid_overlay))
 			break;
@@ -1798,21 +2175,27 @@  static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
 		CLEAR_AFTER_FIELD(p, fmt.sliced.io_size);
 		return ops->vidioc_try_fmt_sliced_vbi_cap(file, fh, arg);
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
-		if (unlikely(!ops->vidioc_try_fmt_vid_out))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix);
-		ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
-		/* just in case the driver zeroed it again */
-		p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
-		return ret;
+		if (ops->vidioc_try_fmt_vid_out) {
+			CLEAR_AFTER_FIELD(p, fmt.pix);
+			ret = ops->vidioc_try_fmt_vid_out(file, fh, arg);
+			/* just in case the driver zeroed it again */
+			p->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+			return ret;
+		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
+			return v4l_try_fmt_ext_pix(ops, file, fh, p);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
-		if (unlikely(!ops->vidioc_try_fmt_vid_out_mplane))
-			break;
-		CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
-		for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
-			CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
-					  bytesperline);
-		return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
+		if (ops->vidioc_try_fmt_vid_out_mplane) {
+			CLEAR_AFTER_FIELD(p, fmt.pix_mp.xfer_func);
+			for (i = 0; i < p->fmt.pix_mp.num_planes; i++)
+				CLEAR_AFTER_FIELD(&p->fmt.pix_mp.plane_fmt[i],
+						  bytesperline);
+			return ops->vidioc_try_fmt_vid_out_mplane(file, fh, arg);
+		} else if (ops->vidioc_try_ext_pix_fmt_vid_cap) {
+			return v4l_try_fmt_ext_pix(ops, file, fh, p);
+		}
+		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY:
 		if (unlikely(!ops->vidioc_try_fmt_vid_out_overlay))
 			break;
@@ -1852,6 +2235,49 @@  static int v4l_try_fmt(const struct v4l2_ioctl_ops *ops,
 	return -EINVAL;
 }
 
+static int v4l_try_ext_pix_fmt(const struct v4l2_ioctl_ops *ops,
+			       struct file *file, void *fh, void *arg)
+{
+	struct video_device *vfd = video_devdata(file);
+	struct v4l2_ext_pix_format *ef = arg;
+	struct v4l2_format f;
+	int ret;
+
+	ret = check_fmt(file, ef->type);
+	if (ret)
+		return ret;
+
+	switch (ef->type) {
+	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+		if (ops->vidioc_try_ext_pix_fmt_vid_cap)
+			return ops->vidioc_try_ext_pix_fmt_vid_cap(file, fh,
+								   ef);
+		break;
+	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+		if (ops->vidioc_try_ext_pix_fmt_vid_out)
+			return ops->vidioc_try_ext_pix_fmt_vid_out(file, fh,
+								   ef);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	ret = v4l2_ext_pix_format_to_format(ef, &f,
+					    vfd->device_caps &
+					    (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+					     V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+					     V4L2_CAP_VIDEO_M2M_MPLANE),
+					    false);
+	if (ret)
+		return ret;
+
+	ret = v4l_try_fmt(ops, file, fh, &f);
+	if (ret)
+		return ret;
+
+	return v4l2_format_to_ext_pix_format(&f, ef, true);
+}
+
 static int v4l_streamon(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
@@ -2771,7 +3197,9 @@  static const struct v4l2_ioctl_info v4l2_ioctls[] = {
 	IOCTL_INFO(VIDIOC_QUERYCAP, v4l_querycap, v4l_print_querycap, 0),
 	IOCTL_INFO(VIDIOC_ENUM_FMT, v4l_enum_fmt, v4l_print_fmtdesc, 0),
 	IOCTL_INFO(VIDIOC_G_FMT, v4l_g_fmt, v4l_print_format, 0),
+	IOCTL_INFO(VIDIOC_G_EXT_PIX_FMT, v4l_g_ext_pix_fmt, v4l_print_ext_pix_format, 0),
 	IOCTL_INFO(VIDIOC_S_FMT, v4l_s_fmt, v4l_print_format, INFO_FL_PRIO),
+	IOCTL_INFO(VIDIOC_S_EXT_PIX_FMT, v4l_s_ext_pix_fmt, v4l_print_ext_pix_format, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_REQBUFS, v4l_reqbufs, v4l_print_requestbuffers, INFO_FL_PRIO | INFO_FL_QUEUE),
 	IOCTL_INFO(VIDIOC_QUERYBUF, v4l_querybuf, v4l_print_buffer, INFO_FL_QUEUE | INFO_FL_CLEAR(v4l2_buffer, length)),
 	IOCTL_INFO(VIDIOC_G_FBUF, v4l_stub_g_fbuf, v4l_print_framebuffer, 0),
@@ -2818,6 +3246,7 @@  static const struct v4l2_ioctl_info v4l2_ioctls[] = {
 	IOCTL_INFO(VIDIOC_S_JPEGCOMP, v4l_stub_s_jpegcomp, v4l_print_jpegcompression, INFO_FL_PRIO),
 	IOCTL_INFO(VIDIOC_QUERYSTD, v4l_querystd, v4l_print_std, 0),
 	IOCTL_INFO(VIDIOC_TRY_FMT, v4l_try_fmt, v4l_print_format, 0),
+	IOCTL_INFO(VIDIOC_TRY_EXT_PIX_FMT, v4l_try_ext_pix_fmt, v4l_print_ext_pix_format, 0),
 	IOCTL_INFO(VIDIOC_ENUMAUDIO, v4l_stub_enumaudio, v4l_print_audio, INFO_FL_CLEAR(v4l2_audio, index)),
 	IOCTL_INFO(VIDIOC_ENUMAUDOUT, v4l_stub_enumaudout, v4l_print_audioout, INFO_FL_CLEAR(v4l2_audioout, index)),
 	IOCTL_INFO(VIDIOC_G_PRIORITY, v4l_g_priority, v4l_print_u32, 0),
diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h
index 86878fba332b0..8bbcb74d8ee31 100644
--- a/include/media/v4l2-ioctl.h
+++ b/include/media/v4l2-ioctl.h
@@ -48,11 +48,17 @@  struct v4l2_fh;
  * @vidioc_g_fmt_vid_cap: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video capture
  *	in single plane mode
+ * @vidioc_g_ext_pix_fmt_vid_cap: pointer to the function that implements
+ *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
+ *	capture
  * @vidioc_g_fmt_vid_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  * @vidioc_g_fmt_vid_out: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video out
  *	in single plane mode
+ * @vidioc_g_ext_pix_fmt_vid_out: pointer to the function that implements
+ *	:ref:`VIDIOC_G_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
+ *	out
  * @vidioc_g_fmt_vid_out_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_G_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
  * @vidioc_g_fmt_vbi_cap: pointer to the function that implements
@@ -82,11 +88,16 @@  struct v4l2_fh;
  * @vidioc_s_fmt_vid_cap: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video capture
  *	in single plane mode
+ * @vidioc_s_ext_pix_fmt_vid_cap: pointer to the function that implements
+ *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for video
+ *	capture
  * @vidioc_s_fmt_vid_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  * @vidioc_s_fmt_vid_out: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video out
  *	in single plane mode
+ * @vidioc_s_ext_pix_fmt_vid_out: pointer to the function that implements
+ *	:ref:`VIDIOC_S_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
  * @vidioc_s_fmt_vid_out_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_S_FMT <vidioc_g_fmt>` ioctl logic for video overlay output
  * @vidioc_s_fmt_vbi_cap: pointer to the function that implements
@@ -116,11 +127,16 @@  struct v4l2_fh;
  * @vidioc_try_fmt_vid_cap: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video capture
  *	in single plane mode
+ * @vidioc_try_ext_pix_fmt_vid_cap: pointer to the function that implements
+ *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_ext_pix_fmt>` ioctl logic for
+	video capture
  * @vidioc_try_fmt_vid_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  * @vidioc_try_fmt_vid_out: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video out
  *	in single plane mode
+ * @vidioc_try_ext_pix_fmt_vid_out: pointer to the function that implements
+ *	:ref:`VIDIOC_TRY_EXT_PIX_FMT <vidioc_g_fmt>` ioctl logic for video out
  * @vidioc_try_fmt_vid_out_overlay: pointer to the function that implements
  *	:ref:`VIDIOC_TRY_FMT <vidioc_g_fmt>` ioctl logic for video overlay
  *	output
@@ -319,10 +335,14 @@  struct v4l2_ioctl_ops {
 	/* VIDIOC_G_FMT handlers */
 	int (*vidioc_g_fmt_vid_cap)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_g_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
+					    struct v4l2_ext_pix_format *f);
 	int (*vidioc_g_fmt_vid_overlay)(struct file *file, void *fh,
 					struct v4l2_format *f);
 	int (*vidioc_g_fmt_vid_out)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_g_ext_pix_fmt_vid_out)(struct file *file, void *fh,
+					    struct v4l2_ext_pix_format *f);
 	int (*vidioc_g_fmt_vid_out_overlay)(struct file *file, void *fh,
 					    struct v4l2_format *f);
 	int (*vidioc_g_fmt_vbi_cap)(struct file *file, void *fh,
@@ -349,10 +369,14 @@  struct v4l2_ioctl_ops {
 	/* VIDIOC_S_FMT handlers */
 	int (*vidioc_s_fmt_vid_cap)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_s_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
+					    struct v4l2_ext_pix_format *f);
 	int (*vidioc_s_fmt_vid_overlay)(struct file *file, void *fh,
 					struct v4l2_format *f);
 	int (*vidioc_s_fmt_vid_out)(struct file *file, void *fh,
 				    struct v4l2_format *f);
+	int (*vidioc_s_ext_pix_fmt_vid_out)(struct file *file, void *fh,
+					    struct v4l2_ext_pix_format *f);
 	int (*vidioc_s_fmt_vid_out_overlay)(struct file *file, void *fh,
 					    struct v4l2_format *f);
 	int (*vidioc_s_fmt_vbi_cap)(struct file *file, void *fh,
@@ -379,10 +403,14 @@  struct v4l2_ioctl_ops {
 	/* VIDIOC_TRY_FMT handlers */
 	int (*vidioc_try_fmt_vid_cap)(struct file *file, void *fh,
 				      struct v4l2_format *f);
+	int (*vidioc_try_ext_pix_fmt_vid_cap)(struct file *file, void *fh,
+					      struct v4l2_ext_pix_format *f);
 	int (*vidioc_try_fmt_vid_overlay)(struct file *file, void *fh,
 					  struct v4l2_format *f);
 	int (*vidioc_try_fmt_vid_out)(struct file *file, void *fh,
 				      struct v4l2_format *f);
+	int (*vidioc_try_ext_pix_fmt_vid_out)(struct file *file, void *fh,
+					      struct v4l2_ext_pix_format *f);
 	int (*vidioc_try_fmt_vid_out_overlay)(struct file *file, void *fh,
 					     struct v4l2_format *f);
 	int (*vidioc_try_fmt_vbi_cap)(struct file *file, void *fh,
@@ -724,6 +752,12 @@  long int video_usercopy(struct file *file, unsigned int cmd,
 long int video_ioctl2(struct file *file,
 		      unsigned int cmd, unsigned long int arg);
 
+int v4l2_format_to_ext_pix_format(const struct v4l2_format *f,
+				  struct v4l2_ext_pix_format *e, bool strict);
+int v4l2_ext_pix_format_to_format(const struct v4l2_ext_pix_format *e,
+				  struct v4l2_format *f,
+				  bool mplane_cap, bool strict);
+
 /*
  * The user space interpretation of the 'v4l2_event' differs
  * based on the 'time_t' definition on 32-bit architectures, so
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index c7b70ff53bc1d..7123c6a4d9569 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -2254,6 +2254,58 @@  struct v4l2_pix_format_mplane {
 	__u8				reserved[7];
 } __attribute__ ((packed));
 
+/**
+ * struct v4l2_plane_ext_pix_format - additional, per-plane format definition
+ * @sizeimage:		maximum size in bytes required for data, for which
+ *			this plane will be used.
+ *			Should be set to zero for unused planes.
+ * @bytesperline:	distance in bytes between the leftmost pixels in two
+ *			adjacent lines
+ * @reserved:		extra space reserved for future fields, must be set to 0
+ */
+struct v4l2_plane_ext_pix_format {
+	__u32 sizeimage;
+	__u32 bytesperline;
+	__u32 reserved;
+};
+
+/**
+ * struct v4l2_ext_pix_format - extended single/multiplanar format definition
+ * @type:		type of the data stream; V4L2_BUF_TYPE_VIDEO_CAPTURE or
+ *			V4L2_BUF_TYPE_VIDEO_OUTPUT
+ * @width:		image width in pixels
+ * @height:		image height in pixels
+ * @field:		enum v4l2_field; field order (for interlaced video)
+ * @pixelformat:	little endian four character code (fourcc)
+ * @modifier:		modifier applied to the format (used for tiled formats
+ *			and other kind of HW-specific formats, like compressed
+ *			formats)
+ * @colorspace:		enum v4l2_colorspace; supplemental to pixelformat
+ * @plane_fmt:		per-plane information
+ * @ycbcr_enc:		enum v4l2_ycbcr_encoding, Y'CbCr encoding
+ * @hsv_enc:		enum v4l2_hsv_encoding, HSV encoding
+ * @quantization:	enum v4l2_quantization, colorspace quantization
+ * @xfer_func:		enum v4l2_xfer_func, colorspace transfer function
+ * @reserved:		extra space reserved for future fields, must be set to 0
+ */
+struct v4l2_ext_pix_format {
+	__u32 type;
+	__u32 width;
+	__u32 height;
+	__u32 field;
+	__u64 modifier;
+	__u32 pixelformat;
+	__u32 colorspace;
+	struct v4l2_plane_ext_pix_format plane_fmt[VIDEO_MAX_PLANES];
+	union {
+		__u32 ycbcr_enc;
+		__u32 hsv_enc;
+	};
+	__u32 quantization;
+	__u32 xfer_func;
+	__u32 reserved[9];
+};
+
 /**
  * struct v4l2_sdr_format - SDR format definition
  * @pixelformat:	little endian four character code (fourcc)
@@ -2571,6 +2623,10 @@  struct v4l2_create_buffers {
 
 #define VIDIOC_QUERY_EXT_CTRL	_IOWR('V', 103, struct v4l2_query_ext_ctrl)
 
+#define VIDIOC_G_EXT_PIX_FMT	_IOWR('V', 104, struct v4l2_ext_pix_format)
+#define VIDIOC_S_EXT_PIX_FMT	_IOWR('V', 105, struct v4l2_ext_pix_format)
+#define VIDIOC_TRY_EXT_PIX_FMT	_IOWR('V', 106, struct v4l2_ext_pix_format)
+
 /* Reminder: when adding new ioctls please add support for them to
    drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */