From patchwork Thu Apr 16 14:56:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dafna Hirschfeld X-Patchwork-Id: 11493097 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6457D14DD for ; Thu, 16 Apr 2020 14:56:27 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 40DF6206A2 for ; Thu, 16 Apr 2020 14:56:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="uh1wMiGb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40DF6206A2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=collabora.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-rockchip-bounces+patchwork-linux-rockchip=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=/mirWvuaYxHG+9YxBdB6jhLrnV+/Rn6+RLy56oWJoSI=; b=uh1 wMiGbDrKOjUomOVdNLvF1B4j0plRPxd4hCA67b/VTHvgM4L8vsD4kzpzTdKQouwsMEaAS8IbtXX0F bVJLO3odgJNEunRT6OOu2WRjxn2+AICvUNoNX1QWoCc9Emu/okpDltO34yqBHqMHAuN8IfwrzHK5q OP9pSDvrbwpXvfg4PgkNvvF1kwwYhrmWsGcFlV6hsKFFKDOza2KtZy7++QO6gvk5ZITCFJD587QOl W86IFjfMXqFinHct5LyxsLrZ2KSzQtn5gOS6n2weIlGgjSIH6/IRXWKSeJGIMS/tIREi10PTy3rBy lK21/bIqB67ysCatGG/22hmHIxypycA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jP5wE-0000u5-TJ; Thu, 16 Apr 2020 14:56:22 +0000 Received: from bhuna.collabora.co.uk ([2a00:1098:0:82:1000:25:2eeb:e3e3]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jP5w7-0000pS-Sf for linux-rockchip@lists.infradead.org; Thu, 16 Apr 2020 14:56:19 +0000 Received: from [127.0.0.1] (localhost [127.0.0.1]) (Authenticated sender: dafna) with ESMTPSA id 9DD042A05E1 From: Dafna Hirschfeld To: linux-media@vger.kernel.org, dafna.hirschfeld@collabora.com, helen.koike@collabora.com, ezequiel@collabora.com, hverkuil@xs4all.nl, kernel@collabora.com, dafna3@gmail.com, laurent.pinchart@ideasonboard.com, linux-rockchip@lists.infradead.org, sakari.ailus@linux.intel.com Subject: [RFC v3 1/2] v4l2: add support for colorspace conversion for video capture Date: Thu, 16 Apr 2020 16:56:04 +0200 Message-Id: <20200416145605.12399-1-dafna.hirschfeld@collabora.com> X-Mailer: git-send-email 2.17.1 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200416_075616_769517_2A9AC2F1 X-CRM114-Status: GOOD ( 22.95 ) X-Spam-Score: -0.0 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 UNPARSEABLE_RELAY Informational: message has unparseable relay lines X-BeenThere: linux-rockchip@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Upstream kernel work for Rockchip platforms List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Hans Verkuil , Philipp Zabel MIME-Version: 1.0 Sender: "Linux-rockchip" Errors-To: linux-rockchip-bounces+patchwork-linux-rockchip=patchwork.kernel.org@lists.infradead.org From: Philipp Zabel For video capture it is the driver that reports the colorspace, Y'CbCr/HSV encoding, quantization range and transfer function used by the video, and there is no way to request something different, even though many HDTV receivers have some sort of colorspace conversion capabilities. For output video this feature already exists since the application specifies this information for the video format it will send out, and the transmitter will enable any available CSC if a format conversion has to be performed in order to match the capabilities of the sink. For video capture we propose adding new pix_format flag: V4L2_PIX_FMT_FLAG_HAS_CSC. The flag is set by the application, the driver will interpret the ycbcr_enc/hsv_enc, and quantization fields as the requested colorspace information and will attempt to do the conversion it supports. Drivers set the flags V4L2_FMT_FLAG_CSC_YCBCR_ENC, V4L2_FMT_FLAG_CSC_HSV_ENC, V4L2_FMT_FLAG_CSC_HSV_QUANTIZATION, in the flags field of the struct v4l2_fmtdesc during enumeration to indicate that they support colorspace conversion for the respective field. Currently the conversion of the fields colorspace, xfer_func is not supported since there are no drivers that support it. The same API is added for the subdevices. With the flag V4L2_MBUS_FRAMEFMT_HAS_CSC set by the application in the VIDIOC_SUBDEV_S_FMT ioctl and the flags V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC, V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION set by the driver in the VIDIOC_SUBDEV_ENUM_MBUS_CODE ioctl. For subdevices, new 'flags' fields were added to the structs v4l2_subdev_mbus_code_enum, v4l2_mbus_framefmt which are borrowed from the 'reserved' field Drivers do not have to actually look at the flags: if the flags are not set, then the colorspace, ycbcr_enc and quantization fields are set to the default values by the core, i.e. just pass on the received format without conversion. Signed-off-by: Hans Verkuil Signed-off-by: Philipp Zabel Signed-off-by: Dafna Hirschfeld --- This is v3 of the RFC suggested originaly by Hans Verkuil: https://patchwork.linuxtv.org/patch/28847/ And then a v2 from Philipp Zabel: https://patchwork.kernel.org/project/linux-media/list/?series=15483 changes in v3: I added the API to subdevices as well and added fixes according to comments from Hans. I also added a usecase for the new API for the rkisp1 driver. changes in v2 (reported by Philipp Zabel): - convert to rst - split V4L2_PIX_FMT_FLAG_REQUEST_CSC into four separate flags for colorspace, ycbcr_enc/hsv_enc, quantization, and xfer_func - let driver set flags to indicate supported features [1] https://patchwork.linuxtv.org/patch/28847/ .../media/v4l/pixfmt-reserved.rst | 6 +++ .../media/v4l/pixfmt-v4l2-mplane.rst | 16 ++----- .../userspace-api/media/v4l/pixfmt-v4l2.rst | 36 +++++++++++++--- .../media/v4l/subdev-formats.rst | 42 +++++++++++++++++-- .../media/v4l/vidioc-enum-fmt.rst | 18 ++++++++ .../v4l/vidioc-subdev-enum-mbus-code.rst | 23 ++++++++++ drivers/media/v4l2-core/v4l2-ioctl.c | 19 ++++++++- drivers/media/v4l2-core/v4l2-subdev.c | 7 ++++ include/uapi/linux/v4l2-mediabus.h | 5 ++- include/uapi/linux/v4l2-subdev.h | 5 ++- include/uapi/linux/videodev2.h | 4 ++ 11 files changed, 158 insertions(+), 23 deletions(-) diff --git a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst index 59b9e7238f90..fa8dada69f8c 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt-reserved.rst @@ -280,3 +280,9 @@ please make a proposal on the linux-media mailing list. by RGBA values (128, 192, 255, 128), the same pixel described with premultiplied colors would be described by RGBA values (64, 96, 128, 128) + * - ``V4L2_PIX_FMT_FLAG_HAS_CSC`` + - 0x00000002 + - Set by the application. It is only used for capture and is + ignored for output streams. If set, then request the driver to do + colorspace conversion from the received colorspace, only conversions + of Ycbcr encoding, HSV encoding and quantization are supported. diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst index 444b4082684c..66f3365d7b72 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2-mplane.rst @@ -105,29 +105,21 @@ describing all planes of that format. * - __u8 - ``ycbcr_enc`` - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`. - This information supplements the ``colorspace`` and must be set by - the driver for capture streams and by the application for output - streams, see :ref:`colorspaces`. + See struct :c:type:`v4l2_pix_format`. * - __u8 - ``hsv_enc`` - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`. - This information supplements the ``colorspace`` and must be set by - the driver for capture streams and by the application for output - streams, see :ref:`colorspaces`. + See struct :c:type:`v4l2_pix_format`. * - } - * - __u8 - ``quantization`` - Quantization range, from enum :c:type:`v4l2_quantization`. - This information supplements the ``colorspace`` and must be set by - the driver for capture streams and by the application for output - streams, see :ref:`colorspaces`. + See struct :c:type:`v4l2_pix_format`. * - __u8 - ``xfer_func`` - Transfer function, from enum :c:type:`v4l2_xfer_func`. - This information supplements the ``colorspace`` and must be set by - the driver for capture streams and by the application for output - streams, see :ref:`colorspaces`. + See struct :c:type:`v4l2_pix_format`. * - __u8 - ``reserved[7]`` - Reserved for future extensions. Should be zeroed by drivers and diff --git a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst index 759420a872d6..ce57718cd66b 100644 --- a/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst +++ b/Documentation/userspace-api/media/v4l/pixfmt-v4l2.rst @@ -110,8 +110,8 @@ Single-planar format structure - ``colorspace`` - Image colorspace, from enum :c:type:`v4l2_colorspace`. This information supplements the ``pixelformat`` and must be set - by the driver for capture streams and by the application for - output streams, see :ref:`colorspaces`. + by the driver for capture streams and by the application for output + streams, see :ref:`colorspace`. * - __u32 - ``priv`` - This field indicates whether the remaining fields of the @@ -148,13 +148,31 @@ Single-planar format structure - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`. This information supplements the ``colorspace`` and must be set by the driver for capture streams and by the application for output - streams, see :ref:`colorspaces`. + streams, see :ref:`colorspaces`. If the application sets the + flag ``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set + this field for a capture stream to request a specific Y'CbCr encoding + for the captured image data. The driver will attempt to do the + conversion to the specified Y'CbCr encoding or return the encoding it + will use if it can't do the conversion. This field is ignored for + non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion + is supported by setting the flag V4L2_FMT_FLAG_CSC_YCBCR_ENC on the + on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration. + See :ref:`fmtdesc-flags` * - __u32 - ``hsv_enc`` - HSV encoding, from enum :c:type:`v4l2_hsv_encoding`. This information supplements the ``colorspace`` and must be set by the driver for capture streams and by the application for output - streams, see :ref:`colorspaces`. + streams, see :ref:`colorspaces`. If the application sets the flag + ``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set this + field for a capture stream to request a specific HSV encoding for the + captured image data. The driver will attempt to do the conversion to + the specified HSV encoding or return the encoding it will use if it + can't do the conversion. This field is ignored for non-HSV + pixelformats. The driver indicates that hsv_enc conversion + is supported by setting the flag V4L2_FMT_FLAG_CSC_HSV_ENC on the + on the corresponding struct :c:type:`v4l2_fmtdesc` during enumeration. + See :ref:`fmtdesc-flags` * - } - * - __u32 @@ -162,7 +180,15 @@ Single-planar format structure - Quantization range, from enum :c:type:`v4l2_quantization`. This information supplements the ``colorspace`` and must be set by the driver for capture streams and by the application for output - streams, see :ref:`colorspaces`. + streams, see :ref:`colorspaces`. If the application sets the flag + ``V4L2_PIX_FMT_FLAG_HAS_CSC`` then the application can set + this field for a capture stream to request a specific quantization + range for the captured image data. The driver will attempt to do the + conversion to the specified quantization range or return the + quantization it will use if it can't do the conversion. The driver + indicates that quantization conversion is supported by setting the flag + V4L2_FMT_FLAG_CSC_QUANTIZATION on the on the corresponding struct + :c:type:`v4l2_fmtdesc` during enumeration. See :ref:`fmtdesc-flags` * - __u32 - ``xfer_func`` - Transfer function, from enum :c:type:`v4l2_xfer_func`. diff --git a/Documentation/userspace-api/media/v4l/subdev-formats.rst b/Documentation/userspace-api/media/v4l/subdev-formats.rst index 9a4d61b0d76f..f1d4ca29a3e8 100644 --- a/Documentation/userspace-api/media/v4l/subdev-formats.rst +++ b/Documentation/userspace-api/media/v4l/subdev-formats.rst @@ -49,13 +49,33 @@ Media Bus Formats - Y'CbCr encoding, from enum :c:type:`v4l2_ycbcr_encoding`. This information supplements the ``colorspace`` and must be set by the driver for capture streams and by the application for output - streams, see :ref:`colorspaces`. + streams, see :ref:`colorspaces`. If the application sets the + flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set + this field for a capture stream to request a specific Y'CbCr encoding + for the mbus data. The driver will attempt to do the + conversion to the specified Y'CbCr encoding or return the encoding it + will use if it can't do the conversion. This field is ignored for + non-Y'CbCr pixelformats. The driver indicates that ycbcr_enc conversion + is supported by setting the flag V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC on + the corresponding struct c:type:`v4l2_subdev_mbus_code_enum` during + enumeration. See :ref:`vidioc-subdev-enum-mbus-code.rst ` + * - __u16 - ``quantization`` - Quantization range, from enum :c:type:`v4l2_quantization`. This information supplements the ``colorspace`` and must be set by the driver for capture streams and by the application for output - streams, see :ref:`colorspaces`. + streams, see :ref:`colorspaces`. If the application sets the + flag ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` the the application can set + this field for a capture stream to request a specific quantization + encoding for the mbus data. The driver will attempt to do the + conversion to the specified quantization or return the quantization it + will use if it can't do the conversion. The driver indicates that + quantization conversion is supported by setting the flag + V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION on the corresponding struct + c:type:`v4l2_subdev_mbus_code_enum` during enumeration. + See :ref:`vidioc-subdev-enum-mbus-code.rst ` + * - __u16 - ``xfer_func`` - Transfer function, from enum :c:type:`v4l2_xfer_func`. @@ -63,10 +83,26 @@ Media Bus Formats the driver for capture streams and by the application for output streams, see :ref:`colorspaces`. * - __u16 - - ``reserved``\ [11] + - ``flags`` + - flags see: :ref:v4l2-mbus-framefmt-flags + * - __u16 + - ``reserved``\ [10] - Reserved for future extensions. Applications and drivers must set the array to zero. +.. _v4l2-mbus-framefmt-flags: + +.. flat-table:: v4l2_mbus_framefmt Flags + :header-rows: 0 + :stub-columns: 0 + :widths: 3 1 4 + + * - ``V4L2_MBUS_FRAMEFMT_HAS_CSC`` + - 0x0001 + - Set by the application. It is only used for capture and is + ignored for output streams. If set, then request the subdevice to do + colorspace conversion from the received colorspace, only conversions + of Ycbcr encoding, and quantization are supported. .. _v4l2-mbus-pixelcode: diff --git a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst index 7e3142e11d77..125f074543af 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-enum-fmt.rst @@ -145,6 +145,24 @@ formats in preference order, where preferred formats are returned before parameters are detected. This flag can only be used in combination with the ``V4L2_FMT_FLAG_COMPRESSED`` flag, since this applies to compressed formats only. It is also only applies to stateful codecs. + * - ``V4L2_FMT_FLAG_CSC_YCBCR_ENC`` + - 0x0010 + - The driver allows the application to try to change the default + ycbcr encoding. This flag is relevant only for capture devices. + The application can ask to configure the ycbcr_enc of the capture device + when calling the :c:func:`VIDIOC_S_FMT` ioctl. + * - ``V4L2_FMT_FLAG_CSC_HSV_ENC`` + - 0x0010 + - The driver allows the application to try to change the default + hsv encoding. This flag is relevant only for capture devices. + The application can ask to configure the hsv_enc of the capture device + when calling the :c:func:`VIDIOC_S_FMT` ioctl. + * - ``V4L2_FMT_FLAG_CSC_QUANTIZATION`` + - 0x0020 + - The driver allows the application to try to change the default + quantization. This flag is relevant only for capture devices. + The application can ask to configure the quantization of the capture device + when calling the :c:func:`VIDIOC_S_FMT` ioctl. Return Value diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst index 35b8607203a4..4ad87cb74f57 100644 --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst @@ -78,11 +78,34 @@ information about the try formats. - ``which`` - Media bus format codes to be enumerated, from enum :ref:`v4l2_subdev_format_whence `. + * - __u32 + - ``flags`` + - see :ref:`v4l2-subdev-mbus-code-flags` * - __u32 - ``reserved``\ [8] - Reserved for future extensions. Applications and drivers must set the array to zero. +.. _v4l2-subdev-mbus-code-flags: + + +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{7.7cm}| + +.. flat-table:: flags in struct v4l2_subdev_mbus_code_enum + :header-rows: 0 + :stub-columns: 0 + :widths: 1 1 2 + + * - V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC + - 0x00000001 + - The driver supports setting the ycbcr encoding by the application + when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format` + on how to do this. + * - V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION + - 0x00000002 + - The driver supports setting the quantization by the application + when calling the VIDIOC_SUBDEV_S_FMT ioctl. see :ref:`v4l2-mbus-format` + on how to do this. Return Value ============ diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index b2ef8e60ea7d..3c7ffb6b15cb 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -1029,6 +1029,15 @@ static void v4l_sanitize_format(struct v4l2_format *fmt) fmt->fmt.pix_mp.num_planes = min_t(u32, fmt->fmt.pix_mp.num_planes, VIDEO_MAX_PLANES); + if (fmt->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) { + if (fmt->fmt.pix_mp.flags & V4L2_PIX_FMT_FLAG_HAS_CSC) + return; + fmt->fmt.pix_mp.colorspace = V4L2_COLORSPACE_DEFAULT; + fmt->fmt.pix_mp.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT; + fmt->fmt.pix_mp.quantization = V4L2_QUANTIZATION_DEFAULT; + fmt->fmt.pix_mp.xfer_func = V4L2_XFER_FUNC_DEFAULT; + } + /* * The v4l2_pix_format structure has been extended with fields that were * not previously required to be set to zero by applications. The priv @@ -1043,8 +1052,16 @@ static void v4l_sanitize_format(struct v4l2_format *fmt) fmt->type != V4L2_BUF_TYPE_VIDEO_OUTPUT) return; - if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC) + if (fmt->fmt.pix.priv == V4L2_PIX_FMT_PRIV_MAGIC) { + if (fmt->type != V4L2_BUF_TYPE_VIDEO_CAPTURE || + fmt->fmt.pix.flags & V4L2_PIX_FMT_FLAG_HAS_CSC) + return; + fmt->fmt.pix.colorspace = V4L2_COLORSPACE_DEFAULT; + fmt->fmt.pix.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT; + fmt->fmt.pix.quantization = V4L2_QUANTIZATION_DEFAULT; + fmt->fmt.pix.xfer_func = V4L2_XFER_FUNC_DEFAULT; return; + } fmt->fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC; diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c index a376b351135f..51e009936aad 100644 --- a/drivers/media/v4l2-core/v4l2-subdev.c +++ b/drivers/media/v4l2-core/v4l2-subdev.c @@ -477,6 +477,13 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg) case VIDIOC_SUBDEV_S_FMT: { struct v4l2_subdev_format *format = arg; + if (!(format->format.flags & V4L2_MBUS_FRAMEFMT_HAS_CSC)) { + format->format.colorspace = V4L2_COLORSPACE_DEFAULT; + format->format.xfer_func = V4L2_XFER_FUNC_DEFAULT; + format->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT; + format->format.quantization = V4L2_QUANTIZATION_DEFAULT; + } + memset(format->reserved, 0, sizeof(format->reserved)); memset(format->format.reserved, 0, sizeof(format->format.reserved)); return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->pad, format); diff --git a/include/uapi/linux/v4l2-mediabus.h b/include/uapi/linux/v4l2-mediabus.h index 123a231001a8..89ff0ad6a631 100644 --- a/include/uapi/linux/v4l2-mediabus.h +++ b/include/uapi/linux/v4l2-mediabus.h @@ -16,6 +16,8 @@ #include #include +#define V4L2_MBUS_FRAMEFMT_HAS_CSC 0x0001 + /** * struct v4l2_mbus_framefmt - frame format on the media bus * @width: image width @@ -36,7 +38,8 @@ struct v4l2_mbus_framefmt { __u16 ycbcr_enc; __u16 quantization; __u16 xfer_func; - __u16 reserved[11]; + __u16 flags; + __u16 reserved[10]; }; #ifndef __KERNEL__ diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h index 03970ce30741..4410b26a7158 100644 --- a/include/uapi/linux/v4l2-subdev.h +++ b/include/uapi/linux/v4l2-subdev.h @@ -65,6 +65,8 @@ struct v4l2_subdev_crop { __u32 reserved[8]; }; +#define V4L2_SUBDEV_MBUS_CODE_CSC_YCBCR_ENC 0x00000001 +#define V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION 0x00000002 /** * struct v4l2_subdev_mbus_code_enum - Media bus format enumeration * @pad: pad number, as reported by the media API @@ -77,7 +79,8 @@ struct v4l2_subdev_mbus_code_enum { __u32 index; __u32 code; __u32 which; - __u32 reserved[8]; + __u32 flags; + __u32 reserved[7]; }; /** diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 9817b7e2c968..adc9dd1080b8 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -772,6 +772,7 @@ struct v4l2_pix_format { /* Flags */ #define V4L2_PIX_FMT_FLAG_PREMUL_ALPHA 0x00000001 +#define V4L2_PIX_FMT_FLAG_HAS_CSC 0x00000002 /* * F O R M A T E N U M E R A T I O N @@ -789,6 +790,9 @@ struct v4l2_fmtdesc { #define V4L2_FMT_FLAG_EMULATED 0x0002 #define V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM 0x0004 #define V4L2_FMT_FLAG_DYN_RESOLUTION 0x0008 +#define V4L2_FMT_FLAG_CSC_YCBCR_ENC 0x0010 +#define V4L2_FMT_FLAG_CSC_HSV_ENC 0x0010 +#define V4L2_FMT_FLAG_CSC_QUANTIZATION 0x0020 /* Frame Size and frame rate enumeration */ /* From patchwork Thu Apr 16 14:56:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dafna Hirschfeld X-Patchwork-Id: 11493095 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B5C6081 for ; Thu, 16 Apr 2020 14:56:24 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 93C422051A for ; Thu, 16 Apr 2020 14:56:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="oO6cRGj2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 93C422051A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=collabora.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-rockchip-bounces+patchwork-linux-rockchip=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=tU+yOqXt3p56qYMEWISoV60ZpRGmYfL1QXu8M2+Uw68=; b=oO6cRGj2PIWEYu 1Qv51lgofZ8FbMuiI6ysqCwFCj/CAFnWcIGNRJNNJ0sw5EoYSLO75b7kwVtOyTckd0pfdxuuyIk6J zID4kgESmcjllzbkcfRULfOgvzaz5/GqK1Z7aKFlnelzjC6xbv94yRnbZ5qe79VmfBmnOaKcBl7zs mBlzRtuLnsqKNa0KDEUT3ENNkqCGonDxuyZWH2J4KKJDFT29fYw3Nzpc2P/aBd0WhXrLdh0J3SO42 rW2D6t7slRMMYapL0958GaUTZJRe2AZ+VTKQwhap8rSRa6U9F3KGFGgcb18Ni37Znrh9M7QJuqgdr QbzRw+03SruhtaFQ16FA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jP5wC-0000sg-Qq; Thu, 16 Apr 2020 14:56:20 +0000 Received: from bhuna.collabora.co.uk ([46.235.227.227]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jP5w8-0000pZ-N1 for linux-rockchip@lists.infradead.org; Thu, 16 Apr 2020 14:56:19 +0000 Received: from [127.0.0.1] (localhost [127.0.0.1]) (Authenticated sender: dafna) with ESMTPSA id 48E5D2A220A From: Dafna Hirschfeld To: linux-media@vger.kernel.org, dafna.hirschfeld@collabora.com, helen.koike@collabora.com, ezequiel@collabora.com, hverkuil@xs4all.nl, kernel@collabora.com, dafna3@gmail.com, laurent.pinchart@ideasonboard.com, linux-rockchip@lists.infradead.org, sakari.ailus@linux.intel.com Subject: [RFC v3 2/2] media: staging: rkisp1: allow quantization conversion from userspace for isp source pad Date: Thu, 16 Apr 2020 16:56:05 +0200 Message-Id: <20200416145605.12399-2-dafna.hirschfeld@collabora.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200416145605.12399-1-dafna.hirschfeld@collabora.com> References: <20200416145605.12399-1-dafna.hirschfeld@collabora.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200416_075617_026960_E3B907ED X-CRM114-Status: GOOD ( 12.97 ) X-Spam-Score: -0.0 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [46.235.227.227 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 UNPARSEABLE_RELAY Informational: message has unparseable relay lines X-BeenThere: linux-rockchip@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Upstream kernel work for Rockchip platforms List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "Linux-rockchip" Errors-To: linux-rockchip-bounces+patchwork-linux-rockchip=patchwork.kernel.org@lists.infradead.org The isp entity has a hardware support to force full range quantization for YUV formats. Use the new API to indicate userspace that quantization conversion is supported by adding the flag V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION during media code enumeration. Then uppon s_fmt on the video source pad, we assign the quantization from userspace for YUV formats. Also in the capture and resizer entities we retrieve the colorspace from the isp entity. Signed-off-by: Dafna Hirschfeld --- drivers/staging/media/rkisp1/rkisp1-capture.c | 28 ++++++------- drivers/staging/media/rkisp1/rkisp1-common.h | 2 + drivers/staging/media/rkisp1/rkisp1-isp.c | 39 +++++++++++++++++-- drivers/staging/media/rkisp1/rkisp1-resizer.c | 13 +++++++ 4 files changed, 65 insertions(+), 17 deletions(-) diff --git a/drivers/staging/media/rkisp1/rkisp1-capture.c b/drivers/staging/media/rkisp1/rkisp1-capture.c index fbf62399fe3d..aca0f93bc772 100644 --- a/drivers/staging/media/rkisp1/rkisp1-capture.c +++ b/drivers/staging/media/rkisp1/rkisp1-capture.c @@ -1066,14 +1066,13 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap, const struct v4l2_format_info **fmt_info) { const struct rkisp1_capture_config *config = cap->config; - struct rkisp1_capture *other_cap = - &cap->rkisp1->capture_devs[cap->id ^ 1]; const struct rkisp1_capture_fmt_cfg *fmt; const struct v4l2_format_info *info; const unsigned int max_widths[] = { RKISP1_RSZ_MP_SRC_MAX_WIDTH, RKISP1_RSZ_SP_SRC_MAX_WIDTH }; const unsigned int max_heights[] = { RKISP1_RSZ_MP_SRC_MAX_HEIGHT, RKISP1_RSZ_SP_SRC_MAX_HEIGHT}; + struct v4l2_subdev_format isp_sd_fmt; fmt = rkisp1_find_fmt_cfg(cap, pixm->pixelformat); if (!fmt) { @@ -1081,24 +1080,27 @@ static void rkisp1_try_fmt(const struct rkisp1_capture *cap, pixm->pixelformat = fmt->fourcc; } + rkisp1_get_isp_src_fmt(cap->rkisp1, &isp_sd_fmt); + pixm->field = isp_sd_fmt.format.field; + pixm->colorspace = isp_sd_fmt.format.colorspace; + pixm->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc; + pixm->xfer_func = isp_sd_fmt.format.xfer_func; + + /* + * isp has a feature to set full range quantization for yuv formats. + * so we need to get the format from the isp. + */ + pixm->quantization = isp_sd_fmt.format.quantization; + if (!v4l2_is_format_yuv(cap->pix.info)) + pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE; + pixm->width = clamp_t(u32, pixm->width, RKISP1_RSZ_SRC_MIN_WIDTH, max_widths[cap->id]); pixm->height = clamp_t(u32, pixm->height, RKISP1_RSZ_SRC_MIN_HEIGHT, max_heights[cap->id]); - pixm->field = V4L2_FIELD_NONE; - pixm->colorspace = V4L2_COLORSPACE_DEFAULT; - pixm->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT; - info = rkisp1_fill_pixfmt(pixm, cap->id); - /* can not change quantization when stream-on */ - if (other_cap->is_streaming) - pixm->quantization = other_cap->pix.fmt.quantization; - /* output full range by default, take effect in params */ - else if (!pixm->quantization || - pixm->quantization > V4L2_QUANTIZATION_LIM_RANGE) - pixm->quantization = V4L2_QUANTIZATION_FULL_RANGE; if (fmt_cfg) *fmt_cfg = fmt; diff --git a/drivers/staging/media/rkisp1/rkisp1-common.h b/drivers/staging/media/rkisp1/rkisp1-common.h index 2d7b7e078636..7a5576fa14c9 100644 --- a/drivers/staging/media/rkisp1/rkisp1-common.h +++ b/drivers/staging/media/rkisp1/rkisp1-common.h @@ -300,6 +300,8 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1, struct v4l2_device *v4l2_dev); void rkisp1_isp_unregister(struct rkisp1_device *rkisp1); +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1, + struct v4l2_subdev_format *sd_fmt); const struct rkisp1_isp_mbus_info *rkisp1_isp_mbus_info_get(u32 mbus_code); irqreturn_t rkisp1_isp_isr(struct rkisp1_device *rkisp1); diff --git a/drivers/staging/media/rkisp1/rkisp1-isp.c b/drivers/staging/media/rkisp1/rkisp1-isp.c index dee8e96f3900..6fdf5ed0b6b1 100644 --- a/drivers/staging/media/rkisp1/rkisp1-isp.c +++ b/drivers/staging/media/rkisp1/rkisp1-isp.c @@ -613,6 +613,10 @@ static int rkisp1_isp_enum_mbus_code(struct v4l2_subdev *sd, if (code->index == pos - 1) { code->code = fmt->mbus_code; + if (fmt->pixel_enc == V4L2_PIXEL_ENC_YUV && + dir == RKISP1_DIR_SRC) + code->flags = + V4L2_SUBDEV_MBUS_CODE_CSC_QUANTIZATION; return 0; } } @@ -639,12 +643,21 @@ static int rkisp1_isp_init_config(struct v4l2_subdev *sd, sink_crop->height = RKISP1_DEFAULT_HEIGHT; sink_crop->left = 0; sink_crop->top = 0; + sink_fmt->colorspace = V4L2_COLORSPACE_SRGB; + sink_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(sink_fmt->colorspace); + sink_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(sink_fmt->colorspace); + sink_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE; + src_fmt = v4l2_subdev_get_try_format(sd, cfg, RKISP1_ISP_PAD_SOURCE_VIDEO); *src_fmt = *sink_fmt; src_fmt->code = RKISP1_DEF_SRC_PAD_FMT; - src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE; + src_fmt->colorspace = V4L2_COLORSPACE_SRGB; + src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace); + src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace); + src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE; + src_crop = v4l2_subdev_get_try_crop(sd, cfg, RKISP1_ISP_PAD_SOURCE_VIDEO); @@ -687,10 +700,17 @@ static void rkisp1_isp_set_src_fmt(struct rkisp1_isp *isp, isp->src_fmt = mbus_info; src_fmt->width = src_crop->width; src_fmt->height = src_crop->height; - src_fmt->quantization = format->quantization; - /* full range by default */ - if (!src_fmt->quantization) + + src_fmt->colorspace = V4L2_COLORSPACE_SRGB; + src_fmt->ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(src_fmt->colorspace); + src_fmt->xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(src_fmt->colorspace); + + if (mbus_info->pixel_enc == V4L2_PIXEL_ENC_BAYER) src_fmt->quantization = V4L2_QUANTIZATION_FULL_RANGE; + else if (format->quantization == V4L2_QUANTIZATION_DEFAULT) + src_fmt->quantization = V4L2_QUANTIZATION_LIM_RANGE; + else + src_fmt->quantization = format->quantization; *format = *src_fmt; } @@ -1068,6 +1088,17 @@ int rkisp1_isp_register(struct rkisp1_device *rkisp1, return ret; } +int rkisp1_get_isp_src_fmt(struct rkisp1_device *rkisp1, + struct v4l2_subdev_format *sd_fmt) +{ + struct rkisp1_isp *isp = &rkisp1->isp; + + sd_fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE; + sd_fmt->pad = RKISP1_ISP_PAD_SOURCE_VIDEO; + + return v4l2_subdev_call(&isp->sd, pad, get_fmt, NULL, sd_fmt); +} + void rkisp1_isp_unregister(struct rkisp1_device *rkisp1) { struct v4l2_subdev *sd = &rkisp1->isp.sd; diff --git a/drivers/staging/media/rkisp1/rkisp1-resizer.c b/drivers/staging/media/rkisp1/rkisp1-resizer.c index 7b6b7ddd4169..8705b133de68 100644 --- a/drivers/staging/media/rkisp1/rkisp1-resizer.c +++ b/drivers/staging/media/rkisp1/rkisp1-resizer.c @@ -525,6 +525,7 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz, const struct rkisp1_isp_mbus_info *mbus_info; struct v4l2_mbus_framefmt *sink_fmt, *src_fmt; struct v4l2_rect *sink_crop; + struct v4l2_subdev_format isp_sd_fmt; sink_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SINK, which); src_fmt = rkisp1_rsz_get_pad_fmt(rsz, cfg, RKISP1_RSZ_PAD_SRC, which); @@ -539,8 +540,20 @@ static void rkisp1_rsz_set_sink_fmt(struct rkisp1_resizer *rsz, if (which == V4L2_SUBDEV_FORMAT_ACTIVE) rsz->pixel_enc = mbus_info->pixel_enc; + rkisp1_get_isp_src_fmt(rsz->rkisp1, &isp_sd_fmt); + + sink_fmt->field = isp_sd_fmt.format.field; + sink_fmt->colorspace = isp_sd_fmt.format.colorspace; + sink_fmt->ycbcr_enc = isp_sd_fmt.format.ycbcr_enc; + sink_fmt->xfer_func = isp_sd_fmt.format.xfer_func; + /* Propagete to source pad */ src_fmt->code = sink_fmt->code; + src_fmt->field = sink_fmt->field; + src_fmt->colorspace = sink_fmt->colorspace; + src_fmt->ycbcr_enc = sink_fmt->ycbcr_enc; + src_fmt->xfer_func = sink_fmt->xfer_func; + src_fmt->quantization = sink_fmt->quantization; sink_fmt->width = clamp_t(u32, format->width, rsz->config->min_rsz_width,