From patchwork Fri Nov 22 10:06:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sakari Ailus X-Patchwork-Id: 13883015 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1619A1DED53 for ; Fri, 22 Nov 2024 10:06:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.9 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732270018; cv=none; b=boRoysFEpfDepx53j2yJZ+N6heJecQfVXfx3ngD4lGa5yR6+NDcJA0Yg1u2N2FpC2bjORULNoNJ6Z+7tqF96Uq7TaKS6d29bee5xOqNc0yEqDrXNLltjfkhTCNHKCTcqkzLJgpDY2+0fiCxsKFQTDq8kZmi2jvbDDlFIYByC9JM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1732270018; c=relaxed/simple; bh=tJsom9Lw1f0xcgHsaOpi52x91chLoU1mAB23TRtdTlM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pv9xbAu1vxaRGTwZCN8ijrCb437e3G7oYpU9ssMRpDdydHmmEIJBhT6IRIvjxE7ULJ/iJMFFrvgMfW2HFODMvjDj6KGmY0Ef56VmepnnkE0uFhm1KvWPGz5kr2k2aYM6u1lxv+LcXZMgwjHzfFwO8YyOwjVOpmLxG8X77419ItU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=himuXO2Z; arc=none smtp.client-ip=198.175.65.9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="himuXO2Z" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1732270015; x=1763806015; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tJsom9Lw1f0xcgHsaOpi52x91chLoU1mAB23TRtdTlM=; b=himuXO2ZxBqzCq33lEkKwPLc84XkoqS6euFN6/2R8UYI/hRwH78k2IKb 6XByq2y781pjaT4H0QV3gR0kbyI6n/+3omkLf4TG2SU3EQH+RI7s2eUQj /oye/3lllVFH2Jq1l2UmRvVP8/3Iw2UpAeTpzVsU0fulh/HaY2l+IgkDj xsvpQVmn/ox+NEbfVH1oQqCi40j8UVWdBFsHkEdhcMm8/UdV0Uw8OonEP rmNi7XCuhVA1kcbVVtS/47lShRNbBHOyj795clOOB+9YF6afsBor0Odcx nzvd7bhvTvEKYKoxKUkYE7TWEBbAIMQZODUQENQ3oyET80x6g95ZD+EaH Q==; X-CSE-ConnectionGUID: wBlF9Rt8RFazzH/y4zZoPg== X-CSE-MsgGUID: wffmJhMmRsmb/5m7Cb9BZQ== X-IronPort-AV: E=McAfee;i="6700,10204,11263"; a="54927599" X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="54927599" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by orvoesa101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Nov 2024 02:06:53 -0800 X-CSE-ConnectionGUID: e8VSYI2XTEGdo2Nod1rJ2A== X-CSE-MsgGUID: 1QPbZuR9SN+vWl+vMjlY7A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,175,1728975600"; d="scan'208";a="121403064" Received: from turnipsi.fi.intel.com (HELO kekkonen.fi.intel.com) ([10.237.72.44]) by orviesa002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Nov 2024 02:06:47 -0800 Received: from punajuuri.localdomain (punajuuri.localdomain [192.168.240.130]) by kekkonen.fi.intel.com (Postfix) with ESMTP id 9039F11FBCC; Fri, 22 Nov 2024 12:06:43 +0200 (EET) Received: from sailus by punajuuri.localdomain with local (Exim 4.96) (envelope-from ) id 1tEQYl-0002LR-1r; Fri, 22 Nov 2024 12:06:43 +0200 Organization: Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo From: Sakari Ailus To: linux-media@vger.kernel.org Cc: hverkuil@xs4all.nl, laurent.pinchart@ideasonboard.com, Prabhakar , Kate Hsuan , Alexander Shiyan , Mikhail Rudenko , Dave Stevenson , Tommaso Merciai , Umang Jain , Benjamin Mugnier , Sylvain Petinot , Christophe JAILLET , Julien Massot , Naushir Patuck , "Yan, Dongcheng" , "Cao, Bingbu" , "Qiu, Tian Shu" , "Wang, Hongju" , Stefan Klug , Mirela Rabulea , =?utf-8?q?Andr=C3=A9_Apitzsch?= , Heimir Thor Sverrisson , Kieran Bingham , Stanislaw Gruszka , Mehdi Djait Subject: [PATCH 4/8] media: Documentation: Add subdev configuration models, raw sensor model Date: Fri, 22 Nov 2024 12:06:29 +0200 Message-Id: <20241122100633.8971-5-sakari.ailus@linux.intel.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20241122100633.8971-1-sakari.ailus@linux.intel.com> References: <20241122100633.8971-1-sakari.ailus@linux.intel.com> Precedence: bulk X-Mailing-List: linux-media@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Sub-device configuration models define what V4L2 API elements are available on a compliant sub-device and how do they behave. The patch also adds a model for common raw sensors. Signed-off-by: Sakari Ailus --- .../media/drivers/camera-sensor.rst | 4 + .../media/v4l/common-raw-sensor.dia | 441 ++++++++++++++++++ .../media/v4l/common-raw-sensor.svg | 134 ++++++ .../userspace-api/media/v4l/dev-subdev.rst | 2 + .../media/v4l/subdev-config-model.rst | 203 ++++++++ 5 files changed, 784 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.dia create mode 100644 Documentation/userspace-api/media/v4l/common-raw-sensor.svg create mode 100644 Documentation/userspace-api/media/v4l/subdev-config-model.rst diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst index bc55c861fb69..5bc4c79d230c 100644 --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst @@ -18,6 +18,8 @@ binning functionality. The sensor drivers belong to two distinct classes, freely configurable and register list-based drivers, depending on how the driver configures this functionality. +Also see :ref:`media_subdev_config_model_common_raw_sensor`. + Freely configurable camera sensor drivers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -105,6 +107,8 @@ values programmed by the register sequences. The default values of these controls shall be 0 (disabled). Especially these controls shall not be inverted, independently of the sensor's mounting rotation. +.. _media_using_camera_sensor_drivers_embedded_data: + Embedded data ------------- diff --git a/Documentation/userspace-api/media/v4l/common-raw-sensor.dia b/Documentation/userspace-api/media/v4l/common-raw-sensor.dia new file mode 100644 index 000000000000..aa927527eae3 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/common-raw-sensor.dia @@ -0,0 +1,441 @@ + + + + + + + + + + + + + #A4# + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + #image data (1)# + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + #embedded data (2)# + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + #source pad (0)# + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Documentation/userspace-api/media/v4l/common-raw-sensor.svg b/Documentation/userspace-api/media/v4l/common-raw-sensor.svg new file mode 100644 index 000000000000..1d6055da2519 --- /dev/null +++ b/Documentation/userspace-api/media/v4l/common-raw-sensor.svg @@ -0,0 +1,134 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst index dcfcbd52490d..4d145bd3bd09 100644 --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst @@ -838,3 +838,5 @@ stream while it may be possible to enable and disable the embedded data stream. The embedded data format does not need to be configured on the sensor's pads as the format is dictated by the pixel data format in this case. + +.. include:: subdev-config-model.rst diff --git a/Documentation/userspace-api/media/v4l/subdev-config-model.rst b/Documentation/userspace-api/media/v4l/subdev-config-model.rst new file mode 100644 index 000000000000..25030b80734f --- /dev/null +++ b/Documentation/userspace-api/media/v4l/subdev-config-model.rst @@ -0,0 +1,203 @@ +.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later + +.. _media_subdev_config_model: + +Sub-device configuration models +=============================== + +A sub-device configuration model specifies in detail what the user space can +expect from a sub-device in terms of V4L2 sub-device interface support, +including IOCTL (including selection targets and controls) semantics. + +A sub-device may implement more than one configuration model at the same +time. The implemented configuration models can be obtained from the sub-device's +``V4L2_CID_CONFIG_MODEL`` control. + +.. _media_subdev_config_model_common_raw_sensor: + +Common raw camera sensor model +------------------------------ + +The common raw camera sensor model defines a set of enumeration and +configuration interfaces (formats, selections etc.) that cover the vast majority +of funcitionality of raw camera sensors. Not all of the interfaces are +necessarily offered by all drivers. + +A sub-device complies with the common raw sensor model if the +``V4L2_CONFIG_MODEL_COMMON_RAW`` bit is set in the ``V4L2_CID_CONFIG_MODEL`` +control of the sub-device. + +The common raw camera sensor model is aligned with +:ref:`media_using_camera_sensor_drivers`. Please refer to that regarding aspects +not specified here. + +Each camera sensor implementing the common raw sensor model exposes a single +V4L2 sub-device. The sub-device contains a single source pad (0) and two or more +internal pads: an image data internal pad (1) and optionally an embedded data +pad (2). Additionally, further internal pads may be supported for other +features, in which case they are documented separately for the given device. + +This is shown in :ref:`media_subdev_config_model_common_raw_sensor_subdev`. + +.. _media_subdev_config_model_common_raw_sensor_subdev: + +.. kernel-figure:: common-raw-sensor.svg + :alt: common-raw-sensor.svg + :align: center + + **Common raw sensor sub-device** + +Routes +^^^^^^ + +A sub-device conforming to common raw camera sensor model implements the +following routes. + +.. flat-table:: Routes + :header-rows: 1 + + * - Sink pad/stream + - Source pad/stream + - Static (X/M(aybe)/-) + - Mandatory (X/-) + - Synopsis + * - 1/0 + - 0/0 + - X + - X + - Image data + * - 2/0 + - 0/1 + - M + - \- + - Embedded data + +Some devices do not support the embedded data stream, others do support it and +in some of the latter, it can be turned on and off before streaming is started. + +Sensor pixel array size, cropping and binning +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The sensor's pixel array is divided into one or more areas. The areas around the +edge of the pixel array, usually one or more sides, may contain optical black +pixels, dummy pixels and other non-image pixels. + +A rectangle within the pixel array contains the visible pixels. Capturing the +non-visible pixels may be supported by the sensor. + +The sensor can perform three operations that affect the output image size. First +comes analogue crop. This configuration limits parts of the pixel array which +the sensor will read, affecting sensor timing as well. The granularity of the +analogue crop configuration varies greatly across sensors: some sensors support +a few different analogue crop configurations whereas others may support anything +divisible by a given number of pixels. + +The default analogue crop rectangle corresponds to the visible pixel area. + +In the next step, binning is performed on the image data read from camera +sensor's pixel array. This will effectively result in an image smaller than the +original by given binning factors horizontally and vertically. Typical values +are 1/2 and 1/3 but others may well be supported by the hardware as well. + +Sub-sampling follows binning. Sub-sampling, like binning, reduces the size of +the image by including only a subset of samples read from the sensor's pixel +matrix, typically every n'th pixel horizontally and vertically, taking the +sensor's colour pattern into account. Sub-sampling is generally configurable +separately horizontally and vertically. + +Scaling is further done after sub-sampling. Scaling generally is fine grainer +than binning or sub-sampling. + +The binning, sub-sampling and scaling operations are configured using the +compose rectangle. It depends on the driver which of these operations are being +used to achieve the resulting size. + +The combination of the analogue crop and binning, sub-sampling and scaling +operations may result in an image size that may be larger than desirable. For +this purpose, a digital crop operation may be performed on the image after these +operations. The resulting image size is further output by the sensor. + +Drivers may only support some of even none of these configurations, in which +case they do not expose the corresponding selection rectangles. + +Also refer to :ref:`Selection targets `. + +.. flat-table:: Selection targets on pads + :header-rows: 1 + + * - Pad/Stream + - Selection target/format + - Mandatory (X/-) + - Modifiable (X/-) + - Synopsis + * - 1/0 + - Format + - X + - \- + - Image data format. The width and height fields of this format are the + same than those for the V4L2_SEL_TGT_CROP_BOUNDS rectangle. The media + bus code of this format reflects the native pixel depth of the sensor. + * - 1/0 + - V4L2_SEL_TGT_CROP_BOUNDS + - X + - \- + - The full pixel area, including non-visible pixels. No pixels outside + this area can be captured. This rectangle is relative to the format on + the same (pad, stream). + * - 1/0 + - V4L2_SEL_TGT_CROP_DEFAULT + - X + - \ + - The visible pixel area. This rectangle is relative to the format on the + same (pad, stream). + * - 1/0 + - V4L2_SEL_TGT_CROP + - \- + - X + - Analogue crop. Analogue crop typically has a coarse granularity. This + rectangle is relative to the format on the same (pad, stream). + * - 1/0 + - V4L2_SEL_TGT_COMPOSE + - \- + - X + - Binning, sub-sampling and scaling. This rectangle is relative to the + V4L2_SEL_TGT_CROP rectangle on the same (pad, stream). The combination + of binning, sub-sampling and scaling is configured using this selection + target. + * - 2/0 + - Format + - X + - \- + - Embedded data format. + * - 0/0 + - V4L2_SEL_TGT_CROP + - \- + - X + - Digital crop. This rectangle is relative to the V4L2_SEL_TGT_COMPOSE + rectangle on (pad, stream) pair 1/0. + * - 0/0 + - Format + - X + - X + - Image data source format. The width and height fields of the format are + the same than for the V4L2_SEL_TGT_CROP rectangle on (pad, stream) pair + 0/0 and the media bus code reflects the pixel data output of the sensor. + * - 0/1 + - Format + - X + - \- + - Embedded data source format. + +Embedded data +^^^^^^^^^^^^^ + +The embedded data stream is produced by the sensor when the corresponding route +is enabled. The embedded data route may also be immutable or not exist at all, +in case the sensor (or the driver) does not support it. + +Generally the sensor embedded data width is determined by the width of the image +data whereas the number of lines are constant for the embedded data. The user +space may obtain the size of the embedded data once the image data size on the +source pad has been configured. + +Also see :ref:`media_using_camera_sensor_drivers_embedded_data`.