From patchwork Wed Aug 2 21:45:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sakari Ailus X-Patchwork-Id: 13338878 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8BB2C04FDF for ; Wed, 2 Aug 2023 21:47:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231687AbjHBVrK (ORCPT ); Wed, 2 Aug 2023 17:47:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233863AbjHBVqn (ORCPT ); Wed, 2 Aug 2023 17:46:43 -0400 Received: from mgamail.intel.com (unknown [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21996421E for ; Wed, 2 Aug 2023 14:46:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1691012768; x=1722548768; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=+ouCd1zGZEjvkQkdTZTQJLdN3CnXlVKiGGig5j1B6U0=; b=fwFeXnzYbsuxQcVAvngYlcF/zyeeBbvkikFp+yNx6ioRbmAIWX+fas0k tFUnXAShtTUSpUlvtC+UhM5CNDIsLKB7d5RHtJ7glmM0ZlDtnSfaY9eAG usdJ0uhfGKQ8zworblbVXobqruT7jgjqbCf5WsnTYqw+7a4S2YkBygqAE eDr2oddltLRjjYjt10ddzxAJQZQCOkp4lwMMZZ+XqKqFVnDaaI7RAJfrj 8SOgyLkLZiYchNvDXin1AkLPShoiyPigYyUvqbBXymkvS9MJTFB192Vrl KjpKmSbQFqukWMkmlAEB6r3FnTCY78LneMl/X6gOG9di8uNqYAuQyVH++ w==; X-IronPort-AV: E=McAfee;i="6600,9927,10790"; a="372442040" X-IronPort-AV: E=Sophos;i="6.01,250,1684825200"; d="scan'208";a="372442040" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Aug 2023 14:46:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10790"; a="853009265" X-IronPort-AV: E=Sophos;i="6.01,250,1684825200"; d="scan'208";a="853009265" Received: from turnipsi.fi.intel.com (HELO kekkonen.fi.intel.com) ([10.237.72.44]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Aug 2023 14:46:00 -0700 Received: from svinhufvud.ger.corp.intel.com (localhost [IPv6:::1]) by kekkonen.fi.intel.com (Postfix) with ESMTP id AC6AE1204CF; Thu, 3 Aug 2023 00:45:56 +0300 (EEST) From: Sakari Ailus To: linux-media@vger.kernel.org Cc: Laurent Pinchart , tomi.valkeinen@ideasonboard.com, bingbu.cao@intel.com, hongju.wang@intel.com, hverkuil@xs4all.nl, Andrey Konovalov , Jacopo Mondi , Dmitry Perchanov Subject: [PATCH v2 0/9] Generic line based metadata support, internal pads Date: Thu, 3 Aug 2023 00:45:47 +0300 Message-Id: <20230802214556.180589-1-sakari.ailus@linux.intel.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org Hi folks, Here are a few patches to add support generic, line based metadata as well as internal pads. While the amount of code is not very large, to the contrary it is quite small actually IMO, I presume what this is about and why it is being proposed requires some explaining. Metadata mbus codes and formats have existed for some time in V4L2. They however have been only used by drivers that produce the data itself and effectively this metadata has always been statistics of some sort (at least when it comes to ISPs). What is different here is that we intend to add support for metadata originating from camera sensors. Camera sensors produce different kinds of metadata, embedded data (usually register address--value pairs used to capture the frame, in a more or less sensor specific format), histograms (in a very sensor specific format), dark pixels etc. The number of these formats is probably going to be about as large as image data formats if not larger, as the image data formats are much better standardised but a smaller subset of them will be supported by V4L2, at least initially but possibly much more in the long run. Having this many device specific formats would be a major problem for all the other drivers along that pipeline (not to mention the users of those drivers), including bridge (e.g. CSI-2 to parallel) but especially CSI-2 receiver drivers that have DMA: the poor driver developer would not only need to know camera sensor specific formats but to choose the specific packing of that format suitable for the DMA used by the hardware. It is unlikely many of these would ever get tested while being present on the driver API. Also adding new sensors with new embedded data formats would involve updating all bridge and CSI-2 receiver drivers. I don't expect this to be a workable approach. Instead what I'm proposing is to use specific metadata formats on the sensor devices only, on internal pads (more about those soon) of the sensors, only visible in the UAPI, and then generic mbus formats along the pipeline and finally generic V4L2 metadata formats on the DMAs (specific to bit depth and packing). This would unsnarl the two, defining what data there is (specific mbus code) and how that is transported and packed (generic mbus codes and V4L2 formats). The user space would be required to "know" the path of that data from the sensor's internal pad to the V4L2 video node. I do not see this as these devices require at least some knowledge of the pipeline, i.e. hardware at hand. Separating what the data means and how it is packed may even be beneficial: it allows separating code that interprets the data (sensor internal mbus code) from the code that accesses it (packing). These formats are in practice line based, meaning that there may be padding at the end of the line, depending on the bus as well as the DMA. If non-line based formats are needed, it is always possible to set the "height" field to 1. The internal (source) pads are an alternative to source routes [1]. The source routes were not universally liked and I do have to say I like re-using existing interface concepts (pads and everything you can do with pads, including access format, selections etc.) wherever it makes sense, instead of duplicating functionality. Effectively internal source pads behave mostly just like sink pads, but they describe a flow of data that originates from a sub-device instead of arriving to a sub-device. The SUBDEV_S_ROUTING IOCTLs are used to enable and disable routes from internal source pads to sub-device's source pads. The subdev format IOCTLs are usable, too, so one can find which subdev format is available on given internal source pad. This set depends on these patches: I've also pushed these here and I'll keep updating the branch: Questions and comments are most welcome. Driver support is to be added, as well as an example. [1] since v2: - Added an example demonstrating the use of internal pads. --- Is the level of detail enough for the purpose? - Improved documentation. - Added a macro to tell whether a format is a metadata format. (Documentation could be added.) - A small ReST syntax fix in the same section. - Drop leftovers of a patch checking for the INTERNAL_SOURCE flag. since v1: - Make the new pad flag just "INTERNAL", requiring either SINK or SOURCE pad flag to accompany it. Removed the union in struct v4l2_subdev_route. - Add the term "stream" to MC glossary. - Improved and fixed documentation (according to comments). - Note these formats are little endian. - Remove 1X8 from the names of the mbus codes. These formats have generally 8 bits per pixel. - Fix mbus code numbering (had holes in RFC). - Add new metadata fields to debug prints. - Fix a minor documentation build issue. Sakari Ailus (9): media: Documentation: Align numbered list media: mc: Check pad flag validity media: mc: Add INTERNAL pad flag media: uapi: v4l: Document source routes media: uapi: Add generic serial metadata mbus formats media: uapi: Document which mbus format fields are valid for metadata media: uapi: Add a macro to tell whether an mbus code is metadata media: uapi: Add generic 8-bit metadata format definitions media: v4l: Support line-based metadata capture .../userspace-api/media/glossary.rst | 6 + .../media/mediactl/media-types.rst | 6 + .../userspace-api/media/v4l/dev-meta.rst | 15 + .../userspace-api/media/v4l/dev-subdev.rst | 144 +++++++- .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-generic.rst | 331 ++++++++++++++++++ .../media/v4l/subdev-formats.rst | 257 ++++++++++++++ .../media/v4l/vidioc-enum-fmt.rst | 7 + .../media/videodev2.h.rst.exceptions | 1 + drivers/media/mc/mc-entity.c | 17 +- drivers/media/v4l2-core/v4l2-ioctl.c | 13 +- include/uapi/linux/media-bus-format.h | 12 + include/uapi/linux/media.h | 1 + include/uapi/linux/v4l2-mediabus.h | 18 +- include/uapi/linux/videodev2.h | 19 + 15 files changed, 824 insertions(+), 24 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-generic.rst