From patchwork Thu Aug 5 14:42:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrzej Pietrasiewicz X-Patchwork-Id: 12421327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, UNPARSEABLE_RELAY,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4544CC4338F for ; Thu, 5 Aug 2021 14:45:32 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 160D761151 for ; Thu, 5 Aug 2021 14:45:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 160D761151 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=collabora.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=pQ9md3k0ky83ncMEAUS4vuvGjbjcAGhVA75T1/Z6xdg=; b=LM4RVYXFDLL2W2 YyELonU/zwCBRZUryVNt/B4xnreuaMiq8n3gIkTutpqfFUvGEWD3TZRYnw1iW5TyTBf/m7d3sSN8i DYjGvSftef0rLjaqgkoYrOW2vYrssfT94GfQ6Bh97H/QZbLJadftj2ObhQ51Xq1yhXn3Hodi4vCey 9DLjBVzsGcS6VSpzwSOl3efUIsbb4JRzkUOuajAVj6gQuPLeIEbXRUkKRE+UI/J+8tdUV2WLaj62p Sznk9oXWnB67RQqWIBUxEXzDKljcPZ3bvQ7fGUIMcvwI+PF8UVKJ7vB7UXrfkHYHZVa+0p3xvgAJn yujgqAkMoWQNH+YyxJ+w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mBeaw-009tCN-UO; Thu, 05 Aug 2021 14:43:39 +0000 Received: from bhuna.collabora.co.uk ([46.235.227.227]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mBeaI-009sob-FK; Thu, 05 Aug 2021 14:43:01 +0000 Received: from [127.0.0.1] (localhost [127.0.0.1]) (Authenticated sender: andrzej.p) with ESMTPSA id C14C51F44097 From: Andrzej Pietrasiewicz To: linux-media@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-rockchip@lists.infradead.org, linux-staging@lists.linux.dev Cc: Andrzej Pietrasiewicz , Benjamin Gaignard , Boris Brezillon , Ezequiel Garcia , Fabio Estevam , Greg Kroah-Hartman , Hans Verkuil , Heiko Stuebner , Jernej Skrabec , Mauro Carvalho Chehab , Nicolas Dufresne , NXP Linux Team , Pengutronix Kernel Team , Philipp Zabel , Sascha Hauer , Shawn Guo , kernel@collabora.com Subject: [PATCH v3 00/10] VP9 codec V4L2 control interface Date: Thu, 5 Aug 2021 16:42:36 +0200 Message-Id: <20210805144246.11998-1-andrzej.p@collabora.com> X-Mailer: git-send-email 2.17.1 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210805_074258_745430_0CA42596 X-CRM114-Status: GOOD ( 20.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Dear all, This patch series adds VP9 codec V4L2 control interface and two drivers using the new controls. It is a follow-up of previous RFC v2 series [1]. In this new iteration, we've implemented VP9 hardware decoding on two devices: Rockchip VDEC and Hantro G2, and tested on RK3399, i.MX8MQ and i.MX8MP. The i.MX8M driver needs proper power domains support, though, which is a subject of a different effort, but in all 3 cases we were able to run the drivers. GStreamer support is also available, the needed changes have been submitted by Daniel Almeida [2]. This MR is ready to be merged, and just needs the VP9 V4L2 controls to be merged and released. Both rkvdec and hantro drivers are passing a significant number of VP9 tests using Fluster[3]. There are still a few tests that are not passing, due to dynamic frame resize (not yet supported by V4L2) and small size videos (due to IP block limitations). The series adds the VP9 codec V4L2 control API as uAPI, so it aims at being merged without passing through staging, as agreed[4]. The ABI has been checked for padding and verified to contain no holes. Please note, though, that I kindly ask not to merge this until Nicolas Dufresne reviews it. [1] https://patchwork.linuxtv.org/project/linux-media/cover/20210505123836.9573-1-andrzej.p@collabora.com/ [2] https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2144 [3] https://github.com/fluendo/fluster [4] https://lore.kernel.org/linux-media/b8f83c93-67fd-09f5-9314-15746cbfdc61@xs4all.nl/ Changes related to the RFC v2: - added another driver including a postprocessor to de-tile codec-specific tiling - reworked uAPI structs layout to follow VP8 style - changed validation of loop filter params - changed validation of segmentation params - changed validation of VP9 frame params - removed level lookup array from loop filter struct (can be computed by drivers) - renamed some enum values to match the spec more closely - V4L2 VP9 library changed the 'eob' member of 'struct v4l2_vp9_frame_symbol_counts' so that it is an array of pointers instead of an array of pointers to arrays (IPs such as g2 creatively pass parts of the 'eob' counts in the 'coeff' counts) - factored out several repeated portions of code - minor nitpicks and cleanups The series depends on the YUV tiled format support prepared by Ezequiel: https://patchwork.linuxtv.org/project/linux-media/list/?series=6049. Andrzej Pietrasiewicz (4): media: uapi: Add VP9 stateless decoder controls media: Add VP9 v4l2 library media: hantro: Prepare for other G2 codecs media: hantro: Support VP9 on the G2 core Boris Brezillon (1): media: rkvdec: Add the VP9 backend Ezequiel Garcia (5): hantro: postproc: Fix motion vector space size hantro: postproc: Introduce struct hantro_postproc_ops hantro: Simplify postprocessor hantro: Add quirk for NV12/NV12_4L4 capture format media: hantro: Support NV12 on the G2 core .../userspace-api/media/v4l/biblio.rst | 10 + .../media/v4l/ext-ctrls-codec-stateless.rst | 545 +++++ .../media/v4l/pixfmt-compressed.rst | 15 + .../media/v4l/vidioc-g-ext-ctrls.rst | 8 + .../media/v4l/vidioc-queryctrl.rst | 12 + .../media/videodev2.h.rst.exceptions | 2 + drivers/media/v4l2-core/Kconfig | 4 + drivers/media/v4l2-core/Makefile | 1 + drivers/media/v4l2-core/v4l2-ctrls-core.c | 174 ++ drivers/media/v4l2-core/v4l2-ctrls-defs.c | 8 + drivers/media/v4l2-core/v4l2-ioctl.c | 1 + drivers/media/v4l2-core/v4l2-vp9.c | 1850 +++++++++++++++++ drivers/staging/media/hantro/Kconfig | 1 + drivers/staging/media/hantro/Makefile | 7 +- drivers/staging/media/hantro/hantro.h | 40 +- drivers/staging/media/hantro/hantro_drv.c | 18 +- drivers/staging/media/hantro/hantro_g2.c | 27 + .../staging/media/hantro/hantro_g2_hevc_dec.c | 31 - drivers/staging/media/hantro/hantro_g2_regs.h | 104 + .../staging/media/hantro/hantro_g2_vp9_dec.c | 978 +++++++++ drivers/staging/media/hantro/hantro_hw.h | 83 +- .../staging/media/hantro/hantro_postproc.c | 79 +- drivers/staging/media/hantro/hantro_v4l2.c | 20 + drivers/staging/media/hantro/hantro_vp9.c | 240 +++ drivers/staging/media/hantro/hantro_vp9.h | 103 + drivers/staging/media/hantro/imx8m_vpu_hw.c | 38 +- .../staging/media/hantro/rockchip_vpu_hw.c | 7 +- .../staging/media/hantro/sama5d4_vdec_hw.c | 3 +- drivers/staging/media/rkvdec/Kconfig | 1 + drivers/staging/media/rkvdec/Makefile | 2 +- drivers/staging/media/rkvdec/rkvdec-vp9.c | 1078 ++++++++++ drivers/staging/media/rkvdec/rkvdec.c | 52 +- drivers/staging/media/rkvdec/rkvdec.h | 12 +- include/media/v4l2-ctrls.h | 4 + include/media/v4l2-vp9.h | 182 ++ include/uapi/linux/v4l2-controls.h | 433 ++++ include/uapi/linux/videodev2.h | 6 + 37 files changed, 6108 insertions(+), 71 deletions(-) create mode 100644 drivers/media/v4l2-core/v4l2-vp9.c create mode 100644 drivers/staging/media/hantro/hantro_g2.c create mode 100644 drivers/staging/media/hantro/hantro_g2_vp9_dec.c create mode 100644 drivers/staging/media/hantro/hantro_vp9.c create mode 100644 drivers/staging/media/hantro/hantro_vp9.h create mode 100644 drivers/staging/media/rkvdec/rkvdec-vp9.c create mode 100644 include/media/v4l2-vp9.h base-commit: bfee75f73c37a2f46a6326eaa06f5db701f76f01