From patchwork Mon Mar 27 03:13:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Moudy Ho X-Patchwork-Id: 13188513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2CB3DC7619A for ; Mon, 27 Mar 2023 03:14:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:CC:To:From:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=5dmmAN53knGP8A38vOJ+8chP7YSe5P6lPPiDtcUaDnc=; b=iC/tFwL20BExWUhTn3y3mhOuGr QJEyjkREP5yM/09Y+dHwJWoLN4bfi/VRJLX8T7Wfi3Hzh0WAvclcrTgvjBZzf1b/rXJH5op6nkKea V2Uh4bPVNZ6ZyCWatDUgAbhKiiv8Eq/FhMwJo6//+Ihs5gn5aAMS1abapzUC9A3bQ5BECTLKh1x8F YtSJut3Ng7yXEG2fUfK9jalTzRxuWvIxKMTtoan+HxtcOKNQQOBgCT4SDv0qCPmtkUJJ8x0vzkfkR ILgjkPNqE6IdqkBVArm4ttA7IhG6PYPIjzybAF4uNAx6m8RMuRrivJZW3benTU1vrXxVfxkGTbhgO h60FYZXQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pgdJP-009c7h-1d; Mon, 27 Mar 2023 03:14:23 +0000 Received: from mailgw02.mediatek.com ([216.200.240.185]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pgdJL-009c4V-2F; Mon, 27 Mar 2023 03:14:21 +0000 X-UUID: 731f227ecc4d11ed8687db9d93187ff1-20230326 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mediatek.com; s=dk; h=Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:CC:To:From; bh=5dmmAN53knGP8A38vOJ+8chP7YSe5P6lPPiDtcUaDnc=; b=OjrRAIex9hIVUVhmnj8ikieVk4Vi3v3eqYWTIXrFKj8RzWB2lLxY0y5fl5M2mt08+p6YmkItzlhw4nXpvdCrlK6XTrwl9rYmJudhFgpkDmSRMTq+30HYXAv3r+Fu82Efjmev2X30lJaICWhNmBObRJclhUgKv8JMHs4LbL05y6A=; X-CID-P-RULE: Release_Ham X-CID-O-INFO: VERSION:1.1.22,REQID:7cb025b7-e894-4758-a84a-7d2c6421fdac,IP:0,U RL:0,TC:0,Content:0,EDM:0,RT:0,SF:0,FILE:0,BULK:0,RULE:Release_Ham,ACTION: release,TS:0 X-CID-META: VersionHash:120426c,CLOUDID:a8d88f29-564d-42d9-9875-7c868ee415ec,B ulkID:nil,BulkQuantity:0,Recheck:0,SF:102,TC:nil,Content:0,EDM:-3,IP:nil,U RL:0,File:nil,Bulk:nil,QS:nil,BEC:nil,COL:0,OSI:0,OSA:0,AV:0 X-CID-BVR: 0,NGT X-CID-BAS: 0,NGT,0,_ X-UUID: 731f227ecc4d11ed8687db9d93187ff1-20230326 Received: from mtkmbs13n2.mediatek.inc [(172.21.101.108)] by mailgw02.mediatek.com (envelope-from ) (musrelay.mediatek.com ESMTP with TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 256/256) with ESMTP id 931500606; Sun, 26 Mar 2023 20:14:13 -0700 Received: from mtkmbs13n2.mediatek.inc (172.21.101.108) by mtkmbs10n2.mediatek.inc (172.21.101.183) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.25; Mon, 27 Mar 2023 11:13:36 +0800 Received: from mtksdccf07.mediatek.inc (172.21.84.99) by mtkmbs13n2.mediatek.inc (172.21.101.73) with Microsoft SMTP Server id 15.2.1118.25 via Frontend Transport; Mon, 27 Mar 2023 11:13:36 +0800 From: Moudy Ho To: Mauro Carvalho Chehab , Matthias Brugger , Hans Verkuil , "Ping-Hsun Wu" CC: , , , , , Moudy Ho Subject: [PATCH v7 01/12] media: platform: mtk-mdp3: fix potential frame size overflow in mdp_try_fmt_mplane() Date: Mon, 27 Mar 2023 11:13:24 +0800 Message-ID: <20230327031335.9663-2-moudy.ho@mediatek.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20230327031335.9663-1-moudy.ho@mediatek.com> References: <20230327031335.9663-1-moudy.ho@mediatek.com> MIME-Version: 1.0 X-MTK: N X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230326_201419_742945_E19BDD8F X-CRM114-Status: GOOD ( 10.64 ) X-BeenThere: linux-mediatek@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-mediatek" Errors-To: linux-mediatek-bounces+linux-mediatek=archiver.kernel.org@lists.infradead.org Fix overflow risk when setting certain formats whose frame size exceeds a RGB24 with 7723x7723 resolution. For example, a 7723x7724 RGB24 frame: 1. bpl (byte per line) = 7723 * 3. 2. Overflow occurs when bpl * 7724 * depth. Fixes: 61890ccaefaf ("media: platform: mtk-mdp3: add MediaTek MDP3 driver") Signed-off-by: Moudy Ho --- drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c b/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c index 4e84a37ecdfc..36336d169bd9 100644 --- a/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c +++ b/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c @@ -4,6 +4,7 @@ * Author: Ping-Hsun Wu */ +#include #include #include #include @@ -428,14 +429,15 @@ const struct mdp_format *mdp_try_fmt_mplane(struct v4l2_format *f, u32 bpl = pix_mp->plane_fmt[i].bytesperline; u32 min_si, max_si; u32 si = pix_mp->plane_fmt[i].sizeimage; + u64 di; bpl = clamp(bpl, min_bpl, max_bpl); pix_mp->plane_fmt[i].bytesperline = bpl; - min_si = (bpl * pix_mp->height * fmt->depth[i]) / - fmt->row_depth[i]; - max_si = (bpl * s.max_height * fmt->depth[i]) / - fmt->row_depth[i]; + di = (u64)bpl * pix_mp->height * fmt->depth[i]; + min_si = (u32)div_u64(di, fmt->row_depth[i]); + di = (u64)bpl * s.max_height * fmt->depth[i]; + max_si = (u32)div_u64(di, fmt->row_depth[i]); si = clamp(si, min_si, max_si); pix_mp->plane_fmt[i].sizeimage = si;