diff mbox series

[v5,01/12] media: platform: mtk-mdp3: fix potential frame size overflow in mdp_try_fmt_mplane()

Message ID 20230113064104.22402-2-moudy.ho@mediatek.com (mailing list archive)
State New, archived
Headers show
Series Add support for multiple chips | expand

Commit Message

Moudy Ho Jan. 13, 2023, 6:40 a.m. UTC 
Fix overflow risk when setting certain formats whose frame size exceeds
a RGB24 with 7723x7723 resolution.

For example, a 7723x7724 RGB24 frame:
    1. bpl (byte per line) = 7723 * 3.
    2. Overflow occurs when bpl * 7724 * depth.

Fixes: 61890ccaefaf ("media: platform: mtk-mdp3: add MediaTek MDP3 driver")
Signed-off-by: Moudy Ho <moudy.ho@mediatek.com>
---
 drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
diff mbox series

Patch

diff --git a/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c b/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c
index 4e84a37ecdfc..c63bd5e479e0 100644
--- a/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c
+++ b/drivers/media/platform/mediatek/mdp3/mtk-mdp3-regs.c
@@ -432,9 +432,9 @@  const struct mdp_format *mdp_try_fmt_mplane(struct v4l2_format *f,
 		bpl = clamp(bpl, min_bpl, max_bpl);
 		pix_mp->plane_fmt[i].bytesperline = bpl;
 
-		min_si = (bpl * pix_mp->height * fmt->depth[i]) /
+		min_si = ((u64)bpl * pix_mp->height * fmt->depth[i]) /
 			 fmt->row_depth[i];
-		max_si = (bpl * s.max_height * fmt->depth[i]) /
+		max_si = ((u64)bpl * s.max_height * fmt->depth[i]) /
 			 fmt->row_depth[i];
 
 		si = clamp(si, min_si, max_si);