From patchwork Mon Oct 8 19:08:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans Verkuil X-Patchwork-Id: 10631305 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A437E933 for ; Mon, 8 Oct 2018 19:08:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 96E832992E for ; Mon, 8 Oct 2018 19:08:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8995329978; Mon, 8 Oct 2018 19:08:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 294352992E for ; Mon, 8 Oct 2018 19:08:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726828AbeJICVo (ORCPT ); Mon, 8 Oct 2018 22:21:44 -0400 Received: from lb2-smtp-cloud8.xs4all.net ([194.109.24.25]:54374 "EHLO lb2-smtp-cloud8.xs4all.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726721AbeJICVo (ORCPT ); Mon, 8 Oct 2018 22:21:44 -0400 Received: from [192.168.2.10] ([212.251.195.8]) by smtp-cloud8.xs4all.net with ESMTPA id 9atHgoUy70ZZE9atKg8Xoo; Mon, 08 Oct 2018 21:08:31 +0200 To: Linux Media Mailing List , Mauro Carvalho Chehab From: Hans Verkuil Subject: [PATCH] vivid: fix kernel oops when enabling HFLIP and OSD Message-ID: <407e067b-47be-e8da-848d-edb6c04f5c1c@xs4all.nl> Date: Mon, 8 Oct 2018 21:08:27 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 Content-Language: en-US X-CMAE-Envelope: MS4wfOcTPYB2yZradZ4VAVDukyJxiUuTg5a0vSxyqVRSHttqVBBgxoRDUYIBk0450ZOvgQNWmHlaugD3surmjYNuUPJ4bjD4efDx9HD1rJLk00z+qxHO46Ky tPdD90ujQrU0qSZhI+rBUjznmzpSBKj8WL9dwWC5n2UwJfI3h8N7mOCtAFl4b4YTt5guCWM2wmMz45bpvX5Go6Fc7jnFn9oggg4= Sender: linux-media-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When the OSD is on (i.e. vivid displays text on top of the test pattern), and you enable hflip, then the driver crashes. The cause turned out to be a division of a negative number by an unsigned value. You expect that -8 / 2 would be -4, but in reality it is 2147483644 :-( Signed-off-by: Hans Verkuil Reported-by: Mauro Carvalho Chehab diff --git a/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c b/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c index f3d9c1140ffa..e76f87dc4368 100644 --- a/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c +++ b/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c @@ -1773,7 +1773,7 @@ typedef struct { u16 __; u8 _; } __packed x24; pos[7] = (chr & (0x01 << 0) ? fg : bg); \ } \ \ - pos += (tpg->hflip ? -8 : 8) / hdiv; \ + pos += (tpg->hflip ? -8 : 8) / (int)hdiv; \ } \ } \ } while (0)