From patchwork Tue Jan 29 09:49:58 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans Verkuil X-Patchwork-Id: 2060671 Return-Path: X-Original-To: patchwork-linux-media@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork1.kernel.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by patchwork1.kernel.org (Postfix) with ESMTP id 745953FCD5 for ; Tue, 29 Jan 2013 09:50:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753242Ab3A2Juf (ORCPT ); Tue, 29 Jan 2013 04:50:35 -0500 Received: from ams-iport-2.cisco.com ([144.254.224.141]:26630 "EHLO ams-iport-2.cisco.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755064Ab3A2Jue (ORCPT ); Tue, 29 Jan 2013 04:50:34 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av0EABOZB1GQ/khL/2dsb2JhbABEvlcWc4JMEz+BPhOIEbBMkCSNCIN9A5YNiUyGfIJ4gWU X-IronPort-AV: E=Sophos;i="4.84,559,1355097600"; d="scan'208";a="80081872" Received: from ams-core-2.cisco.com ([144.254.72.75]) by ams-iport-2.cisco.com with ESMTP; 29 Jan 2013 09:50:32 +0000 Received: from cobaltpc1.localnet (dhcp-10-54-92-107.cisco.com [10.54.92.107]) by ams-core-2.cisco.com (8.14.5/8.14.5) with ESMTP id r0T9oWJt006468; Tue, 29 Jan 2013 09:50:32 GMT From: Hans Verkuil To: "linux-media" Subject: [RFC PATCH] em28xx: fix bytesperline calculation in TRY_FMT Date: Tue, 29 Jan 2013 10:49:58 +0100 User-Agent: KMail/1.13.7 (Linux/3.7-1.slh.1-aptosid-amd64; KDE/4.8.4; x86_64; ; ) Cc: Devin Heitmueller MIME-Version: 1.0 Message-Id: <201301291049.58085.hverkuil@xs4all.nl> Sender: linux-media-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org This was part of my original em28xx patch series. That particular patch combined two things: this fix and the change where TRY_FMT would no longer return -EINVAL for unsupported pixelformats. The latter change was rejected (correctly), but we all forgot about the second part of the patch which fixed a real bug. I'm reposting just that fix. Regards, Hans The bytesperline calculation was incorrect: it used the old width instead of the provided width, and it miscalculated the bytesperline value for the depth == 12 case. Signed-off-by: Hans Verkuil Reviewed-by: Devin Heitmueller --- drivers/media/usb/em28xx/em28xx-video.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/media/usb/em28xx/em28xx-video.c b/drivers/media/usb/em28xx/em28xx-video.c index 2eabf2a..070506d 100644 --- a/drivers/media/usb/em28xx/em28xx-video.c +++ b/drivers/media/usb/em28xx/em28xx-video.c @@ -906,7 +906,7 @@ static int vidioc_try_fmt_vid_cap(struct file *file, void *priv, f->fmt.pix.width = width; f->fmt.pix.height = height; f->fmt.pix.pixelformat = fmt->fourcc; - f->fmt.pix.bytesperline = (dev->width * fmt->depth + 7) >> 3; + f->fmt.pix.bytesperline = width * ((fmt->depth + 7) >> 3); f->fmt.pix.sizeimage = f->fmt.pix.bytesperline * height; f->fmt.pix.colorspace = V4L2_COLORSPACE_SMPTE170M; if (dev->progressive)