From patchwork Mon Sep 10 15:00:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans Verkuil X-Patchwork-Id: 10594307 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F420A6CB for ; Mon, 10 Sep 2018 15:00:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E276D292B7 for ; Mon, 10 Sep 2018 15:00:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E0C5B29221; Mon, 10 Sep 2018 15:00:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7B7D4292CD for ; Mon, 10 Sep 2018 15:00:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728588AbeIJTzP (ORCPT ); Mon, 10 Sep 2018 15:55:15 -0400 Received: from lb3-smtp-cloud9.xs4all.net ([194.109.24.30]:54968 "EHLO lb3-smtp-cloud9.xs4all.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728219AbeIJTzP (ORCPT ); Mon, 10 Sep 2018 15:55:15 -0400 Received: from tschai.fritz.box ([212.251.195.8]) by smtp-cloud9.xs4all.net with ESMTPA id zNg8fEftlMsEFzNgBfBSip; Mon, 10 Sep 2018 17:00:44 +0200 From: Hans Verkuil To: linux-media@vger.kernel.org Cc: Ezequiel Garcia , Hans Verkuil Subject: [PATCH 1/2] vicodec: check for valid format in v4l2_fwht_en/decode Date: Mon, 10 Sep 2018 17:00:39 +0200 Message-Id: <20180910150040.39265-1-hverkuil@xs4all.nl> X-Mailer: git-send-email 2.18.0 X-CMAE-Envelope: MS4wfNkM7+yOC/6OagRvNEh6OXhbNZLl0JKnrlox50pue7IKGPg12A3zV8eGvUSSiGlQVlyuFEn2q9QglJ8bliiq00/JI6Kw+pHFUysbUP8DY1c5A/4wtL5Q WQUvZAB/9YHqVpd4+ECUdapysvoCnwpVV1rjrj4NpCvyvLHBMIJvgbg02l8YH0igyt9UgwmhHxqNxEe/74FJsq2g44ynS7qSKsTM+tU1C/Jz8+2evUynzCUy zWQea6FcEZR6zquFZLYUSw== Sender: linux-media-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Hans Verkuil These functions did not return an error if state->info was NULL or an unsupported pixelformat was selected (should not happen, but just to be on the safe side). Signed-off-by: Hans Verkuil --- drivers/media/platform/vicodec/codec-v4l2-fwht.c | 15 +++++++++++---- drivers/media/platform/vicodec/codec-v4l2-fwht.h | 7 ++----- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/drivers/media/platform/vicodec/codec-v4l2-fwht.c b/drivers/media/platform/vicodec/codec-v4l2-fwht.c index cfcf84b8574d..1b86eb9868c3 100644 --- a/drivers/media/platform/vicodec/codec-v4l2-fwht.c +++ b/drivers/media/platform/vicodec/codec-v4l2-fwht.c @@ -51,8 +51,7 @@ const struct v4l2_fwht_pixfmt_info *v4l2_fwht_get_pixfmt(u32 idx) return v4l2_fwht_pixfmts + idx; } -unsigned int v4l2_fwht_encode(struct v4l2_fwht_state *state, - u8 *p_in, u8 *p_out) +int v4l2_fwht_encode(struct v4l2_fwht_state *state, u8 *p_in, u8 *p_out) { unsigned int size = state->width * state->height; const struct v4l2_fwht_pixfmt_info *info = state->info; @@ -62,6 +61,8 @@ unsigned int v4l2_fwht_encode(struct v4l2_fwht_state *state, u32 encoding; u32 flags = 0; + if (!info) + return -EINVAL; rf.width = state->width; rf.height = state->height; rf.luma = p_in; @@ -137,6 +138,8 @@ unsigned int v4l2_fwht_encode(struct v4l2_fwht_state *state, rf.cr = rf.cb + 2; rf.luma++; break; + default: + return -EINVAL; } cf.width = state->width; @@ -180,8 +183,7 @@ unsigned int v4l2_fwht_encode(struct v4l2_fwht_state *state, return cf.size + sizeof(*p_hdr); } -int v4l2_fwht_decode(struct v4l2_fwht_state *state, - u8 *p_in, u8 *p_out) +int v4l2_fwht_decode(struct v4l2_fwht_state *state, u8 *p_in, u8 *p_out) { unsigned int size = state->width * state->height; unsigned int chroma_size = size; @@ -191,6 +193,9 @@ int v4l2_fwht_decode(struct v4l2_fwht_state *state, struct fwht_cframe cf; u8 *p; + if (!state->info) + return -EINVAL; + p_hdr = (struct fwht_cframe_hdr *)p_in; cf.width = ntohl(p_hdr->width); cf.height = ntohl(p_hdr->height); @@ -320,6 +325,8 @@ int v4l2_fwht_decode(struct v4l2_fwht_state *state, *p++ = 0; } break; + default: + return -EINVAL; } return 0; } diff --git a/drivers/media/platform/vicodec/codec-v4l2-fwht.h b/drivers/media/platform/vicodec/codec-v4l2-fwht.h index 7794c186d905..22ae0e4d7315 100644 --- a/drivers/media/platform/vicodec/codec-v4l2-fwht.h +++ b/drivers/media/platform/vicodec/codec-v4l2-fwht.h @@ -41,10 +41,7 @@ struct v4l2_fwht_state { const struct v4l2_fwht_pixfmt_info *v4l2_fwht_find_pixfmt(u32 pixelformat); const struct v4l2_fwht_pixfmt_info *v4l2_fwht_get_pixfmt(u32 idx); -unsigned int v4l2_fwht_encode(struct v4l2_fwht_state *state, - u8 *p_in, u8 *p_out); - -int v4l2_fwht_decode(struct v4l2_fwht_state *state, - u8 *p_in, u8 *p_out); +int v4l2_fwht_encode(struct v4l2_fwht_state *state, u8 *p_in, u8 *p_out); +int v4l2_fwht_decode(struct v4l2_fwht_state *state, u8 *p_in, u8 *p_out); #endif From patchwork Mon Sep 10 15:00:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans Verkuil X-Patchwork-Id: 10594309 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5875D109C for ; Mon, 10 Sep 2018 15:00:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 455AE292CB for ; Mon, 10 Sep 2018 15:00:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 42EF129239; Mon, 10 Sep 2018 15:00:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CEED2292EC for ; Mon, 10 Sep 2018 15:00:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728587AbeIJTzP (ORCPT ); Mon, 10 Sep 2018 15:55:15 -0400 Received: from lb3-smtp-cloud9.xs4all.net ([194.109.24.30]:42323 "EHLO lb3-smtp-cloud9.xs4all.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728127AbeIJTzP (ORCPT ); Mon, 10 Sep 2018 15:55:15 -0400 Received: from tschai.fritz.box ([212.251.195.8]) by smtp-cloud9.xs4all.net with ESMTPA id zNg8fEftlMsEFzNgCfBSjE; Mon, 10 Sep 2018 17:00:44 +0200 From: Hans Verkuil To: linux-media@vger.kernel.org Cc: Ezequiel Garcia , Hans Verkuil Subject: [PATCH 2/2] vicodec: set state->info before calling the encode/decode funcs Date: Mon, 10 Sep 2018 17:00:40 +0200 Message-Id: <20180910150040.39265-2-hverkuil@xs4all.nl> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180910150040.39265-1-hverkuil@xs4all.nl> References: <20180910150040.39265-1-hverkuil@xs4all.nl> X-CMAE-Envelope: MS4wfNkM7+yOC/6OagRvNEh6OXhbNZLl0JKnrlox50pue7IKGPg12A3zV8eGvUSSiGlQVlyuFEn2q9QglJ8bliiq00/JI6Kw+pHFUysbUP8DY1c5A/4wtL5Q WQUvZAB/9YHqVpd4+ECUdapysvoCnwpVV1rjrj4NpCvyvLHBMIJvgbg02l8YH0igyt9UgwmhHxqNxEe/74FJsq2g44ynS7qSKsTM+tU1C/Jz8+2evUynzCUy zWQea6FcEZR6zquFZLYUSw== Sender: linux-media-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Hans Verkuil state->info was NULL since I completely forgot to set state->info. Oops. Reported-by: Ezequiel Garcia Signed-off-by: Hans Verkuil Tested-by: Ezequiel Garcia --- drivers/media/platform/vicodec/vicodec-core.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/drivers/media/platform/vicodec/vicodec-core.c b/drivers/media/platform/vicodec/vicodec-core.c index fdd77441a47b..5d42a8414283 100644 --- a/drivers/media/platform/vicodec/vicodec-core.c +++ b/drivers/media/platform/vicodec/vicodec-core.c @@ -176,12 +176,15 @@ static int device_process(struct vicodec_ctx *ctx, } if (ctx->is_enc) { - unsigned int size = v4l2_fwht_encode(state, p_in, p_out); - - vb2_set_plane_payload(&out_vb->vb2_buf, 0, size); + state->info = q_out->info; + ret = v4l2_fwht_encode(state, p_in, p_out); + if (ret < 0) + return ret; + vb2_set_plane_payload(&out_vb->vb2_buf, 0, ret); } else { + state->info = q_cap->info; ret = v4l2_fwht_decode(state, p_in, p_out); - if (ret) + if (ret < 0) return ret; vb2_set_plane_payload(&out_vb->vb2_buf, 0, q_cap->sizeimage); }