From patchwork Tue Apr 23 10:51:37 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Katsuya Matsubara X-Patchwork-Id: 2477031 Return-Path: X-Original-To: patchwork-linux-sh@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork2.kernel.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by patchwork2.kernel.org (Postfix) with ESMTP id 6C6B4DF2E5 for ; Tue, 23 Apr 2013 10:52:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756152Ab3DWKwi (ORCPT ); Tue, 23 Apr 2013 06:52:38 -0400 Received: from mail-pb0-f45.google.com ([209.85.160.45]:58182 "EHLO mail-pb0-f45.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754777Ab3DWKwh (ORCPT ); Tue, 23 Apr 2013 06:52:37 -0400 Received: by mail-pb0-f45.google.com with SMTP id ro2so132043pbb.4 for ; Tue, 23 Apr 2013 03:52:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:from:to:cc:subject:date:message-id:x-mailer:in-reply-to :references:x-gm-message-state; bh=dV31Lclp8PT8AL3jEnX4LsRT6DgzHtEMTK/JNPtSgGU=; b=ViZ7n8pkc+zWbSLCFykvuFaBR94lGqmOJAQuf4qTZXYWPwczwmPbU6KYS7E3095/iP GBm92D61UTJCod2wnwDbQq98XdfVPAY+uHxbcYYFPdptsK/Rg8bbtjKH2u4TYEzTjFLH a8m7E76z6kA95JxYYrleG/PjXJc/EF3pROsgVnMeeFj81Ia04DnvF4B5gODILST3Skbn EA1hoWFWi1O9ma8CP8hba5lMjmo3Tj97vOvXUWT4asaSrk23EMfVe6uErqqXNpcCMmc5 vsHbsBVc3LKb0vnVrpZ6mMQcX/WSwfFl90PPZ6bUsqU8P2yKhZCmQvAXIVGMV8DRDYw2 Z/5g== X-Received: by 10.68.228.161 with SMTP id sj1mr40448588pbc.151.1366714357277; Tue, 23 Apr 2013 03:52:37 -0700 (PDT) Received: from localhost.localdomain ([219.106.231.132]) by mx.google.com with ESMTPS id ih1sm26463098pbb.44.2013.04.23.03.52.35 (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 23 Apr 2013 03:52:36 -0700 (PDT) From: Katsuya Matsubara To: Guennadi Liakhovetski Cc: Mauro Carvalho Chehab , Hans Verkuil , linux-media@vger.kernel.org, linux-sh@vger.kernel.org, Katsuya Matsubara Subject: [PATCH 3/3] [media] sh_veu: fix the buffer size calculation Date: Tue, 23 Apr 2013 19:51:37 +0900 Message-Id: <1366714297-2784-4-git-send-email-matsu@igel.co.jp> X-Mailer: git-send-email 1.7.0.4 In-Reply-To: <1366714297-2784-1-git-send-email-matsu@igel.co.jp> References: <1366714297-2784-1-git-send-email-matsu@igel.co.jp> X-Gm-Message-State: ALoCoQncEV9CX3GEnfqVNEfwMYUqZPejpVcqnqk45aje7yXZj8/lhpKqEoM5lNtW8Ynt3HAkpOF5 Sender: linux-sh-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-sh@vger.kernel.org The 'bytesperline' value only indicates the stride of the Y plane if the color format is planar, such as NV12. When calculating the total plane size, the size of CbCr plane must also be considered. Signed-off-by: Katsuya Matsubara --- drivers/media/platform/sh_veu.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/media/platform/sh_veu.c b/drivers/media/platform/sh_veu.c index fa86c6f..358dac5 100644 --- a/drivers/media/platform/sh_veu.c +++ b/drivers/media/platform/sh_veu.c @@ -904,11 +904,11 @@ static int sh_veu_queue_setup(struct vb2_queue *vq, if (ftmp.fmt.pix.width != pix->width || ftmp.fmt.pix.height != pix->height) return -EINVAL; - size = pix->bytesperline ? pix->bytesperline * pix->height : - pix->width * pix->height * fmt->depth >> 3; + size = pix->bytesperline ? pix->bytesperline * pix->height * fmt->depth / fmt->ydepth : + pix->width * pix->height * fmt->depth / fmt->ydepth; } else { vfmt = sh_veu_get_vfmt(veu, vq->type); - size = vfmt->bytesperline * vfmt->frame.height; + size = vfmt->bytesperline * vfmt->frame.height * vfmt->fmt->depth / vfmt->fmt->ydepth; } if (count < 2)