From patchwork Sun Feb 14 19:51:37 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hans de Goede X-Patchwork-Id: 8302971 Return-Path: X-Original-To: patchwork-linux-media@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id E02099F38B for ; Sun, 14 Feb 2016 19:51:57 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 0C1B0204DE for ; Sun, 14 Feb 2016 19:51:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1923C204D5 for ; Sun, 14 Feb 2016 19:51:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751502AbcBNTvs (ORCPT ); Sun, 14 Feb 2016 14:51:48 -0500 Received: from mx1.redhat.com ([209.132.183.28]:59935 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751283AbcBNTvo (ORCPT ); Sun, 14 Feb 2016 14:51:44 -0500 Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (Postfix) with ESMTPS id 0FE10C0C2356; Sun, 14 Feb 2016 19:51:44 +0000 (UTC) Received: from shalem.localdomain.com (vpn1-7-74.ams2.redhat.com [10.36.7.74]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u1EJpdwr029190; Sun, 14 Feb 2016 14:51:40 -0500 From: Hans de Goede To: Hans Verkuil , Mauro Carvalho Chehab Cc: Stas Sergeev , Linux Media Mailing List , Hans de Goede , stable@vger.kernel.org Subject: [PATCH] saa7134: Fix bytesperline not being set correctly for planar formats Date: Sun, 14 Feb 2016 20:51:37 +0100 Message-Id: <1455479497-6324-1-git-send-email-hdegoede@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 Sender: linux-media-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP bytesperline should be the bytesperline for the first plane for planar formats, not that of all planes combined. This fixes a crash in xawtv caused by the wrong bpl. Cc: stable@vger.kernel.org BugLink: https://bugzilla.redhat.com/show_bug.cgi?id=1305389 Reported-and-tested-by: Stas Sergeev Signed-off-by: Hans de Goede --- drivers/media/pci/saa7134/saa7134-video.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/drivers/media/pci/saa7134/saa7134-video.c b/drivers/media/pci/saa7134/saa7134-video.c index a63c136..1293563b 100644 --- a/drivers/media/pci/saa7134/saa7134-video.c +++ b/drivers/media/pci/saa7134/saa7134-video.c @@ -1219,10 +1219,13 @@ static int saa7134_g_fmt_vid_cap(struct file *file, void *priv, f->fmt.pix.height = dev->height; f->fmt.pix.field = dev->field; f->fmt.pix.pixelformat = dev->fmt->fourcc; - f->fmt.pix.bytesperline = - (f->fmt.pix.width * dev->fmt->depth) >> 3; + if (dev->fmt->planar) + f->fmt.pix.bytesperline = f->fmt.pix.width; + else + f->fmt.pix.bytesperline = + (f->fmt.pix.width * dev->fmt->depth) / 8; f->fmt.pix.sizeimage = - f->fmt.pix.height * f->fmt.pix.bytesperline; + (f->fmt.pix.height * f->fmt.pix.width * dev->fmt->depth) / 8; f->fmt.pix.colorspace = V4L2_COLORSPACE_SMPTE170M; return 0; } @@ -1298,10 +1301,13 @@ static int saa7134_try_fmt_vid_cap(struct file *file, void *priv, if (f->fmt.pix.height > maxh) f->fmt.pix.height = maxh; f->fmt.pix.width &= ~0x03; - f->fmt.pix.bytesperline = - (f->fmt.pix.width * fmt->depth) >> 3; + if (fmt->planar) + f->fmt.pix.bytesperline = f->fmt.pix.width; + else + f->fmt.pix.bytesperline = + (f->fmt.pix.width * fmt->depth) / 8; f->fmt.pix.sizeimage = - f->fmt.pix.height * f->fmt.pix.bytesperline; + (f->fmt.pix.height * f->fmt.pix.width * fmt->depth) / 8; f->fmt.pix.colorspace = V4L2_COLORSPACE_SMPTE170M; return 0;