From patchwork Tue Jul 9 13:29:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Scally X-Patchwork-Id: 13728034 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5A24BC2BD09 for ; Tue, 9 Jul 2024 13:33:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=KPIMgrur3zorH78ok84GEte0NHdwKg53SOXsMuhMOyU=; b=KWbt1AaXfX0+9RhK18ZXNtXRoo XdwmMiGY5j3m+4GlaiB0OgAXL47C+87rMVnsLPhHPnK8N1B92WUAzVjfmizAcf4Ez7/ZsGZn8QaB1 62ZouW+dkHW2k4FMQFnuPqXdRV5/kYsa7KpJGwOMdxziLBBNrvvKKz6SF/XX5WQ2q9eiBjguj3mvH jxjRj1ksihHky5QsyBvlvn7XdLuP3vX8UTkE2YnG8aLkljesuM9rKPMXNehhoD9EmkAlCq8I6tmrz YOIda1GTo70dI6EgGH7eK1OAr6lQFw4FNsqv3SbuwYffS5fL8LuE0kF4YFqmcwjmfyXo1GXKDdwgD umeXWNWw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sRAxs-00000007N5w-2RKy; Tue, 09 Jul 2024 13:33:04 +0000 Received: from perceval.ideasonboard.com ([213.167.242.64]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sRAuM-00000007LNq-3km4 for linux-arm-kernel@lists.infradead.org; Tue, 09 Jul 2024 13:29:31 +0000 Received: from mail.ideasonboard.com (cpc141996-chfd3-2-0-cust928.12-3.cable.virginm.net [86.13.91.161]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 0785023D1; Tue, 9 Jul 2024 15:28:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1720531733; bh=LliVrk7ThACXHcwvohk1Z1BKhstOHa4tVcwZi2N8Qwo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EGAnzhqv8RfVNrEo/rk++UsfKklBAEevezVWefGmBCA8DbqbhScHp+drmv3EPawki R5HRQDA647/oGBUPzZXF37zwLbanCo0d4kZ+YFvku+2Th4JV/jCyB9DGlU/3B96w0H scoPQCxFD9Ef9m8//Skdyffu7jzVpoR9DkeE+AzI= From: Daniel Scally To: linux-media@vger.kernel.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: jacopo.mondi@ideasonboard.com, nayden.kanchev@arm.com, robh+dt@kernel.org, mchehab@kernel.org, krzysztof.kozlowski+dt@linaro.org, conor+dt@kernel.org, jerome.forissier@linaro.org, kieran.bingham@ideasonboard.com, laurent.pinchart@ideasonboard.com, sakari.ailus@iki.fi, Daniel Scally Subject: [PATCH v6 14/18] Documentation: mali-c55: Add Statistics documentation Date: Tue, 9 Jul 2024 14:29:02 +0100 Message-Id: <20240709132906.3198927-15-dan.scally@ideasonboard.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240709132906.3198927-1-dan.scally@ideasonboard.com> References: <20240709132906.3198927-1-dan.scally@ideasonboard.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240709_062927_224390_11D026E1 X-CRM114-Status: GOOD ( 17.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add documentation explaining the ability to capture statistics from the mali-c55 driver's new V4L2 device, as well as the various tap points from which those statistics can be drawn in the ISP's processing flow. Additionally add a page detailing the new V4L2 meta format for the mali-c55 statistics. Reviewed-by: Laurent Pinchart Acked-by: Nayden Kanchev Co-developed-by: Jacopo Mondi Signed-off-by: Jacopo Mondi Signed-off-by: Daniel Scally --- Changes in v6: - None Changes in v5: - New patch Documentation/admin-guide/media/mali-c55.rst | 60 ++++++++++++++++++- .../userspace-api/media/v4l/meta-formats.rst | 1 + .../media/v4l/metafmt-arm-mali-c55.rst | 29 +++++++++ MAINTAINERS | 1 + 4 files changed, 90 insertions(+), 1 deletion(-) create mode 100644 Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst index 72cdded507b3..7eaeac63ddf7 100644 --- a/Documentation/admin-guide/media/mali-c55.rst +++ b/Documentation/admin-guide/media/mali-c55.rst @@ -67,10 +67,11 @@ The driver has 4 V4L2 subdevices: - `mali_c55 resizer fr`: The Full-Resolution pipe resizer - `mali_c55 resizer ds`: The Downscale pipe resizer -The driver has 2 V4L2 video devices: +The driver has 3 V4L2 video devices: - `mali-c55 fr`: The full-resolution pipe's capture device - `mali-c55 ds`: The downscale pipe's capture device +- `mali-c55 3a stats`: The 3A statistics capture device Frame sequences are synchronised across to two capture devices, meaning if one pipe is started later than the other the sequence numbers returned in its @@ -333,6 +334,63 @@ configured, followed by formats in the appropriate places: # Set format on the video device and stream yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 +.. _mali-c55-3a-stats: + +Capturing ISP Statistics +======================== + +The ISP is capable of producing statistics for consumption by image processing +algorithms running in userspace. These statistics can be captured by queueing +buffers to the `mali-c55 3a stats` V4L2 Device whilst the ISP is streaming. Only +the :ref:`V4L2_META_FMT_MALI_C55_STATS ` +format is supported, so no format-setting need be done: + +.. code-block:: none + + # We assume the media graph has been configured to support RGB565 capture + # from the mali-c55 fr V4L2 Device, which is at /dev/video0. The statistics + # V4L2 device is at /dev/video3 + + yavta -f RGB565 -s 1920x1080 -c32 /dev/video0 && \ + yavta -c10 -F /dev/video3 + +The layout of the buffer is described by :c:type:`mali_c55_stats_buffer`, +but broadly statistics are generated to support three image processing +algorithms; AEXP (Auto-Exposure), AWB (Auto-White Balance) and AF (Auto-Focus). +These stats can be drawn from various places in the Mali C55 ISP pipeline, known +as "tap points". This high-level block diagram is intended to explain where in +the processing flow the statistics can be drawn from:: + + +--> AEXP-2 +----> AEXP-1 +--> AF-0 + | +----> AF-1 | + | | | + +---------+ | +--------------+ | +--------------+ | + | Input +-+-->+ Digital Gain +---+-->+ Black Level +---+---+ + +---------+ +--------------+ +--------------+ | + +-----------------------------------------------------------------+ + | + | +--------------+ +---------+ +----------------+ + +-->| Sinter Noise +-+ White +--+--->| Lens Shading +--+---------------+ + | Reduction | | Balance | | | | | | + +--------------+ +---------+ | +----------------+ | | + +---> AEXP-0 (A) +--> AEXP-0 (B) | + +--------------------------------------------------------------------------+ + | + | +----------------+ +--------------+ +----------------+ + +-->| Tone mapping +-+--->| Demosaicing +->+ Purple Fringe +-+-----------+ + | | | +--------------+ | Correction | | | + +----------------+ +-> AEXP-IRIDIX +----------------+ +---> AWB-0 | + +----------------------------------------------------------------------------+ + | +-------------+ +-------------+ + +------------------->| Colour +---+--->| Output | + | Correction | | | Pipelines | + +-------------+ | +-------------+ + +--> AWB-1 + +At present all statistics are drawn from the 0th tap point for each algorithm; +I.E. AEXP statistics from AEXP-0 (A), AWB statistics from AWB-0 and AF +statistics from AF-0. In the future this will be configurable. + References ========== .. [1] https://git.linuxtv.org/v4l-utils.git/ diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst index c23aac823d2c..319ca7b91f6b 100644 --- a/Documentation/userspace-api/media/v4l/meta-formats.rst +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst @@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata` interface only. .. toctree:: :maxdepth: 1 + metafmt-arm-mali-c55 metafmt-d4xx metafmt-generic metafmt-intel-ipu3 diff --git a/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst new file mode 100644 index 000000000000..186e0deb9ece --- /dev/null +++ b/Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst @@ -0,0 +1,29 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. _v4l2-meta-fmt-mali-c55-3a-stats: + +************************************* +V4L2_META_FMT_MALI_C55_STATS ('C55S') +************************************* + +3A Statistics +============= + +The ISP device collects different statistics over an input bayer frame. Those +statistics can be obtained by userspace from the +:ref:`mali-c55 3a stats ` metadata capture video node, using +the :c:type:`v4l2_meta_format` interface. The buffer contains a single instance +of the C structure :c:type:`mali_c55_stats_buffer` defined in +``mali-c55-config.h``, so the structure can be obtained from the buffer by: + +.. code-block:: C + + struct mali_c55_stats_buffer *stats = + (struct mali_c55_stats_buffer *)buf; + +For details of the statistics see :c:type:`mali_c55_stats_buffer`. + +Arm Mali-C55 uAPI data types +============================ + +.. kernel-doc:: include/uapi/linux/media/arm/mali-c55-config.h diff --git a/MAINTAINERS b/MAINTAINERS index 3b5c39b820cc..c8b20a1a1c1c 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1764,6 +1764,7 @@ T: git git://linuxtv.org/media_tree.git F: Documentation/admin-guide/media/mali-c55-graph.dot F: Documentation/admin-guide/media/mali-c55.rst F: Documentation/devicetree/bindings/media/arm,mali-c55.yaml +F: Documentation/userspace-api/media/v4l/metafmt-arm-mali-c55.rst F: drivers/media/platform/arm/mali-c55/ F: include/uapi/linux/media/arm/mali-c55-config.h