From patchwork Mon Oct 22 14:48:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tomasz Figa X-Patchwork-Id: 10652195 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5763917DE for ; Mon, 22 Oct 2018 14:49:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5D01328C96 for ; Mon, 22 Oct 2018 14:49:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5105728C99; Mon, 22 Oct 2018 14:49:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B2BED28C96 for ; Mon, 22 Oct 2018 14:49:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728411AbeJVXIF (ORCPT ); Mon, 22 Oct 2018 19:08:05 -0400 Received: from mail-pl1-f195.google.com ([209.85.214.195]:36862 "EHLO mail-pl1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728271AbeJVXIE (ORCPT ); Mon, 22 Oct 2018 19:08:04 -0400 Received: by mail-pl1-f195.google.com with SMTP id y11-v6so19185703plt.3 for ; Mon, 22 Oct 2018 07:49:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=+1KZXHHhqgavw6jNVFM3//RcnjVG71f4baEma6XtfFs=; b=hZybMC2LMgJucbEzD0FSUvnym7PMzsVnvjzB+1Q9B+lafUFxNV2t09ZdG0g8MqN5LD 4vbPBY60EQa80BLnwD6cjqMSmX6SVfnHe6Kqb4xKutFDiCRMi2LS6AqUpkdrdqxOjCaL B90OQoNFLvMJOLtmi3ClN/uN1kV8GmiM+37is= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=+1KZXHHhqgavw6jNVFM3//RcnjVG71f4baEma6XtfFs=; b=RF/SS8u3dKEcNNk/yoAaz8ey4nd11VOcSvTM+xeYA+E+/F50pGa/Flytpdc3dkrroX HqiPd+PYtyWbkBP8UZ1WbG8m4To5oiMc+vnJ7XFiAaq+vv+od3j+GCUBvg2DdRWDUJsH GLpIyVWEJlPEfjEdA4B2sXFjA1VwVPCD2lRZhMJutJf7jnOHuwkwrQVdBWWSP8xa+dAv nU2fs38k0EppIa2zHsI50EAJcv2EIbM7zeHHDED0xZ12Br8eHa+t/dGpPHVKNyKB7PXp gTvoqYGwS4ODWxsnzVMOE/t9MGHGVC6pRzRf2xKqzR/eyGcMVhCJoxY0xHDYbHFLkA7E /mJA== X-Gm-Message-State: ABuFfogGZsxl/fRCph2hMXQazK1Il2bOKeIyUNJ+xI5jcM78LSYNzMwT u2194lI/GK6Bt54V8oZIkgzYcKSEcQw= X-Google-Smtp-Source: ACcGV632jqkTDMZt4EF5PYWHIc/821qp3JU8sLVc3VLvwZxVR/bWl7bco929OsctcHJPuH2VFO8npg== X-Received: by 2002:a17:902:1c3:: with SMTP id b61-v6mr32102828plb.65.1540219752537; Mon, 22 Oct 2018 07:49:12 -0700 (PDT) Received: from tfiga.tok.corp.google.com ([2401:fa00:4:4:5b21:5966:1198:d1e9]) by smtp.gmail.com with ESMTPSA id s80-v6sm42245429pfa.114.2018.10.22.07.49.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Oct 2018 07:49:11 -0700 (PDT) From: Tomasz Figa To: linux-media@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Mauro Carvalho Chehab , Hans Verkuil , =?utf-8?b?UGF3ZcWCIE/Fm2NpYWs=?= , Alexandre Courbot , Kamil Debski , Andrzej Hajda , Kyungmin Park , Jeongtae Park , Philipp Zabel , Tiffany Lin , Andrew-CT Chen , Stanimir Varbanov , Todor Tomov , Nicolas Dufresne , Paul Kocialkowski , Laurent Pinchart , dave.stevenson@raspberrypi.org, Ezequiel Garcia , Maxime Jourdan , Tomasz Figa Subject: [PATCH v2 0/2] Document memory-to-memory video codec interfaces Date: Mon, 22 Oct 2018 23:48:58 +0900 Message-Id: <20181022144901.113852-1-tfiga@chromium.org> X-Mailer: git-send-email 2.19.1.568.g152ad8e336-goog MIME-Version: 1.0 Sender: linux-media-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP It's been a while, but here is the v2 of the stateful mem2mem codec interfaces documentation. Sorry for taking so long time to respin. This series attempts to add the documentation of what was discussed during Media Workshops at LinuxCon Europe 2012 in Barcelona and then later Embedded Linux Conference Europe 2014 in Düsseldorf and then eventually written down by Pawel Osciak and tweaked a bit by Chrome OS video team (but mostly in a cosmetic way or making the document more precise), during the several years of Chrome OS using the APIs in production. Note that most, if not all, of the API is already implemented in existing mainline drivers, such as s5p-mfc or mtk-vcodec. Intention of this series is just to formalize what we already have. Thanks everyone for the huge amount of useful comments for the RFC and v1. Much of the credits should go to Pawel Osciak too, for writing most of the original text of the initial RFC. Changes since v1: (https://lore.kernel.org/patchwork/project/lkml/list/?series=360520) Decoder: - Removed a note about querying all combinations of OUTPUT and CAPTURE frame sizes, since it would conflict with scaling/composiion support to be added later. - Removed the source change event after setting non-zero width and height on OUTPUT queue, since the change happens as a direct result of a client action. - Moved all the setup steps for CAPTURE queue out of Initialization and Dynamic resolution change into a common sequence called Capture setup, since they were mostly duplicate of each other. - Described steps to allocate buffers for higher resolution than the stream to prepare for future resolution changes. - Described a way to skip the initial header parsing and speculatively configure the CAPTURE queue (for gstreamer/ffmpeg compatibility). - Reordered CAPTURE setup steps so that all the driver queries are done first and only then a reconfiguration may be attempted or skipped. - Described VIDIOC_CREATE_BUFS as another way of allocating buffers. - Made the decoder signal the source change event as soon as the change is detected, to reduce pipeline stalls in case of buffers already good to continue decoding. - Stressed out the fact that a source change may happen even without a change in the coded resolution. - Described querying pixel aspect ratio using VIDIOC_CROPCAP. - Extended documentation of VIDIOC_DECODER_CMD and VIDIOC_G/S/TRY_FMT to more precisely describe the behavior of mem2mem decoders. - Clarified that 0 width and height are allowed for OUTPUT side of mem2mem decoders in the documentation of the v4l2_pix_fmt struct. Encoder: - Removed width and height from CAPTURE (coded) format, since the coded resolution of the stream is an internal detail of the encoded stream. - Made the VIDIOC_S_FMT on OUTPUT mandatory, since the default format normally does not make sense (even if technically valid). - Changed the V4L2_SEL_TGT_CROP_BOUNDS and V4L2_SEL_TGT_CROP_DEFAULT selection targets to be equal to the full source frame to simplify internal handling in drivers for simple hardware. - Changed the V4L2_SEL_TGT_COMPOSE_DEFAULT selection target to be equal to |crop width|x|crop height|@(0,0) to simplify internal handling in drivers for simple hardware. - Removed V4L2_SEL_TGT_COMPOSE_PADDED, since the encoder does not write to the raw buffers. - Extended documentation of VIDIOC_ENCODER_CMD to more precisely describe the behavior of mem2mem encoders. - Clarified that 0 width and height are allowed for CAPTURE side of mem2mem encoders in the documentation of the v4l2_pix_fmt struct. General: - Clarified that the Drain sequence valid only if both queues are streaming and stopping any of the queues would abort it, since there is nothing to drain, if OUTPUT is stopped and there is no way to signal the completion if CAPTURE is stopped. - Clarified that VIDIOC_STREAMON on any of the queues would resume the codec from stopped state, to be consistent with the documentation of VIDIOC_ENCODER/DECODER_CMD. - Documented the relation between timestamps of OUTPUT and CAPTURE buffers and how special cases of non-1:1 relation are handled. - Added missing sizeimage to bitstream format operations and removed the mistaken mentions from descriptions of respective REQBUFS calls. - Removed the Pause sections, since there is no notion of pause for mem2mem devices. - Added state machine diagrams. - Merged both glossaries into one in the decoder document and a reference to it in the encoder document. - Added missing terms to the glossary. - Added "Stateful" to the interface names. - Reworded the text to be more userspace-centric. - A number of other readability improvements suggested in review comments. For changes since RFC see the v1: https://lore.kernel.org/patchwork/project/lkml/list/?series=360520 Tomasz Figa (2): media: docs-rst: Document memory-to-memory video decoder interface media: docs-rst: Document memory-to-memory video encoder interface Documentation/media/uapi/v4l/dev-decoder.rst | 1082 +++++++++++++++++ Documentation/media/uapi/v4l/dev-encoder.rst | 579 +++++++++ Documentation/media/uapi/v4l/devices.rst | 2 + Documentation/media/uapi/v4l/pixfmt-v4l2.rst | 10 + Documentation/media/uapi/v4l/v4l2.rst | 12 +- .../media/uapi/v4l/vidioc-decoder-cmd.rst | 40 +- .../media/uapi/v4l/vidioc-encoder-cmd.rst | 38 +- Documentation/media/uapi/v4l/vidioc-g-fmt.rst | 14 + 8 files changed, 1747 insertions(+), 30 deletions(-) create mode 100644 Documentation/media/uapi/v4l/dev-decoder.rst create mode 100644 Documentation/media/uapi/v4l/dev-encoder.rst Signed-off-by: Hans Verkuil