From patchwork Tue Dec 7 19:02:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: David Gow X-Patchwork-Id: 12662443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E36BC433EF for ; Tue, 7 Dec 2021 19:02:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240828AbhLGTGZ (ORCPT ); Tue, 7 Dec 2021 14:06:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236140AbhLGTGY (ORCPT ); Tue, 7 Dec 2021 14:06:24 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 29B0AC061574 for ; Tue, 7 Dec 2021 11:02:54 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id m15-20020a170902bb8f00b0014382b67873so2881005pls.19 for ; Tue, 07 Dec 2021 11:02:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc :content-transfer-encoding; bh=8q51zUwHuJfhXkm4t3PDBh8T8/WWWNZrP/ln+mfXd9E=; b=nbm20WZ+jVXjHrFDAZT9Mtyr2NdB1tcau+0H4kMeB4KlEFiJbD4alL28ZPdMNlgkJj FFTYwHQObwr4VdA4539Cagxu4xysq98EiP7KK22SWOjG0NUYCHylNtMzw5XA3OQnrKlT g5s0wu1wkUKqRhBzA0PFEVbpuXuihMaraIZ5Bsb/NQvO1quDnJxtWZjspjevglDo29kq 876I0QUm/ZO2jcaivw+koc7Ozm3a6jKyKsrGdTKs4+wFE7VMADdwxLxYHhBMs8caOzdK mr1rUBNZiuMdmRsBuijgzgYuCWmeYcC5Rvw2IigqhJGrr4ZVJvSNuIS0cCeRA++gTlD3 58BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc :content-transfer-encoding; bh=8q51zUwHuJfhXkm4t3PDBh8T8/WWWNZrP/ln+mfXd9E=; b=lgf2WcMY9Rm7KIh0h/8OK06oxujkMplrbXvn/DgcByDB1iz9FR7qwJhcPNxK/ON6Sz a13pxlTfBxGRvZ7B8Y48KYXMMoZIbv+ykWawQ6Jq9hbfzEHf4tDm/pcAVlYq5HukhRmI NSc+OydDVfIC908LmVzuCiUms5zm4ByrV3ECPwj5ATqV7JF89QU/q+bcC/m2XOh1+vMa R9pmHL2QKAQXKZxOtQ+bALtku/EBsIRIdUoTu9cXS156qreqsqX7J5v2ZIqdCCUCEeQR I0rgLUlZT9glqUp34sCy8noMcEW/UlX+W1jM0okH/4rtKwd62RwZ5Fglg3bKu+GaBSpK q9ZQ== X-Gm-Message-State: AOAM532TjMP9qhHOQOgPwGwmhvWypBg1esd0tOkIoGiHBPS7Abhxlf2b o358suGv9c9+YS2HG+IGVmWWjRfseKlMqw== X-Google-Smtp-Source: ABdhPJwxiMjj0eRUrmptJlP62mAjB2juBg2U8ySx/Cfl7N7nEZkB/AoFVpzNeaC8pQJ8VjC8cBwtrI+DB1NNIQ== X-Received: from spirogrip.svl.corp.google.com ([2620:15c:2cb:201:36b6:7f4b:d39a:e417]) (user=davidgow job=sendgmr) by 2002:a17:90b:4b51:: with SMTP id mi17mr1298440pjb.48.1638903773471; Tue, 07 Dec 2021 11:02:53 -0800 (PST) Date: Tue, 7 Dec 2021 11:02:51 -0800 Message-Id: <20211207190251.18426-1-davidgow@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.34.1.400.ga245620fadb-goog Subject: [RFC PATCH v2] Documentation: dev-tools: Add KTAP specification From: David Gow To: Brendan Higgins , Tim.Bird@sony.com, shuah@kernel.org, Jonathan Corbet , Kees Cook Cc: rmr167@gmail.com, guillaume.tucker@collabora.com, dlatypov@google.com, kernelci@groups.io, kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, David Gow Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org From: Rae Moar It does not make any significant additions or changes other than those already in use in the kernel: additional features can be added as they become necessary and used. [1]: https://testanything.org/tap-version-13-specification.html Signed-off-by: Rae Moar Co-developed-by: David Gow Signed-off-by: David Gow Reviewed-by: Kees Cook Reviewed-by: Shuah Khan --- Changes since RFC v1: https://lore.kernel.org/linux-kselftest/20211203064840.2871751-1-davidgow@google.com/ - Add a "see also" section with some useful links. - Remove the XPASS directive, which isn't used anywhere. - Clear up / reorganise the discussion around differences between KTAP and TAP14. - Improve the wording around some directives. - Fix a bunch of typos. See prior discussion in the following RFC: https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqaJk+r-K1YJzPggFDQ@mail.gmail.com/. --- Documentation/dev-tools/index.rst | 1 + Documentation/dev-tools/ktap.rst | 298 ++++++++++++++++++++++++++++++ 2 files changed, 299 insertions(+) create mode 100644 Documentation/dev-tools/ktap.rst diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst index 010a2af1e7d9..4621eac290f4 100644 --- a/Documentation/dev-tools/index.rst +++ b/Documentation/dev-tools/index.rst @@ -32,6 +32,7 @@ Documentation/dev-tools/testing-overview.rst kgdb kselftest kunit/index + ktap .. only:: subproject and html diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst new file mode 100644 index 000000000000..878530cb9c27 --- /dev/null +++ b/Documentation/dev-tools/ktap.rst @@ -0,0 +1,298 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======================================== +The Kernel Test Anything Protocol (KTAP) +======================================== + +TAP, or the Test Anything Protocol is a format for specifying test results used +by a number of projects. It's website and specification are found at this `link +`_. The Linux Kernel largely uses TAP output for test +results. However, Kernel testing frameworks have special needs for test results +which don't align with the original TAP specification. Thus, a "Kernel TAP" +(KTAP) format is specified to extend and alter TAP to support these use-cases. +This specification describes the generally accepted format of KTAP as it is +currently used in the kernel. + +KTAP test results describe a series of tests (which may be nested: i.e., test +can have subtests), each of which can contain both diagnostic data -- e.g., log +lines -- and a final result. The test structure and results are +machine-readable, whereas the diagnostic data is unstructured and is there to +aid human debugging. + +KTAP output is built from four different types of lines: +- Version lines +- Plan lines +- Test case result lines +- Diagnostic lines + +In general, valid KTAP output should also form valid TAP output, but some +information, in particular nested test results, may be lost. Also note that +there is a stagnant draft specification for TAP14, KTAP diverges from this in +a couple of places (notably the "Subtest" header), which are described where +relevant later in this document. + +Version lines +------------- + +All KTAP-formatted results begin with a "version line" which specifies which +version of the (K)TAP standard the result is compliant with. + +For example: +- "KTAP version 1" +- "TAP version 13" +- "TAP version 14" + +Note that, in KTAP, subtests also begin with a version line, which denotes the +start of the nested test results. This differs from TAP14, which uses a +separate "Subtest" line. + +While, going forward, "KTAP version 1" should be used by compliant tests, it +is expected that most parsers and other tooling will accept the other versions +listed here for compatibility with existing tests and frameworks. + +Plan lines +---------- + +A test plan provides the number of tests (or subtests) in the KTAP output. + +Plan lines must follow the format of "1..N" where N is the number of tests or subtests. +Plan lines follow version lines to indicate the number of nested tests. + +While there are cases where the number of tests is not known in advance -- in +which case the test plan may be omitted -- it is strongly recommended one is +present where possible. + +Test case result lines +---------------------- + +Test case result lines indicate the final status of a test. +They are required and must have the format: + +.. code-block:: + + [][ # [] []] + +The result can be either "ok", which indicates the test case passed, +or "not ok", which indicates that the test case failed. + + represents the number of the test being performed. The first test must +have the number 1 and the number then must increase by 1 for each additional +subtest within the same test at the same nesting level. + +The description is a description of the test, generally the name of +the test, and can be any string of words (can't include #). The +description is optional, but recommended. + +The directive and any diagnostic data is optional. If either are present, they +must follow a hash sign, "#". + +A directive is a keyword that indicates a different outcome for a test other +than passed and failed. The directive is optional, and consists of a single +keyword preceding the diagnostic data. In the event that a parser encounters +a directive it doesn't support, it should fall back to the "ok" / "not ok" +result. + +Currently accepted directives are: + +- "SKIP", which indicates a test was skipped (note the result of the test case + result line can be either "ok" or "not ok" if the SKIP directive is used) +- "TODO", which indicates that a test is not expected to pass at the moment, + e.g. because the feature it is testing is known to be broken. While this + directive is inherited from TAP, its use in the kernel is discouraged. +- "XFAIL", which indicates that a test is expected to fail. This is similar + to "TODO", above, and is used by some kselftest tests. +- “TIMEOUT”, which indicates a test has timed out (note the result of the test + case result line should be “not ok” if the TIMEOUT directive is used) +- “ERROR”, which indicates that the execution of a test has failed due to a + specific error that is included in the diagnostic data. (note the result of + the test case result line should be “not ok” if the ERROR directive is used) + +The diagnostic data is a plain-text field which contains any additional details +about why this result was produced. This is typically an error message for ERROR +or failed tests, or a description of missing dependencies for a SKIP result. + +The diagnostic data field is optional, and results which have neither a +directive nor any diagnostic data do not need to include the "#" field +separator. + +Example result lines include: + +.. code-block:: + + ok 1 test_case_name + +The test "test_case_name" passed. + +.. code-block:: + + not ok 1 test_case_name + +The test "test_case_name" failed. + +.. code-block:: + + ok 1 test # SKIP necessary dependency unavailable + +The test "test" was SKIPPED with the diagnostic message "necessary dependency +unavailable". + +.. code-block:: + + not ok 1 test # TIMEOUT 30 seconds + +The test "test" timed out, with diagnostic data "30 seconds". + +.. code-block:: + + ok 5 check return code # rcode=0 + +The test "check return code" passed, with additional diagnostic data “rcode=0” + + +Diagnostic lines +---------------- + +If tests wish to output any further information, they should do so using +"diagnostic lines". Diagnostic lines are optional, freeform text, and are +often used to describe what is being tested and any intermediate results in +more detail than the final result and diagnostic data line provides. + +Diagnostic lines are formatted as "# ", where the +description can be any string. Diagnostic lines can be anywhere in the test +output. As a rule, diagnostic lines regarding a test are directly before the +test result line for that test. + +Note that most tools will treat unknown lines (see below) as diagnostic lines, +even if they do not start with a "#": this is to capture any other useful +kernel output which may help debug the test. It is nevertheless recommended +that tests always prefix any diagnostic output they have with a "#" character. + +Unknown lines +------------- + +There may be lines within KTAP output that do not follow the format of one of +the four formats for lines described above. This is allowed, however, they will +not influence the status of the tests. + +Nested tests +------------ + +In KTAP, tests can be nested. This is done by having a test include within its +output an entire set of KTAP-formatted results. This can be used to categorize +and group related tests, or to split out different results from the same test. + +The "parent" test's result should consist of all of its subtests' results, +starting with another KTAP version line and test plan, and end with the overall +result. If one of the subtests fail, for example, the parent test should also +fail. + +Additionally, all result lines in a subtest should be indented. One level of +indentation is two spaces: " ". The indentation should begin at the version +line and should end before the parent test's result line. + +An example of a test with two nested subtests: + +.. code-block:: + + KTAP version 1 + 1..1 + KTAP version 1 + 1..2 + ok 1 test_1 + not ok 2 test_2 + # example failed + not ok 1 example + +An example format with multiple levels of nested testing: + +.. code-block:: + + KTAP version 1 + 1..2 + KTAP version 1 + 1..2 + KTAP version 1 + 1..2 + not ok 1 test_1 + ok 2 test_2 + not ok 1 test_3 + ok 2 test_4 # SKIP + not ok 1 example_test_1 + ok 2 example_test_2 + + +Major differences between TAP and KTAP +-------------------------------------- + +Note the major differences between the TAP and KTAP specification: +- yaml and json are not recommended in diagnostic messages +- TODO directive not recognized +- KTAP allows for an arbitrary number of tests to be nested + +The TAP14 specification does permit nested tests, but instead of using another +nested version line, uses a line of the form +"Subtest: " where is the name of the parent test. + +Example KTAP output +-------------------- +.. code-block:: + + KTAP version 1 + 1..1 + KTAP version 1 + 1..3 + KTAP version 1 + 1..1 + # test_1: initializing test_1 + ok 1 test_1 + ok 1 example_test_1 + KTAP version 1 + 1..2 + ok 1 test_1 # SKIP test_1 skipped + ok 2 test_2 + ok 2 example_test_2 + KTAP version 1 + 1..3 + ok 1 test_1 + # test_2: FAIL + not ok 2 test_2 + ok 3 test_3 # SKIP test_3 skipped + not ok 3 example_test_3 + not ok 1 main_test + +This output defines the following hierarchy: + +A single test called "main_test", which fails, and has three subtests: +- "example_test_1", which passes, and has one subtest: + + - "test_1", which passes, and outputs the diagnostic message "test_1: initializing test_1" + +- "example_test_2", which passes, and has two subtests: + + - "test_1", which is skipped, with the explanation "test_1 skipped" + - "test_2", which passes + +- "example_test_3", which fails, and has three subtests + + - "test_1", which passes + - "test_2", which outputs the diagnostic line "test_2: FAIL", and fails. + - "test_3", which is skipped with the explanation "test_3 skipped" + +Note that the individual subtests with the same names do not conflict, as they +are found in different parent tests. This output also exhibits some sensible +rules for "bubbling up" test results: a test fails if any of its subtests fail. +Skipped tests do not affect the result of the parent test (though it often +makes sense for a test to be marked skipped if _all_ of its subtests have been +skipped). + +See also: +--------- + +- The TAP specification: + https://testanything.org/tap-version-13-specification.html +- The (stagnant) TAP version 14 specification: + https://github.com/TestAnything/Specification/blob/tap-14-specification/specification.md +- The kselftest documentation: + Documentation/dev-tools/kselftest.rst +- The KUnit documentation: + Documentation/dev-tools/kunit/index.rst