From patchwork Thu Aug 5 23:51:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Latypov X-Patchwork-Id: 12422339 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9ABA7C4338F for ; Thu, 5 Aug 2021 23:51:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 79E8461108 for ; Thu, 5 Aug 2021 23:51:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242522AbhHEXwH (ORCPT ); Thu, 5 Aug 2021 19:52:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233076AbhHEXwH (ORCPT ); Thu, 5 Aug 2021 19:52:07 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C2FBC0613D5 for ; Thu, 5 Aug 2021 16:51:52 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id o3-20020a2541030000b0290557cf3415f8so7898484yba.1 for ; Thu, 05 Aug 2021 16:51:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=g4VlaZaJqsE6T03DiXthF+BbWIcuoMN/330Q3/YZIio=; b=KPZrnmtt/GX4apWQRm+0lWsBhov6eO/lA7Q1FFgtlVtXEtHmJW77OxmjJS2VbjPxaL +oebQ+lpb9cVOtK6TQhRDZfwdrHVlOj2Vl0V2JAtMJXIFFfptVmGZN2h/ux2Kia0uCCZ UGco7DkftL786h2DKl9BfYtap44yH3TQKZJ/V3mZTzERPkg6w8JrvQmE21PfO40AKOLM KAZGs5JR3fKydzV4swwpzaW1E58OCFlC0lv2F2C66bP3g/RKOWTz+aRos5XQHMAwBAFl 3JCufPmY+MVK2UydOwiLW1ojRgvKHAyy4y7m0zcqz28sb+OaF4rSPGZdqo54YZVoWbJe k5Aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=g4VlaZaJqsE6T03DiXthF+BbWIcuoMN/330Q3/YZIio=; b=dkrPLkFTdPb1o0MtBxlISIT/ggCAUVPvFbCWS5lezhACLyZq0gQPc9Ts+qZ0mY4SBb Gyoe/YWQG/AU9sapEO8kbv4LZIszeqxqSESjq9K50X0yNbQ3Gle7u1by3DpOAiragweE cd5lRvlNcHVqNjwS4/5dJUmQMJfqig+YtHrEw1wDETwfqNcfq4kRBwQ2yN5xGdCMB5DG 43jhPggA6kEPXH0pAHSmNsx/DY49/543RNtxdWCTzDGx0cCqWPnz9rE16+wKJ/0q4aQL LNpnYlpK/olFdX3EMU7BloM35IoCqjrHS6DwGoiEZRqk8yOl4xm9snfyT7rBgl+B1aj+ LILw== X-Gm-Message-State: AOAM531aedUn0FZyYLXCmZVrfOfWojehGdhcKc815CnU6+jku3VSqnxN 5bShR5RyXYCxq4blCG50j0Be9vU3iR3ptA== X-Google-Smtp-Source: ABdhPJzsGmCDww09UKe4MBlzb8FwZh++khO8jLgyABH8pw2Fo0mG/OpYavE6lkB63cEvlNflqN6ccLeR27IGXg== X-Received: from dlatypov.svl.corp.google.com ([2620:15c:2cd:202:ba4b:612a:402a:3fbd]) (user=dlatypov job=sendgmr) by 2002:a25:abcd:: with SMTP id v71mr8942440ybi.499.1628207511613; Thu, 05 Aug 2021 16:51:51 -0700 (PDT) Date: Thu, 5 Aug 2021 16:51:44 -0700 Message-Id: <20210805235145.2528054-1-dlatypov@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.32.0.605.g8dce9f2422-goog Subject: [PATCH v2 1/2] kunit: tool: make --raw_output support only showing kunit output From: Daniel Latypov To: brendanhiggins@google.com, davidgow@google.com Cc: linux-kernel@vger.kernel.org, kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, skhan@linuxfoundation.org, Daniel Latypov Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org --raw_output is nice, but it would be nicer if could show only output after KUnit tests have started. So change the flag to allow specifying a string ('kunit'). Make it so `--raw_output` alone will default to `--raw_output=all` and have the same original behavior. Drop the small kunit_parser.raw_output() function since it feels wrong to put it in "kunit_parser.py" when the point of it is to not parse anything. E.g. $ ./tools/testing/kunit/kunit.py run --raw_output=kunit ... [15:24:07] Starting KUnit Kernel ... TAP version 14 1..1 # Subtest: example 1..3 # example_simple_test: initializing ok 1 - example_simple_test # example_skip_test: initializing # example_skip_test: You should not see a line below. ok 2 - example_skip_test # SKIP this test should be skipped # example_mark_skipped_test: initializing # example_mark_skipped_test: You should see a line below. # example_mark_skipped_test: You should see this line. ok 3 - example_mark_skipped_test # SKIP this test should be skipped ok 1 - example [15:24:10] Elapsed time: 6.487s total, 0.001s configuring, 3.510s building, 0.000s running Signed-off-by: Daniel Latypov Reviewed-by: David Gow --- Documentation/dev-tools/kunit/kunit-tool.rst | 9 ++++++--- tools/testing/kunit/kunit.py | 20 +++++++++++++++----- tools/testing/kunit/kunit_parser.py | 4 ---- tools/testing/kunit/kunit_tool_test.py | 9 +++++++++ 4 files changed, 30 insertions(+), 12 deletions(-) base-commit: f684616e08e9cd9db3cd53fe2e068dfe02481657 diff --git a/Documentation/dev-tools/kunit/kunit-tool.rst b/Documentation/dev-tools/kunit/kunit-tool.rst index c7ff9afe407a..ae52e0f489f9 100644 --- a/Documentation/dev-tools/kunit/kunit-tool.rst +++ b/Documentation/dev-tools/kunit/kunit-tool.rst @@ -114,9 +114,12 @@ results in TAP format, you can pass the ``--raw_output`` argument. ./tools/testing/kunit/kunit.py run --raw_output -.. note:: - The raw output from test runs may contain other, non-KUnit kernel log - lines. +The raw output from test runs may contain other, non-KUnit kernel log +lines. You can see just KUnit output with ``--raw_output=kunit``: + +.. code-block:: bash + + ./tools/testing/kunit/kunit.py run --raw_output=kunit If you have KUnit results in their raw TAP format, you can parse them and print the human-readable summary with the ``parse`` command for kunit_tool. This diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 7174377c2172..5a931456e718 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -16,6 +16,7 @@ assert sys.version_info >= (3, 7), "Python version is too old" from collections import namedtuple from enum import Enum, auto +from typing import Iterable import kunit_config import kunit_json @@ -114,7 +115,16 @@ def parse_tests(request: KunitParseRequest) -> KunitResult: 'Tests not Parsed.') if request.raw_output: - kunit_parser.raw_output(request.input_data) + output: Iterable[str] = request.input_data + if request.raw_output == 'all': + pass + elif request.raw_output == 'kunit': + output = kunit_parser.extract_tap_lines(output) + else: + print(f'Unknown --raw_output option "{request.raw_output}"', file=sys.stderr) + for line in output: + print(line.rstrip()) + else: test_result = kunit_parser.parse_run_tests(request.input_data) parse_end = time.time() @@ -135,7 +145,6 @@ def parse_tests(request: KunitParseRequest) -> KunitResult: return KunitResult(KunitStatus.SUCCESS, test_result, parse_end - parse_start) - def run_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitRequest) -> KunitResult: run_start = time.time() @@ -181,7 +190,7 @@ def add_common_opts(parser) -> None: parser.add_argument('--build_dir', help='As in the make command, it specifies the build ' 'directory.', - type=str, default='.kunit', metavar='build_dir') + type=str, default='.kunit', metavar='build_dir') parser.add_argument('--make_options', help='X=Y make option, can be repeated.', action='append') @@ -246,8 +255,9 @@ def add_exec_opts(parser) -> None: action='append') def add_parse_opts(parser) -> None: - parser.add_argument('--raw_output', help='don\'t format output from kernel', - action='store_true') + parser.add_argument('--raw_output', help='If set don\'t format output from kernel. ' + 'If set to --raw_output=kunit, filters to just KUnit output.', + type=str, nargs='?', const='all', default=None) parser.add_argument('--json', nargs='?', help='Stores test results in a JSON, and either ' diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index b88db3f51dc5..84938fefbac0 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -106,10 +106,6 @@ def extract_tap_lines(kernel_output: Iterable[str]) -> LineStream: yield line_num, line[prefix_len:] return LineStream(lines=isolate_kunit_output(kernel_output)) -def raw_output(kernel_output) -> None: - for line in kernel_output: - print(line.rstrip()) - DIVIDER = '=' * 60 RESET = '\033[0;0m' diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 628ab00f74bc..619c4554cbff 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -399,6 +399,15 @@ class KUnitMainTest(unittest.TestCase): self.assertNotEqual(call, mock.call(StrContains('Testing complete.'))) self.assertNotEqual(call, mock.call(StrContains(' 0 tests run'))) + def test_run_raw_output_kunit(self): + self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) + kunit.main(['run', '--raw_output=kunit'], self.linux_source_mock) + self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) + self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) + for call in self.print_mock.call_args_list: + self.assertNotEqual(call, mock.call(StrContains('Testing complete.'))) + self.assertNotEqual(call, mock.call(StrContains(' 0 tests run'))) + def test_exec_timeout(self): timeout = 3453 kunit.main(['exec', '--timeout', str(timeout)], self.linux_source_mock) From patchwork Thu Aug 5 23:51:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Latypov X-Patchwork-Id: 12422341 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0989C4338F for ; Thu, 5 Aug 2021 23:52:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7F5A761132 for ; Thu, 5 Aug 2021 23:52:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242607AbhHEXwO (ORCPT ); Thu, 5 Aug 2021 19:52:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57912 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242542AbhHEXwK (ORCPT ); Thu, 5 Aug 2021 19:52:10 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C9FD5C061798 for ; Thu, 5 Aug 2021 16:51:55 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id p71-20020a25424a0000b029056092741626so7708654yba.19 for ; Thu, 05 Aug 2021 16:51:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=QxYsxzaJfWoxD1mihF0RJctGuwELl/VrTwhSBSYHHsw=; b=GTH2Oy1G7yZNM3Wl8w4mfMPVIthiiY+Io8Nm5Z3IonVKTZoMJLhJpe95an6UFSi85B UN03qTtpIP53sA9y/qKdEiyt30bv7kCK4o6aDt0AJzHub5j8uCCUMUZL0vlQTXlEBw23 sXryUyHbCDE4gdKoVU8DU7DQcyYzAoFZIZ33kEdeb7CgE5S5LZzAnuFxguayUhbl3x++ oNNUBNun7P3Yz41J8BAR1+lWcBkq03de+D377zQ6+kgnkwbvLbjGnZFQOWAwg07VNjMR RNJLuTdpXgqoF4MrshdinR0ienKr2krD2RXOfj1SzuPMwdsb5nY8NPNcx1FmnPR9eb1R h2UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=QxYsxzaJfWoxD1mihF0RJctGuwELl/VrTwhSBSYHHsw=; b=Ai7vEG0gmg7Ucd+x3iwcVRnWBeVQXBsBW7lV8blEG6b5qQpODxlPnQ6u2hD6x4wBnN or8fyF8e3lxGXx0u+zve3B69rQR5oQhyRQhgTfLHsOWwsbepJ9iD67TmNMhT2gE8wbnP KLo1nC8PHCFJx7dHLcGBxYuJJk/VCB03dOlVYAhAGeEVUWjvZ3B9khxwpI+ip/KfxD9U K0KBtng6wlLkXCuzNA/HCIi56cDsaoLzh98oMZv4EEeLBvGomLOT5RMHENMEyAADu7jx 3RIoswOsEptKWsCvhm+/ffl3FB3mRf0aSsXXRVB1baEUMMhHY8YtamWTWEmUqiT2HU41 tSJw== X-Gm-Message-State: AOAM533BPJkCvdZth20vNKE2MfxIy/7kVrFJBw5MOiB8eUBfv8o0yAZk hDjt/6RFIz+j/4BwDLYE1UMlSCNtvABmnQ== X-Google-Smtp-Source: ABdhPJzktPLs6OC1HVI1NYznIviv2nE/tjfSMl/sfBdC2aDxMcJ5SJqRhGkhIHyuQ1sNanaM/Sp5O4vVrN+kXg== X-Received: from dlatypov.svl.corp.google.com ([2620:15c:2cd:202:ba4b:612a:402a:3fbd]) (user=dlatypov job=sendgmr) by 2002:a25:e70c:: with SMTP id e12mr8890281ybh.333.1628207515050; Thu, 05 Aug 2021 16:51:55 -0700 (PDT) Date: Thu, 5 Aug 2021 16:51:45 -0700 In-Reply-To: <20210805235145.2528054-1-dlatypov@google.com> Message-Id: <20210805235145.2528054-2-dlatypov@google.com> Mime-Version: 1.0 References: <20210805235145.2528054-1-dlatypov@google.com> X-Mailer: git-send-email 2.32.0.605.g8dce9f2422-goog Subject: [PATCH v2 2/2] kunit: add 'kunit.action' param to allow listing out tests From: Daniel Latypov To: brendanhiggins@google.com, davidgow@google.com Cc: linux-kernel@vger.kernel.org, kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, skhan@linuxfoundation.org, Daniel Latypov Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Context: It's difficult to map a given .kunitconfig => set of enabled tests. Having a standard, easy way of getting the list could be useful in a number of ways. For example, if we also extended kunit.filter_glob to allow filtering on tests, this would allow users to run tests cases one by one if they wanted to debug hermeticity issues. This patch: * adds a kunit.action module param with one valid non-null value, "list" * for the "list" action, it simply prints out "." * does not itself introduce kunit.py changes to make use of this [1]. Note: kunit.filter_glob is respected for this and all future actions. Note: we need a TAP header for kunit.py to isolate the KUnit output. Go with a more generic "action" param, since it seems like we might eventually have more modes besides just running or listing tests, e.g. * perhaps a benchmark mode that reruns test cases and reports timing * perhaps a deflake mode that reruns test cases that failed * perhaps a mode where we randomize test order to try and catch hermeticity bugs like "test a only passes if run after test b" Tested: $ ./tools/testing/kunit/kunit.py run --kernel_arg=kunit.action=list --raw_output=kunit ... TAP version 14 1..1 example.example_simple_test example.example_skip_test example.example_mark_skipped_test reboot: System halted [1] The interface for this can work in a few ways. We could add a --list_tests flag or a new subcommand. But this change is enough to allow people to split each suite into its own invocation, e.g. via a short script like: #!/bin/bash cd $(git rev-parse --show-toplevel) for suite in $( ./tools/testing/kunit/kunit.py run --kernel_args=kunit.action=list --raw_output=kunit | sed -n '/^TAP version/,$p' | grep -P -o '^[a-z][a-z0-9_-]+\.' | tr -d '.' | sort -u); do ./tools/testing/kunit/kunit.py run "${suite}" done Signed-off-by: Daniel Latypov --- v1 -> v2: write about potential other "actions" in commit desc. --- lib/kunit/executor.c | 46 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 41 insertions(+), 5 deletions(-) diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c index acd1de436f59..77d99ee5ed64 100644 --- a/lib/kunit/executor.c +++ b/lib/kunit/executor.c @@ -15,9 +15,16 @@ extern struct kunit_suite * const * const __kunit_suites_end[]; #if IS_BUILTIN(CONFIG_KUNIT) static char *filter_glob_param; +static char *action_param; + module_param_named(filter_glob, filter_glob_param, charp, 0); MODULE_PARM_DESC(filter_glob, - "Filter which KUnit test suites run at boot-time, e.g. list*"); + "Filter which KUnit test suites run at boot-time, e.g. list*"); +module_param_named(action, action_param, charp, 0); +MODULE_PARM_DESC(action, + "Changes KUnit executor behavior, valid values are:\n" + ": run the tests like normal\n" + "'list' to list test names instead of running them.\n"); static char *kunit_shutdown; core_param(kunit_shutdown, kunit_shutdown, charp, 0644); @@ -109,6 +116,33 @@ static void kunit_print_tap_header(struct suite_set *suite_set) pr_info("1..%d\n", num_of_suites); } +static void kunit_exec_run_tests(struct suite_set *suite_set) +{ + struct kunit_suite * const * const *suites; + + kunit_print_tap_header(suite_set); + + for (suites = suite_set->start; suites < suite_set->end; suites++) + __kunit_test_suites_init(*suites); +} + +static void kunit_exec_list_tests(struct suite_set *suite_set) +{ + unsigned int i; + struct kunit_suite * const * const *suites; + struct kunit_case *test_case; + + /* Hack: print a tap header so kunit.py can find the start of KUnit output. */ + kunit_print_tap_header(suite_set); + + for (suites = suite_set->start; suites < suite_set->end; suites++) + for (i = 0; (*suites)[i] != NULL; i++) { + kunit_suite_for_each_test_case((*suites)[i], test_case) { + pr_info("%s.%s\n", (*suites)[i]->name, test_case->name); + } + } +} + int kunit_run_all_tests(void) { struct kunit_suite * const * const *suites; @@ -120,10 +154,12 @@ int kunit_run_all_tests(void) if (filter_glob_param) suite_set = kunit_filter_suites(&suite_set, filter_glob_param); - kunit_print_tap_header(&suite_set); - - for (suites = suite_set.start; suites < suite_set.end; suites++) - __kunit_test_suites_init(*suites); + if (!action_param) + kunit_exec_run_tests(&suite_set); + else if (strcmp(action_param, "list") == 0) + kunit_exec_list_tests(&suite_set); + else + pr_err("kunit executor: unknown action '%s'\n", action_param); if (filter_glob_param) { /* a copy was made of each array */ for (suites = suite_set.start; suites < suite_set.end; suites++)