From patchwork Sat Jun 10 00:51:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 13274650 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31CDBC87FDC for ; Sat, 10 Jun 2023 00:52:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232602AbjFJAwS (ORCPT ); Fri, 9 Jun 2023 20:52:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232437AbjFJAwR (ORCPT ); Fri, 9 Jun 2023 20:52:17 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D00E730E8 for ; Fri, 9 Jun 2023 17:52:14 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-bacfa4eefcbso5177718276.1 for ; Fri, 09 Jun 2023 17:52:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686358334; x=1688950334; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=uBsX1t2HFlz4ikZu05s+Ij9XO4LQtrN6NyLytDkTSA4=; b=Sn2Dd/x48qKkeIH4X8r0VPCdNgCpAfZC4xE9ujjDXRue4B+pfHp1HmaywFTo0K3+6o lIMdQITumf52CzC5mIO9rOY+PwTnj+07s0iqrmtJgtRE2Z8oTSd6T6sNPkPhlZnY9osq nfZiY0EuUD6jsHoET9sCuAxNiGJ9Hf9JR+wKeOS4o+tt+EjPPcW891tA3Mq4kGdCXoVL SnYw+lK2w+y7vTZIL5IyuiT+SEO32BbAhL6HWhvz289unvwaZ7++RyMiK1UTmhtSO8bG 3aBMxvhnYHTlNasVxX5xJjtP5RRCRCcYd4kAdxCQjIhdVpM4D8LZF/q+hgEfBSnUqHeR 7qvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686358334; x=1688950334; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=uBsX1t2HFlz4ikZu05s+Ij9XO4LQtrN6NyLytDkTSA4=; b=JVVCjJX7EW3vLbXgTrj+RVOPabtPYzM1AB34ZcbTd/zGwcIq8l8nYkSPu3YWw5qnkC QaTv5zXc7Ho2neNQIU+apK5hvukwtR2N7H5pQyAZAvjpnsUHcjc7FNifI0cnj2OrIGHl C9hadoMzdaHtg3YuCDw+rmcXoZAFR8H1d6PvliwGVQmc4+4netJUmbrLzIowh4Av7Ku/ XLih/lzmrM8UzGekLbzV1UxIcBKDGVl9glTahhgHm7Dk77Xi7G6hoUQ8B2u5GV1LKzF8 Me+7q2wjifWaFflwZuY8T2X5DcpXtH6CM4ENebWWsBvmZNycOYXt7N1mphH684wjOe+y MNVQ== X-Gm-Message-State: AC+VfDw4D5f/WdOYru8zm36Wj+fbvV6btShK90KgH32ZQ/lsUN2iLNhk cV4dmRJv8uv9F5DiP5tbbLien/rsLw== X-Google-Smtp-Source: ACHHUZ7SEar8r2ciV3/3VZaJyuZQBccjH4OUT8kuimqEuwg1KKuOjTcXWu1yCDpCVeSKtPjG3+Q6ZcY4OQ== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a25:8c07:0:b0:bad:2b06:da3 with SMTP id k7-20020a258c07000000b00bad2b060da3mr2079249ybl.3.1686358333943; Fri, 09 Jun 2023 17:52:13 -0700 (PDT) Date: Sat, 10 Jun 2023 00:51:47 +0000 In-Reply-To: <20230610005149.1145665-1-rmoar@google.com> Mime-Version: 1.0 References: <20230610005149.1145665-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230610005149.1145665-5-rmoar@google.com> Subject: [RFC v1 4/6] kunit: tool: Add command line interface to filter and report attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-hardening@vger.kernel.org Add ability to use kunit.py to filter attributes and to report a list of tests including attributes without running tests. Add flag "--filter" to input filters on test attributes. Tests will be filtered out if they do not match all inputted filters. Example: --filter speed=slow This filter would run only the tests that are marked as slow. Note there cannot be spaces within a filter. As said in the previous patch, filters can have different operations: <, >, <=, >=, !=, and =. Note that the characters < and > are often interpreted by the shell, so they may need to be quoted or escaped. Example: --filter "speed>=normal" or –filter speed\>=normal This filter would run only the tests that have the speed faster than or equal to normal. Add flag "--list_tests" to output a list of tests and their attributes without running tests. This will be useful to see test attributes and which tests will run with given filters. Example of the output of these tests: example example.test_1 example.test_2 # example.test_2.speed: slow This output includes a suite, example, with two test cases, test_1 and test_2. And in this instance test_2 has been marked as slow. Signed-off-by: Rae Moar --- tools/testing/kunit/kunit.py | 34 +++++++++++++++++---- tools/testing/kunit/kunit_kernel.py | 6 ++-- tools/testing/kunit/kunit_tool_test.py | 41 +++++++++++++------------- 3 files changed, 54 insertions(+), 27 deletions(-) diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 3905c43369c3..661c39f7acf5 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -55,8 +55,10 @@ class KunitExecRequest(KunitParseRequest): build_dir: str timeout: int filter_glob: str + filter: Optional[List[str]] kernel_args: Optional[List[str]] run_isolated: Optional[str] + list_tests: Optional[bool] @dataclass class KunitRequest(KunitExecRequest, KunitBuildRequest): @@ -100,7 +102,7 @@ def config_and_build_tests(linux: kunit_kernel.LinuxSourceTree, return build_tests(linux, request) -def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> List[str]: +def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> Iterable[str]: args = ['kunit.action=list'] if request.kernel_args: args.extend(request.kernel_args) @@ -108,13 +110,17 @@ def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) output = linux.run_kernel(args=args, timeout=request.timeout, filter_glob=request.filter_glob, + filter=request.filter, build_dir=request.build_dir) lines = kunit_parser.extract_tap_lines(output) # Hack! Drop the dummy TAP version header that the executor prints out. lines.pop() # Filter out any extraneous non-test output that might have gotten mixed in. - return [l for l in lines if re.match(r'^[^\s.]+\.[^\s.]+$', l)] + return output + +def _get_tests(output: Iterable[str]) -> List[str]: + return [l for l in output if re.match(r'^[^\s.]+\.[^\s.]+$', l)] def _suites_from_test_list(tests: List[str]) -> List[str]: """Extracts all the suites from an ordered list of tests.""" @@ -132,8 +138,14 @@ def _suites_from_test_list(tests: List[str]) -> List[str]: def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> KunitResult: filter_globs = [request.filter_glob] + if request.list_tests: + output = _list_tests(linux, request) + for line in output: + print(line.rstrip()) + return KunitResult(status=KunitStatus.SUCCESS, elapsed_time=0.0) if request.run_isolated: - tests = _list_tests(linux, request) + output = _list_tests(linux, request) + tests = _get_tests(output) if request.run_isolated == 'test': filter_globs = tests elif request.run_isolated == 'suite': @@ -155,6 +167,7 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) - args=request.kernel_args, timeout=request.timeout, filter_glob=filter_glob, + filter=request.filter, build_dir=request.build_dir) _, test_result = parse_tests(request, metadata, run_result) @@ -341,6 +354,11 @@ def add_exec_opts(parser: argparse.ArgumentParser) -> None: nargs='?', default='', metavar='filter_glob') + parser.add_argument('--filter', + help='Filter which KUnit tests run by attributes' + 'e.g. speed=fast or speed=>low', + type=str, + nargs='*') parser.add_argument('--kernel_args', help='Kernel command-line parameters. Maybe be repeated', action='append', metavar='') @@ -350,6 +368,8 @@ def add_exec_opts(parser: argparse.ArgumentParser) -> None: 'what ran before it.', type=str, choices=['suite', 'test']) + parser.add_argument('--list_tests', help='If set, list all tests and attributes.', + action='store_true') def add_parse_opts(parser: argparse.ArgumentParser) -> None: parser.add_argument('--raw_output', help='If set don\'t parse output from kernel. ' @@ -398,8 +418,10 @@ def run_handler(cli_args: argparse.Namespace) -> None: json=cli_args.json, timeout=cli_args.timeout, filter_glob=cli_args.filter_glob, + filter=cli_args.filter, kernel_args=cli_args.kernel_args, - run_isolated=cli_args.run_isolated) + run_isolated=cli_args.run_isolated, + list_tests=cli_args.list_tests) result = run_tests(linux, request) if result.status != KunitStatus.SUCCESS: sys.exit(1) @@ -441,8 +463,10 @@ def exec_handler(cli_args: argparse.Namespace) -> None: json=cli_args.json, timeout=cli_args.timeout, filter_glob=cli_args.filter_glob, + filter=cli_args.filter, kernel_args=cli_args.kernel_args, - run_isolated=cli_args.run_isolated) + run_isolated=cli_args.run_isolated, + list_tests=cli_args.list_tests) result = exec_tests(linux, exec_request) stdout.print_with_timestamp(( 'Elapsed time: %.3fs\n') % (result.elapsed_time)) diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index 7f648802caf6..62cb8200f60e 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -330,11 +330,13 @@ class LinuxSourceTree: return False return self.validate_config(build_dir) - def run_kernel(self, args: Optional[List[str]]=None, build_dir: str='', filter_glob: str='', timeout: Optional[int]=None) -> Iterator[str]: + def run_kernel(self, args: Optional[List[str]]=None, build_dir: str='', filter_glob: str='', filter: Optional[List[str]]=None, timeout: Optional[int]=None) -> Iterator[str]: if not args: args = [] if filter_glob: - args.append('kunit.filter_glob='+filter_glob) + args.append('kunit.filter_glob=' + filter_glob) + if filter: + args.append('kunit.filter=' + (','.join(filter))) args.append('kunit.enable=1') process = self._ops.start(args, build_dir) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index be35999bb84f..4a7f3112d06c 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -14,6 +14,7 @@ import tempfile, shutil # Handling test_tmpdir import itertools import json import os +import re import signal import subprocess from typing import Iterable @@ -597,7 +598,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 0) self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir='.kunit', filter_glob='', timeout=300) + args=None, build_dir='.kunit', filter_glob='', filter=None, timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_run_passes_args_pass(self): @@ -605,7 +606,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir='.kunit', filter_glob='', timeout=300) + args=None, build_dir='.kunit', filter_glob='', filter=None, timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_exec_passes_args_fail(self): @@ -629,7 +630,7 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run']) self.assertEqual(e.exception.code, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir='.kunit', filter_glob='', timeout=300) + args=None, build_dir='.kunit', filter_glob='', filter=None, timeout=300) self.print_mock.assert_any_call(StrContains(' 0 tests run!')) def test_exec_raw_output(self): @@ -670,13 +671,13 @@ class KUnitMainTest(unittest.TestCase): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) kunit.main(['run', '--raw_output', 'filter_glob']) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir='.kunit', filter_glob='filter_glob', timeout=300) + args=None, build_dir='.kunit', filter_glob='filter_glob', filter=None, timeout=300) def test_exec_timeout(self): timeout = 3453 kunit.main(['exec', '--timeout', str(timeout)]) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir='.kunit', filter_glob='', timeout=timeout) + args=None, build_dir='.kunit', filter_glob='', filter=None, timeout=timeout) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_run_timeout(self): @@ -684,7 +685,7 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run', '--timeout', str(timeout)]) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir='.kunit', filter_glob='', timeout=timeout) + args=None, build_dir='.kunit', filter_glob='', filter=None, timeout=timeout) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_run_builddir(self): @@ -692,7 +693,7 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run', '--build_dir=.kunit']) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir=build_dir, filter_glob='', timeout=300) + args=None, build_dir=build_dir, filter_glob='', filter=None, timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_config_builddir(self): @@ -710,7 +711,7 @@ class KUnitMainTest(unittest.TestCase): build_dir = '.kunit' kunit.main(['exec', '--build_dir', build_dir]) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir=build_dir, filter_glob='', timeout=300) + args=None, build_dir=build_dir, filter_glob='', filter=None, timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_run_kunitconfig(self): @@ -786,7 +787,7 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run', '--kernel_args=a=1', '--kernel_args=b=2']) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=['a=1','b=2'], build_dir='.kunit', filter_glob='', timeout=300) + args=['a=1','b=2'], build_dir='.kunit', filter_glob='', filter=None, timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_list_tests(self): @@ -794,12 +795,13 @@ class KUnitMainTest(unittest.TestCase): self.linux_source_mock.run_kernel.return_value = ['TAP version 14', 'init: random output'] + want got = kunit._list_tests(self.linux_source_mock, - kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'suite')) + kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, None, 'suite', False)) + tests = kunit._get_tests(got) - self.assertEqual(got, want) + self.assertEqual(tests, want) # Should respect the user's filter glob when listing tests. self.linux_source_mock.run_kernel.assert_called_once_with( - args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', timeout=300) + args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', filter=None, timeout=300) @mock.patch.object(kunit, '_list_tests') @@ -809,10 +811,10 @@ class KUnitMainTest(unittest.TestCase): # Should respect the user's filter glob when listing tests. mock_tests.assert_called_once_with(mock.ANY, - kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*', None, 'suite')) + kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*', None, None, 'suite', False)) self.linux_source_mock.run_kernel.assert_has_calls([ - mock.call(args=None, build_dir='.kunit', filter_glob='suite.test*', timeout=300), - mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test*', timeout=300), + mock.call(args=None, build_dir='.kunit', filter_glob='suite.test*', filter=None, timeout=300), + mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test*', filter=None, timeout=300), ]) @mock.patch.object(kunit, '_list_tests') @@ -822,13 +824,12 @@ class KUnitMainTest(unittest.TestCase): # Should respect the user's filter glob when listing tests. mock_tests.assert_called_once_with(mock.ANY, - kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'test')) + kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, None, 'test', False)) self.linux_source_mock.run_kernel.assert_has_calls([ - mock.call(args=None, build_dir='.kunit', filter_glob='suite.test1', timeout=300), - mock.call(args=None, build_dir='.kunit', filter_glob='suite.test2', timeout=300), - mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test1', timeout=300), + mock.call(args=None, build_dir='.kunit', filter_glob='suite.test1', filter=None, timeout=300), + mock.call(args=None, build_dir='.kunit', filter_glob='suite.test2', filter=None, timeout=300), + mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test1', filter=None, timeout=300), ]) - if __name__ == '__main__': unittest.main()