From patchwork Wed Oct 27 01:36:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gow X-Patchwork-Id: 12586099 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AE80C433EF for ; Wed, 27 Oct 2021 01:37:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1038761073 for ; Wed, 27 Oct 2021 01:37:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237590AbhJ0Bjh (ORCPT ); Tue, 26 Oct 2021 21:39:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233859AbhJ0Bjh (ORCPT ); Tue, 26 Oct 2021 21:39:37 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1B40C061745 for ; Tue, 26 Oct 2021 18:37:12 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id d19-20020a25add3000000b005c139e0f641so1508985ybe.7 for ; Tue, 26 Oct 2021 18:37:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=O4M/tQHIU9ygzYdcq00Cuq5xDfJVENaFvMJsaxgv1Uk=; b=ipgwRjf4MbnXNFmRgqWaZKZYpqsFrVjjUM5IUmIj48IJ+uXEjqX3lIvgCj//ujftMU 8lyuEx0+D6Xnz/iTY0d240W6dovoQflTqdnpDdWe8YLh89Rrv+ShMxUyOtljaa8WlHbb 5wiweMoy+DXW/36/PwubPsKHEAAaKv7YfU9R3XQT2rFU3D7L7qM3MjTowcXDnBrmNtYs 9qZFHqe/wWRR7hcYWECszq6Wsg+RbqFeCKfw5Q2rMjbzDWIZcVicwmbUqY3ELtPXBYwS defaJGwEvYHDpPJaAo6mdglkpkARGqBiyHDoDflmT8FbFRtiQ0CJq9mNvcfztI1WilzM aIZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=O4M/tQHIU9ygzYdcq00Cuq5xDfJVENaFvMJsaxgv1Uk=; b=jkH0QvfCAyVBZJYfZBw3YpYIcxXd7qd3APU9aPfBm3y1vNizDwmnD1BOyOpQNGYFyD y66r7FAJbfF9nmIUVrACicDpt19nsjW7gsFxLwpv2FrVNs+UD0SYsiMke74OhfMbWC3D 5nk8ACyRHR5+mYM/WUctBG5M933PpiMkKdeAtUx6kVRdm+lEUJRQX4Evri5WjWotNSzm YDmdotrgI07bdjY1I+4NbDEpr9iIHEQHyGVRtCvqlxeuh6YJa4wy07YeZ1PHptJxegrk pFRMmvc6pT8xSNKwiWSQ7sz/ux0frkvz+0mBwYGawIh2yM1pKxT86jdOBQMe1AYJUaFc a+OQ== X-Gm-Message-State: AOAM532+vXl2BltJ4SFoqS3FN4zvAM5gVIOGYy+3a5k9VYKVuttQThhA EI/MYDfeJZn5T2P/hc8y3HvpkFIPf2vG2A== X-Google-Smtp-Source: ABdhPJy5wGr37Q6+nC4gTXNyECv/6SgaiCo2x/x0NDGAyWhwPAakrLy53+OSWUWSYjsn7ctqK1CdS15Bb/BFbg== X-Received: from spirogrip.svl.corp.google.com ([2620:15c:2cb:201:c628:e925:d58d:6232]) (user=davidgow job=sendgmr) by 2002:a25:3622:: with SMTP id d34mr1601396yba.325.1635298630761; Tue, 26 Oct 2021 18:37:10 -0700 (PDT) Date: Tue, 26 Oct 2021 18:36:59 -0700 Message-Id: <20211027013702.2039566-1-davidgow@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog Subject: [PATCH v2 1/4] kunit: tool: Do not error on tests without test plans From: David Gow To: Brendan Higgins , Rae Moar , Daniel Latypov , Shuah Khan Cc: David Gow , kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org The (K)TAP spec encourages test output to begin with a 'test plan': a count of the number of tests being run of the form: 1..n However, some test suites might not know the number of subtests in advance (for example, KUnit's parameterised tests use a generator function). In this case, it's not possible to print the test plan in advance. kunit_tool already parses test output which doesn't contain a plan, but reports an error. Since we want to use nested subtests with KUnit paramterised tests, remove this error. Signed-off-by: David Gow Reviewed-by: Daniel Latypov --- tools/testing/kunit/kunit_parser.py | 5 ++--- tools/testing/kunit/kunit_tool_test.py | 5 ++++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 3355196d0515..50ded55c168c 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -340,8 +340,8 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: """ Parses test plan line and stores the expected number of subtests in test object. Reports an error if expected count is 0. - Returns False and reports missing test plan error if fails to parse - test plan. + Returns False and sets expected_count to None if there is no valid test + plan. Accepted format: - '1..[number of subtests]' @@ -356,7 +356,6 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: match = TEST_PLAN.match(lines.peek()) if not match: test.expected_count = None - test.add_error('missing plan line!') return False test.log.append(lines.pop()) expected_count = int(match.group(1)) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 9c4126731457..bc8793145713 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -191,7 +191,10 @@ class KUnitParserTest(unittest.TestCase): result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines( file.readlines())) - self.assertEqual(2, result.test.counts.errors) + # A missing test plan is not an error. + self.assertEqual(0, result.test.counts.errors) + # All tests should be accounted for. + self.assertEqual(10, result.test.counts.total()) self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) From patchwork Wed Oct 27 01:37:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gow X-Patchwork-Id: 12586101 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4698C433F5 for ; Wed, 27 Oct 2021 01:37:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9E83860234 for ; Wed, 27 Oct 2021 01:37:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237600AbhJ0Bji (ORCPT ); Tue, 26 Oct 2021 21:39:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237588AbhJ0Bjh (ORCPT ); Tue, 26 Oct 2021 21:39:37 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 65C4BC061745 for ; Tue, 26 Oct 2021 18:37:13 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id i128-20020a252286000000b005beea522aa8so1449059ybi.17 for ; Tue, 26 Oct 2021 18:37:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=KbePLBSXjJc+ItO7+3Jnjlde8rtlRhHvWKH0kKd5B80=; b=CL35BVEpYmt7VeoOZOl3RV+eDnImtLHSZOMLTZp55viCQSTRJNmRrF8NG8os4UrC3U KRPCb4C31XIg7gqpkIZvAb/MpRDtkzcAURmeOPm5OvdRazmBGez2bjuL47tSXpt6je3T fBHrStjoUKnyny8Ii1LE8VvfVfsQ2E04hPavmu04JaZHy9DMp23l1VAcLAfOhVeZwwQy F44ryA5xwg/M3M2UDIAmU+LaKuEQDby2QSPAsdkp9kDkdAGhrA4Lm6E1wdvrBt9uDbDD 8Prno9NKc3Qn3ToEe29ZI1p4eDVkTiobB2ucDC8Ep7m4JmR4jHj63JQ8tH2QHQCLDb8E v+JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=KbePLBSXjJc+ItO7+3Jnjlde8rtlRhHvWKH0kKd5B80=; b=fszFY2heuHviGWAzctKMGLvn93ig2uJqoIWUfUhPyakiD//UNhi02AxXHE3y8j6OBn dxg5nVV7Y/WcnBaupsPPlqK2sUkN6+v0rtioJ6tz/+XkvgQYUd001fm9Wet/a1oB/63q +siYnLF8BIVZWcOnkiHODLM7mrU2yDuoLcQCntnUt0FFsVN0h+uA6Gad6/s7G8M2ZDDF AP/whq/yKlHBUgGXVQbEvAS0OwBXolZM/5KtwCFgS9YB0GMae9+8M8cZielYBV8x/Wse 4VzsjwXV9BbcCAupJ8C7Frktci0l+sszUcTp/6aZhnjVSSrBIrB15yRyrZbaJpF5Wv0t YuOg== X-Gm-Message-State: AOAM531ye8Bq7Ap5YKRVtzJoCEpv5OhyRGTKvk246yAReezM+zBvImxn 9CQVKOPnlfi+TqVPVmLfFxCHe5q5hGga8w== X-Google-Smtp-Source: ABdhPJwFvLepJVEy6r7oUTHD+UvCDDphjgm3QnJT5artkyBGE5h2xv2USvnTsN5zO4Wdsh2SXiKYRa3E0+tzyw== X-Received: from spirogrip.svl.corp.google.com ([2620:15c:2cb:201:c628:e925:d58d:6232]) (user=davidgow job=sendgmr) by 2002:a25:afcd:: with SMTP id d13mr29514222ybj.504.1635298632637; Tue, 26 Oct 2021 18:37:12 -0700 (PDT) Date: Tue, 26 Oct 2021 18:37:00 -0700 In-Reply-To: <20211027013702.2039566-1-davidgow@google.com> Message-Id: <20211027013702.2039566-2-davidgow@google.com> Mime-Version: 1.0 References: <20211027013702.2039566-1-davidgow@google.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog Subject: [PATCH v2 2/4] kunit: tool: Report an error if any test has no subtests From: David Gow To: Brendan Higgins , Rae Moar , Daniel Latypov , Shuah Khan Cc: David Gow , kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org It's possible for a test to have a subtest header, but zero valid subtests. We used to error on this if the test plan had no subtests listed, but it's possible to have subtests without a test plan (indeed, this is how parameterised tests work). Tests with 0 subtests now have the result NO_TESTS, and will report an error (which does not halt test execution, but is printed in a scary red colour and is noted in the results summary). Signed-off-by: David Gow --- tools/testing/kunit/kunit_parser.py | 14 +++++++++----- tools/testing/kunit/kunit_tool_test.py | 9 +++++++++ .../test_is_test_passed-no_tests_no_plan.log | 7 +++++++ 3 files changed, 25 insertions(+), 5 deletions(-) create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 50ded55c168c..3a838423c381 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -360,9 +360,6 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: test.log.append(lines.pop()) expected_count = int(match.group(1)) test.expected_count = expected_count - if expected_count == 0: - test.status = TestStatus.NO_TESTS - test.add_error('0 tests run!') return True TEST_RESULT = re.compile(r'^(ok|not ok) ([0-9]+) (- )?([^#]*)( # .*)?$') @@ -731,6 +728,7 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: # test plan test.name = "main" parse_test_plan(lines, test) + parent_test = True else: # If KTAP/TAP header is not found, test must be subtest # header or test result line so parse attempt to parser @@ -744,7 +742,7 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: expected_count = test.expected_count subtests = [] test_num = 1 - while expected_count is None or test_num <= expected_count: + while parent_test and (expected_count is None or test_num <= expected_count): # Loop to parse any subtests. # Break after parsing expected number of tests or # if expected number of tests is unknown break when test @@ -779,9 +777,15 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: parse_test_result(lines, test, expected_num) else: test.add_error('missing subtest result line!') + + # Check for there being no tests + if parent_test and len(subtests) == 0: + test.status = TestStatus.NO_TESTS + test.add_error('0 tests run!') + # Add statuses to TestCounts attribute in Test object bubble_up_test_results(test) - if parent_test: + if parent_test and not main: # If test has subtests and is not the main test object, print # footer. print_test_footer(test) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index bc8793145713..c59fe0777387 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -208,6 +208,15 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.NO_TESTS, result.status) + no_plan_log = test_data_path('test_is_test_passed-no_tests_no_plan.log') + with open(no_plan_log) as file: + result = kunit_parser.parse_run_tests( + kunit_parser.extract_tap_lines(file.readlines())) + self.assertEqual(0, len(result.test.subtests[0].subtests[0].subtests)) + self.assertEqual( + kunit_parser.TestStatus.NO_TESTS, + result.test.subtests[0].subtests[0].status) + def test_no_kunit_output(self): crash_log = test_data_path('test_insufficient_memory.log') diff --git a/tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log b/tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log new file mode 100644 index 000000000000..dd873c981108 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log @@ -0,0 +1,7 @@ +TAP version 14 +1..1 + # Subtest: suite + 1..1 + # Subtest: case + ok 1 - case # SKIP +ok 1 - suite From patchwork Wed Oct 27 01:37:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gow X-Patchwork-Id: 12586103 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E9AAC433F5 for ; Wed, 27 Oct 2021 01:37:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4ADAD61040 for ; Wed, 27 Oct 2021 01:37:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237622AbhJ0Bjj (ORCPT ); Tue, 26 Oct 2021 21:39:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38288 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237614AbhJ0Bjj (ORCPT ); Tue, 26 Oct 2021 21:39:39 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D08E8C061745 for ; Tue, 26 Oct 2021 18:37:14 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id p28-20020a637f5c000000b002a3c58b5917so659101pgn.23 for ; Tue, 26 Oct 2021 18:37:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=zdnP+CcH+cj0gO5XtMfjV6pCIZrG8LN0fjsgBOUCOoc=; b=b92pU0ZPZUKilkgCeq0jDth2ifnAa83QiCk+5MfUApUiPVy9/uv1rEJVPOPooi3Ye3 YfCUuvbWmOHiEC2kiFUgckBwDx8ENqtu1wYm9f8ZZzOF6ZWP2R2VWrtb4/OJA5Qh7kHC ockwapbrUmbsiLTmpcJfHXvND0KHLx3uk/mBv9dAkylFjmtjylw30n9ZsqovaSunDOyL CC79OcNdjexhtkkPeny6kbLhIxQVyLc1U66yYRui05WVAQJLHWN1ZjSZhaqkKwCfncbx al3w5JaVx8c3mYiPY1ySJmX5e6/pHOd69OcroHkmgDbIdSi9ygOfZOtNrWZjNHMBtRxR fsjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=zdnP+CcH+cj0gO5XtMfjV6pCIZrG8LN0fjsgBOUCOoc=; b=Ay6y6/Ix+SPPuM2aZ1tNK2KUC6YB5Lgf8v/Hp3HmqEz+9DReLSNJv+uWnX1AFN7M55 mzuXf6BKmsroMwCAOeIm/e93xejWiBAhUKQgxSKeh6pjjuEMgzqFFaqKkzZt+y03EArD 6pneSh853Tcj4cHIuuNRBbxfGQTlY/Texxtv5WCbxa1WHIYXGqxo1xskUFN0WWo9EoGA 0Kp6zNEJRa7h80xVIzbKYkXQihBl1APDt58P2i3755romvn4/Bs1tBnku4rvMOTMq+yl riu7F9Z8kQo2MMynGKtd5iYpTPJ55lTBqyqzIL13aRitK44q2fas2CsjaGy4LR/Q/kLS Ftkg== X-Gm-Message-State: AOAM53281zyNkoUBnn7p8+1cEQKbM0w51hM3V0JlEerfpomn6OnTA+83 RXsh/ONKQ32WKAcBmA18kKEZTzMIwWI9zQ== X-Google-Smtp-Source: ABdhPJwg2vr2nzlms3KujpRdCOHHVuWmWK5Bzr5xe8wdyiH+1BuM42b6Pjw14dOKIyYClEOzMDsPm2olcO1xAQ== X-Received: from spirogrip.svl.corp.google.com ([2620:15c:2cb:201:c628:e925:d58d:6232]) (user=davidgow job=sendgmr) by 2002:a17:902:ed87:b0:141:5fe1:e794 with SMTP id e7-20020a170902ed8700b001415fe1e794mr4334276plj.77.1635298634290; Tue, 26 Oct 2021 18:37:14 -0700 (PDT) Date: Tue, 26 Oct 2021 18:37:01 -0700 In-Reply-To: <20211027013702.2039566-1-davidgow@google.com> Message-Id: <20211027013702.2039566-3-davidgow@google.com> Mime-Version: 1.0 References: <20211027013702.2039566-1-davidgow@google.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog Subject: [PATCH v2 3/4] kunit: Don't crash if no parameters are generated From: David Gow To: Brendan Higgins , Rae Moar , Daniel Latypov , Shuah Khan Cc: David Gow , kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org It's possible that a parameterised test could end up with zero parameters. At the moment, the test function will nevertheless be called with NULL as the parameter. Instead, don't try to run the test code, and just mark the test as SKIPped. Reported-by: Daniel Latypov Signed-off-by: David Gow --- lib/kunit/test.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 3bd741e50a2d..e028d98e4f5b 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -500,7 +500,10 @@ int kunit_run_tests(struct kunit_suite *suite) kunit_print_subtest_start(suite); kunit_suite_for_each_test_case(suite, test_case) { - struct kunit test = { .param_value = NULL, .param_index = 0 }; + /* The initial param value is nonzero, as we want + * non-parametrised tests to run once. + */ + struct kunit test = { .param_value = (void *)-1, .param_index = 0 }; struct kunit_result_stats param_stats = { 0 }; test_case->status = KUNIT_SKIPPED; @@ -510,7 +513,7 @@ int kunit_run_tests(struct kunit_suite *suite) test.param_value = test_case->generate_params(NULL, param_desc); } - do { + while (test.param_value) { kunit_run_case_catch_errors(suite, test_case, &test); if (test_case->generate_params) { @@ -530,11 +533,12 @@ int kunit_run_tests(struct kunit_suite *suite) param_desc[0] = '\0'; test.param_value = test_case->generate_params(test.param_value, param_desc); test.param_index++; - } + } else + test.param_value = NULL; kunit_update_stats(¶m_stats, test.status); - } while (test.param_value); + } kunit_print_test_stats(&test, param_stats); From patchwork Wed Oct 27 01:37:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gow X-Patchwork-Id: 12586105 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 573ECC433FE for ; Wed, 27 Oct 2021 01:37:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3D23C61073 for ; Wed, 27 Oct 2021 01:37:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237625AbhJ0Bjn (ORCPT ); Tue, 26 Oct 2021 21:39:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237614AbhJ0Bjl (ORCPT ); Tue, 26 Oct 2021 21:39:41 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E01D1C061745 for ; Tue, 26 Oct 2021 18:37:16 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id d14-20020a17090a2a4e00b001a2955e1247so296328pjg.4 for ; Tue, 26 Oct 2021 18:37:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=7J7hXEZYHRVzHQmivzMp4k5hS/Z1urfbfNYVR0nOOIk=; b=W+jH/TV8ffRFIZYHg3pvL/VMZ8Z391aQ0OipQqW6+SKlDH1rDYEmqGeIV0T6IKGJMh pZNgdbdBThDYpPNomeKbMXwnyP4fjZ90r/hKw2bcZOEyVedQvLydxUw6Ou3onVTmL2uS 1YXM8ythHyylYJjWnRVLGJY0cLDw7Rzjv/3TigU/N7zYZz4L4RjAiynaEgR9p5QOJBJ2 wyM3RYH0blVpXuiXWstvYW6zuYt4gYjGDcTIwAoXYoZksETyoA1Kx4kEjn79QOa2pr2n gJHKcic4NA6xtarDLxUQYRuDrrKjWIQYCBh0vQlLCM0dcqMx5FHuilQkX6RcEDkX+cX1 k9FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=7J7hXEZYHRVzHQmivzMp4k5hS/Z1urfbfNYVR0nOOIk=; b=YV9gJ7f3FbqbIUR/2IkO3kvL1p8V3wEh1wQ/gsutO7tke+NL2is/6HjJTYx/nuG10s 5l3epJ/1W7nTmHLhdtPaOVWCplCoohT9Syg6ke0M2HDuC3Tiz9Kq1Q6QwVn6LNjDVGVd Ioexd0JbOFwCGnWVC8qxlIIPzLvrgkDfHHG4LxSW448e2GIS4TTlF7cvahTHCGHRZgBi ndR8tIzHTdP6PeZXtThPxHJhXh0JhAYrD+KBAfGmX4Te/tDJLYeci5vcium6BQngnfLf +Hi0lTP8aIIP1K99hjLo2gwaqetVFPora2h3O+lVdqCvyl2Xm8iDY20VE5lX8DXuh3Ur AKZA== X-Gm-Message-State: AOAM533StQej6jbSYZTdUvoRALIhRQ1sqZLsRbPF6TiPkmscWnwjN52I FzXUR68EKPXKpGCHaX7KcV2tw+Adr8PgGQ== X-Google-Smtp-Source: ABdhPJxxHWq6OxvqmxnsH7pPvAGClrg/VRxmiCxws1pwjgP5B9Z2euyjsEbMnndi3o7+1pA7Zv9bpwNxTx/bMg== X-Received: from spirogrip.svl.corp.google.com ([2620:15c:2cb:201:c628:e925:d58d:6232]) (user=davidgow job=sendgmr) by 2002:a17:90b:149:: with SMTP id em9mr2506857pjb.133.1635298636276; Tue, 26 Oct 2021 18:37:16 -0700 (PDT) Date: Tue, 26 Oct 2021 18:37:02 -0700 In-Reply-To: <20211027013702.2039566-1-davidgow@google.com> Message-Id: <20211027013702.2039566-4-davidgow@google.com> Mime-Version: 1.0 References: <20211027013702.2039566-1-davidgow@google.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog Subject: [PATCH v2 4/4] kunit: Report test parameter results as (K)TAP subtests From: David Gow To: Brendan Higgins , Rae Moar , Daniel Latypov , Shuah Khan Cc: David Gow , kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Currently, the results for individial parameters in a parameterised test are simply output as (K)TAP diagnostic lines. As kunit_tool now supports nested subtests, report each parameter as its own subtest. For example, here's what the output now looks like: # Subtest: inode_test_xtimestamp_decoding ok 1 - 1901-12-13 Lower bound of 32bit < 0 timestamp, no extra bits ok 2 - 1969-12-31 Upper bound of 32bit < 0 timestamp, no extra bits ok 3 - 1970-01-01 Lower bound of 32bit >=0 timestamp, no extra bits ok 4 - 2038-01-19 Upper bound of 32bit >=0 timestamp, no extra bits ok 5 - 2038-01-19 Lower bound of 32bit <0 timestamp, lo extra sec bit on ok 6 - 2106-02-07 Upper bound of 32bit <0 timestamp, lo extra sec bit on ok 7 - 2106-02-07 Lower bound of 32bit >=0 timestamp, lo extra sec bit on ok 8 - 2174-02-25 Upper bound of 32bit >=0 timestamp, lo extra sec bit on ok 9 - 2174-02-25 Lower bound of 32bit <0 timestamp, hi extra sec bit on ok 10 - 2242-03-16 Upper bound of 32bit <0 timestamp, hi extra sec bit on ok 11 - 2242-03-16 Lower bound of 32bit >=0 timestamp, hi extra sec bit on ok 12 - 2310-04-04 Upper bound of 32bit >=0 timestamp, hi extra sec bit on ok 13 - 2310-04-04 Upper bound of 32bit>=0 timestamp, hi extra sec bit 1. 1 ns ok 14 - 2378-04-22 Lower bound of 32bit>= timestamp. Extra sec bits 1. Max ns ok 15 - 2378-04-22 Lower bound of 32bit >=0 timestamp. All extra sec bits on ok 16 - 2446-05-10 Upper bound of 32bit >=0 timestamp. All extra sec bits on # inode_test_xtimestamp_decoding: pass:16 fail:0 skip:0 total:16 ok 1 - inode_test_xtimestamp_decoding Signed-off-by: David Gow Reviewed-by: Daniel Latypov --- lib/kunit/test.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/kunit/test.c b/lib/kunit/test.c index e028d98e4f5b..fe2ab31b5949 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -511,6 +511,8 @@ int kunit_run_tests(struct kunit_suite *suite) /* Get initial param. */ param_desc[0] = '\0'; test.param_value = test_case->generate_params(NULL, param_desc); + kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT + "# Subtest: %s", test_case->name); } while (test.param_value) { @@ -523,9 +525,8 @@ int kunit_run_tests(struct kunit_suite *suite) } kunit_log(KERN_INFO, &test, - KUNIT_SUBTEST_INDENT - "# %s: %s %d - %s", - test_case->name, + KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT + "%s %d - %s", kunit_status_to_ok_not_ok(test.status), test.param_index + 1, param_desc);