From patchwork Thu Oct 28 06:41:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gow X-Patchwork-Id: 12589395 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C2B0C433EF for ; Thu, 28 Oct 2021 06:42:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4E74860232 for ; Thu, 28 Oct 2021 06:42:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229836AbhJ1God (ORCPT ); Thu, 28 Oct 2021 02:44:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229808AbhJ1God (ORCPT ); Thu, 28 Oct 2021 02:44:33 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 06F5FC061745 for ; Wed, 27 Oct 2021 23:42:06 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id a20-20020a25ae14000000b005c1961310aeso7443339ybj.3 for ; Wed, 27 Oct 2021 23:42:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=6u+AjwrYCRzNV1ppQBIxebat3yD3SQvtae0skIY3lNo=; b=OQdX7s0MYsZvg2MVoinhzq/An17UK9xk4F1ShkS1X7AJIoR0kk77d82PZMHFAjJwe/ 1Z9Fvz0JJ4ZreAjUvJq95p9DPqPVrbNT8+B/Hd1XFG5L9dDc6hgPcGWAzfuQL/IXELUY BJE+AMEWFL9FOSFBP3abGM0OFwv5v+/lgKdQKvw0esnECGg3egjazNBmYhpsF7RcBnIt xUvrK8YMCoCn4yJPuD3qMLEHVFb3u/+Zl8T4dMy0Qte9JTS12OANKQxhU9ZY5CpaK8fF kJirAK2OW4V+m9x+4anTBq0Zb4ATV2YsBxjQwm0M1tKyW56HulxjPfNc8mJO3wWGPynS YCyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=6u+AjwrYCRzNV1ppQBIxebat3yD3SQvtae0skIY3lNo=; b=qF9YFXjgxfRADvHsUNs64UMeqMRsXwW214E8Xs3l+r/XZ6frQB0c/BtF6O7LyDGq1y RnB0OweDvI9HXH2Z0htF/uqPZaFKAjGCe2gpxZU15V3DDWLpiCSoAsbUqZfYQ3vx+pq8 zk17IpTBVa7/+eBd3dj58Xp2Dcm2odK674g4m6nydA/N3ZeHGcwm4yWsCQdABH8HRjHe KZm8RHOQPADwZSslDNPPwmvhecVUAomz04ovqxTYr4OQP0xMkZ35vOObpQjvuf0JKEON NhXlMQ2UdoCitZg6K6+DohZM82QafOsf23a621acl5reEansa+8kfqpIoKMjOo6L6kbm euOQ== X-Gm-Message-State: AOAM5300YFVpVsZYu1LDzQ12iZyGnpjIUaUfliagfsxwNjAy2GObu/rb CMDWJYFwCNAKGFEDOy6xooFRicgwh2b5Mg== X-Google-Smtp-Source: ABdhPJzpI+bgenpEZq6hGo6zkOJiZT9RdDaOIZ+imxux9xFbr9i7mMFrJlRCbW6YT+TUTg2ztqYxeDld8j3XjQ== X-Received: from spirogrip.svl.corp.google.com ([2620:15c:2cb:201:d8b1:a4bf:2b6f:92f9]) (user=davidgow job=sendgmr) by 2002:a25:344d:: with SMTP id b74mr2711522yba.317.1635403326178; Wed, 27 Oct 2021 23:42:06 -0700 (PDT) Date: Wed, 27 Oct 2021 23:41:51 -0700 Message-Id: <20211028064154.2301049-1-davidgow@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog Subject: [PATCH v3 1/4] kunit: tool: Do not error on tests without test plans From: David Gow To: Brendan Higgins , Rae Moar , Daniel Latypov , Shuah Khan Cc: David Gow , kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org The (K)TAP spec encourages test output to begin with a 'test plan': a count of the number of tests being run of the form: 1..n However, some test suites might not know the number of subtests in advance (for example, KUnit's parameterised tests use a generator function). In this case, it's not possible to print the test plan in advance. kunit_tool already parses test output which doesn't contain a plan, but reports an error. Since we want to use nested subtests with KUnit paramterised tests, remove this error. Signed-off-by: David Gow Reviewed-by: Daniel Latypov --- Changes since v2: https://lore.kernel.org/linux-kselftest/20211027013702.2039566-1-davidgow@google.com/ - No code changes. - Added Daniel's Reviewed-by. tools/testing/kunit/kunit_parser.py | 5 ++--- tools/testing/kunit/kunit_tool_test.py | 5 ++++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 3355196d0515..50ded55c168c 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -340,8 +340,8 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: """ Parses test plan line and stores the expected number of subtests in test object. Reports an error if expected count is 0. - Returns False and reports missing test plan error if fails to parse - test plan. + Returns False and sets expected_count to None if there is no valid test + plan. Accepted format: - '1..[number of subtests]' @@ -356,7 +356,6 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: match = TEST_PLAN.match(lines.peek()) if not match: test.expected_count = None - test.add_error('missing plan line!') return False test.log.append(lines.pop()) expected_count = int(match.group(1)) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 9c4126731457..bc8793145713 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -191,7 +191,10 @@ class KUnitParserTest(unittest.TestCase): result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines( file.readlines())) - self.assertEqual(2, result.test.counts.errors) + # A missing test plan is not an error. + self.assertEqual(0, result.test.counts.errors) + # All tests should be accounted for. + self.assertEqual(10, result.test.counts.total()) self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) From patchwork Thu Oct 28 06:41:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gow X-Patchwork-Id: 12589397 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC0BDC433FE for ; Thu, 28 Oct 2021 06:42:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C212860E78 for ; Thu, 28 Oct 2021 06:42:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229899AbhJ1Gog (ORCPT ); Thu, 28 Oct 2021 02:44:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229881AbhJ1Gof (ORCPT ); Thu, 28 Oct 2021 02:44:35 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F673C061745 for ; Wed, 27 Oct 2021 23:42:09 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id s189-20020a252cc6000000b005c1f206d91eso1261024ybs.14 for ; Wed, 27 Oct 2021 23:42:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=eDps66GpWebAOFJkT/oYyV6A6q385IvZARapZ9ebx50=; b=FQhRUBLQrCs4CqOs8Xt1ZZKlqIqHPdRhavV4Y867bpsKQourynNGm2mekZyr0GDtA3 zuHWcqSyM1gNnsz0tarhBMoaMaRPu/g0Up8ACt2DDTHv4AFZ0NX2CIFRqUwREfhdbBIS n7cpkSin6fo3nh7DCj8YLX0C0FgQvstIA/R8AJbsZjskIMSOkvVCowcW3ngxYvjDRjwo lkeBEByi1Zzf8bhE4wDsLf+97kEU3VtpXx9SFhA58i1OmqOIrfJdCj7ISYcuHcR2ba+p UcfTa6GDOwk/m6efQ/fdyqBg5wM8BJcFO5xk1V58n5ArP/cH0B66BnWUUeYaCFrUWVeB thlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=eDps66GpWebAOFJkT/oYyV6A6q385IvZARapZ9ebx50=; b=pV8M7cL+SwIxXfFZutkJWnTNYS2wq3eEG/43i6ym9aBtZ0qPbmAxvnhjiHIvkCr5tj 8uAEunm7nOdQWqfCY6NSZd/Mw4IlWozVC77pwpZA/CX06VMhNGWP+R/H0PbcwaLNgVlH SBc4+amYAUtE3l0VgDeCFZudrhV8suRTPatSwfbQZRoXYulSwlS6iVynbX7lLERFd2aC mN+NWYi5E2cdql/GuYF22SxEUbKfhyAcgpde8fUuI0tJyePkqz/mGRmzkFBidYL2V3NC B5JAqsCF456o1Qx8/U5HvXJW0gPBUxyFDZ7ujnhwEhOUVYBeXbK+4AfVh2+p449v4cr6 0WqA== X-Gm-Message-State: AOAM5330K95ZCvWHTomAKmVTbFfyMcnVsOnX1a7dPENWbPng03Etiq0p HrUSOjH9uRWv1S6MvBhYnYzpdOzfaWMfRw== X-Google-Smtp-Source: ABdhPJxMQm+ThJkQ0X/ukFChGQV4SQd/ZCG7bPIDy7HQPMM9yQXixzVML/g/S2OTgb+VP+/c7Z1zF7HTeP/IVA== X-Received: from spirogrip.svl.corp.google.com ([2620:15c:2cb:201:d8b1:a4bf:2b6f:92f9]) (user=davidgow job=sendgmr) by 2002:a5b:402:: with SMTP id m2mr2540256ybp.422.1635403328629; Wed, 27 Oct 2021 23:42:08 -0700 (PDT) Date: Wed, 27 Oct 2021 23:41:52 -0700 In-Reply-To: <20211028064154.2301049-1-davidgow@google.com> Message-Id: <20211028064154.2301049-2-davidgow@google.com> Mime-Version: 1.0 References: <20211028064154.2301049-1-davidgow@google.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog Subject: [PATCH v3 2/4] kunit: tool: Report an error if any test has no subtests From: David Gow To: Brendan Higgins , Rae Moar , Daniel Latypov , Shuah Khan Cc: David Gow , kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org It's possible for a test to have a subtest header, but zero valid subtests. We used to error on this if the test plan had no subtests listed, but it's possible to have subtests without a test plan (indeed, this is how parameterised tests work). Tests with 0 subtests now have the result NO_TESTS, and will report an error (which does not halt test execution, but is printed in a scary red colour and is noted in the results summary). Signed-off-by: David Gow Reviewed-by: Daniel Latypov --- Changes since v2: https://lore.kernel.org/linux-kselftest/20211027013702.2039566-2-davidgow@google.com/ - Report NO_TESTS as '[NO TESTS RUN]' in yellow, instead of '[FAILED]' in red, particularly since it doesn't get counted as a failure. tools/testing/kunit/kunit_parser.py | 16 +++++++++++----- tools/testing/kunit/kunit_tool_test.py | 9 +++++++++ .../test_is_test_passed-no_tests_no_plan.log | 7 +++++++ 3 files changed, 27 insertions(+), 5 deletions(-) create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 50ded55c168c..68c847e8ca58 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -360,9 +360,6 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: test.log.append(lines.pop()) expected_count = int(match.group(1)) test.expected_count = expected_count - if expected_count == 0: - test.status = TestStatus.NO_TESTS - test.add_error('0 tests run!') return True TEST_RESULT = re.compile(r'^(ok|not ok) ([0-9]+) (- )?([^#]*)( # .*)?$') @@ -589,6 +586,8 @@ def format_test_result(test: Test) -> str: return (green('[PASSED] ') + test.name) elif test.status == TestStatus.SKIPPED: return (yellow('[SKIPPED] ') + test.name) + elif test.status == TestStatus.NO_TESTS: + return (yellow('[NO TESTS RUN] ') + test.name) elif test.status == TestStatus.TEST_CRASHED: print_log(test.log) return (red('[CRASHED] ') + test.name) @@ -731,6 +730,7 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: # test plan test.name = "main" parse_test_plan(lines, test) + parent_test = True else: # If KTAP/TAP header is not found, test must be subtest # header or test result line so parse attempt to parser @@ -744,7 +744,7 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: expected_count = test.expected_count subtests = [] test_num = 1 - while expected_count is None or test_num <= expected_count: + while parent_test and (expected_count is None or test_num <= expected_count): # Loop to parse any subtests. # Break after parsing expected number of tests or # if expected number of tests is unknown break when test @@ -779,9 +779,15 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: parse_test_result(lines, test, expected_num) else: test.add_error('missing subtest result line!') + + # Check for there being no tests + if parent_test and len(subtests) == 0: + test.status = TestStatus.NO_TESTS + test.add_error('0 tests run!') + # Add statuses to TestCounts attribute in Test object bubble_up_test_results(test) - if parent_test: + if parent_test and not main: # If test has subtests and is not the main test object, print # footer. print_test_footer(test) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index bc8793145713..c59fe0777387 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -208,6 +208,15 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual( kunit_parser.TestStatus.NO_TESTS, result.status) + no_plan_log = test_data_path('test_is_test_passed-no_tests_no_plan.log') + with open(no_plan_log) as file: + result = kunit_parser.parse_run_tests( + kunit_parser.extract_tap_lines(file.readlines())) + self.assertEqual(0, len(result.test.subtests[0].subtests[0].subtests)) + self.assertEqual( + kunit_parser.TestStatus.NO_TESTS, + result.test.subtests[0].subtests[0].status) + def test_no_kunit_output(self): crash_log = test_data_path('test_insufficient_memory.log') diff --git a/tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log b/tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log new file mode 100644 index 000000000000..dd873c981108 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log @@ -0,0 +1,7 @@ +TAP version 14 +1..1 + # Subtest: suite + 1..1 + # Subtest: case + ok 1 - case # SKIP +ok 1 - suite From patchwork Thu Oct 28 06:41:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gow X-Patchwork-Id: 12589399 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E138C433EF for ; Thu, 28 Oct 2021 06:42:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 846C860E78 for ; Thu, 28 Oct 2021 06:42:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230041AbhJ1Goo (ORCPT ); Thu, 28 Oct 2021 02:44:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37358 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229921AbhJ1Goi (ORCPT ); Thu, 28 Oct 2021 02:44:38 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6EBDC061745 for ; Wed, 27 Oct 2021 23:42:11 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id v18-20020a25fc12000000b005c1f3933580so650293ybd.2 for ; Wed, 27 Oct 2021 23:42:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=P9SWEypmTPbKVG7Ig7VoDuxi23ZRLA0e0iWZ35O43SQ=; b=XuvBLav4PJf897n1gD6mFbx7dVi4qyYLHwS8nsy3imdYJZ2SMQrqgEszRP8AOk3jnr XEoeNC+xM+lcY3q05C1fE8cz2fYaPHBgDdANlc7X5yJI8w3XiSpRY33MvVodt4F+IAVV z+E9C6iKX9vysN8hfkU08+APtUeHC6qG+SxC4CZ8hm2ILADMo6DnEYVLE/WenBTjo3Q0 h1J6ZRSbkIMAiI54N3OBG35x0ZhBjJjpidMeA6sP5zWT6nLippHJicjdm/b6Vfz7n5DY eXpQDc/FcPQYP++3vzaoW3buWjj7YMUaTeGLxGD7FT6r1Aut5KWaT8bPoHcZdkWF+vVI H6SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=P9SWEypmTPbKVG7Ig7VoDuxi23ZRLA0e0iWZ35O43SQ=; b=5ruCnNIzyg6/QMjYf4p59hNmkVI2RZHQ8HvOvZ2MwjNYJckgr/ku2uS49uGp5MhriP jKqlrm+a5Zt6U61k++P7/j+t2VLVUv+BA4E1oyZEn/Y1WVp7qvLIothEa1KCnB5CXdoE 63P8zPfnxYGUpM/aqgerd4kZoP6mfz0lVTjgCBx+97GxDn/QEj+z1LEj4jvL/pwksA9+ L1oyImWR2jtl83jZlI1dEbooW5Hkn4dyss17vBIVY89JUBcGHmYT8AdmTCyhtXiFxczc 3ANYE1GgZNYJGNOi4yfy7IroAGh5BktySs4YsaCIoJFhjSujxZmD1KZtfmHqY8XwUvVK W2LQ== X-Gm-Message-State: AOAM532UZTbChMbAEqQCqBqBqG7LKxD8A7n+LTHDM9V/3eO57ZQAgP04 DdniDYq7EizF/w1YPassGuVDHXnSC7T22Q== X-Google-Smtp-Source: ABdhPJyrmo5cPPVA73k/5tdPcCtWy7uigQkPXo9HvkjSfLMjgpehGGaOw4rRFXN41aetORDMdi33RFArVVeCjg== X-Received: from spirogrip.svl.corp.google.com ([2620:15c:2cb:201:d8b1:a4bf:2b6f:92f9]) (user=davidgow job=sendgmr) by 2002:a25:e406:: with SMTP id b6mr2631217ybh.134.1635403331110; Wed, 27 Oct 2021 23:42:11 -0700 (PDT) Date: Wed, 27 Oct 2021 23:41:53 -0700 In-Reply-To: <20211028064154.2301049-1-davidgow@google.com> Message-Id: <20211028064154.2301049-3-davidgow@google.com> Mime-Version: 1.0 References: <20211028064154.2301049-1-davidgow@google.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog Subject: [PATCH v3 3/4] kunit: Don't crash if no parameters are generated From: David Gow To: Brendan Higgins , Rae Moar , Daniel Latypov , Shuah Khan Cc: David Gow , kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org It's possible that a parameterised test could end up with zero parameters. At the moment, the test function will nevertheless be called with NULL as the parameter. Instead, don't try to run the test code, and just mark the test as SKIPped. Reported-by: Daniel Latypov Signed-off-by: David Gow Reviewed-by: Daniel Latypov --- Changes since v2: https://lore.kernel.org/linux-kselftest/20211027013702.2039566-3-davidgow@google.com/ - Rework to not share the loop between the parameterised and non-parameterised test cases. - Suggested by Daniel Latypov. - Avoids using a magic non-zero pointer value. lib/kunit/test.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 3bd741e50a2d..dfe1127aacfd 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -508,12 +508,12 @@ int kunit_run_tests(struct kunit_suite *suite) /* Get initial param. */ param_desc[0] = '\0'; test.param_value = test_case->generate_params(NULL, param_desc); - } + kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT + "# Subtest: %s", test_case->name); - do { - kunit_run_case_catch_errors(suite, test_case, &test); + while (test.param_value) { + kunit_run_case_catch_errors(suite, test_case, &test); - if (test_case->generate_params) { if (param_desc[0] == '\0') { snprintf(param_desc, sizeof(param_desc), "param-%d", test.param_index); @@ -530,11 +530,15 @@ int kunit_run_tests(struct kunit_suite *suite) param_desc[0] = '\0'; test.param_value = test_case->generate_params(test.param_value, param_desc); test.param_index++; - } + kunit_update_stats(¶m_stats, test.status); + } + } else { + /* Non-parameterised test. */ + kunit_run_case_catch_errors(suite, test_case, &test); kunit_update_stats(¶m_stats, test.status); + } - } while (test.param_value); kunit_print_test_stats(&test, param_stats); From patchwork Thu Oct 28 06:41:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gow X-Patchwork-Id: 12589401 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 175A1C433F5 for ; Thu, 28 Oct 2021 06:42:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EEE8860F9B for ; Thu, 28 Oct 2021 06:42:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229989AbhJ1Gor (ORCPT ); Thu, 28 Oct 2021 02:44:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37382 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230005AbhJ1Gon (ORCPT ); Thu, 28 Oct 2021 02:44:43 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AC1BC061243 for ; Wed, 27 Oct 2021 23:42:14 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id i128-20020a252286000000b005beea522aa8so7427988ybi.17 for ; Wed, 27 Oct 2021 23:42:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=hV9Sk/D49Ms8sZBEaKHLIRXSQA10JjJL17+Cu699KU8=; b=VzPnKjwyZnii6XyqrJXnRqvZZ+Vw9DMaHfRItQUYZGQirIk5U2yaND42YceIqOA/Vc MHIeB2Z/5Cps7SEm7Pu+Ps+OqrmCwqT1UjZDsGT78lwHTTBLl+B0JcybpQpim1WW3FwV wMcWKt9e4NRzVYJUeUy/jW4dRZPHS+4XFSBfzRTKVm616visszdj7a34znifgi7EHJ63 kqkfSZhOgsr/plDGGtZyLav/LWgo4IRdMs2JstKcBuPbSy0EWBqJfDorTLVMhSsn/Gq5 PV4CECbtC6V9Q4LSs62a9gmz96hJqE5Nbk/7FQ/d073cebPUZwcqraq4wQw2LZld34sI rcSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=hV9Sk/D49Ms8sZBEaKHLIRXSQA10JjJL17+Cu699KU8=; b=gvC2fSFMDX5xEhnQoDDvIFxDeGZm8nkh+SRuqXNa0cM1EIALfEe9NBm05MSsOms7zU ooq5Funfq4XBK8UffQb662DQxQ+/FwwLuim8yNqeU9zuALy5hYQJsuhiarepqgNmfh8c MtbsYt374+eAB+Il/PcN1FxVebM+wr3ydh5y7Mp81zNqUOtf/R3ZsigtbnkgLiP0JxG/ NlbW87AU/VBPsfy33vU8KqTycyYMFn4r8ctYGMKofDXQpzkKyXHPooi+9GuZi08drsoX ZPHfKz3JvGe0882scD+vU74LcWTkUDfu+AlIYX07JVHkfhBPCbDkSEkgjyNAopTAlzUa 8P2Q== X-Gm-Message-State: AOAM5316k1dwzdoKbLO5Qxgr0ix68KV0UWVUQD0aL82YNasuQMgwCVp+ PZrc6lUR+fe1UkGbU9t8d2KgSow4BFhjdA== X-Google-Smtp-Source: ABdhPJwpSgHdj764HU4HJxQ7lTzJR44lBgZlhMUCD+qfXVIbMRqNsQkGsADTHk+snIOmeORQwPsvU6TYgfH7Qw== X-Received: from spirogrip.svl.corp.google.com ([2620:15c:2cb:201:d8b1:a4bf:2b6f:92f9]) (user=davidgow job=sendgmr) by 2002:a25:b45:: with SMTP id 66mr2436928ybl.176.1635403333441; Wed, 27 Oct 2021 23:42:13 -0700 (PDT) Date: Wed, 27 Oct 2021 23:41:54 -0700 In-Reply-To: <20211028064154.2301049-1-davidgow@google.com> Message-Id: <20211028064154.2301049-4-davidgow@google.com> Mime-Version: 1.0 References: <20211028064154.2301049-1-davidgow@google.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog Subject: [PATCH v3 4/4] kunit: Report test parameter results as (K)TAP subtests From: David Gow To: Brendan Higgins , Rae Moar , Daniel Latypov , Shuah Khan Cc: David Gow , kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Currently, the results for individial parameters in a parameterised test are simply output as (K)TAP diagnostic lines. As kunit_tool now supports nested subtests, report each parameter as its own subtest. For example, here's what the output now looks like: # Subtest: inode_test_xtimestamp_decoding ok 1 - 1901-12-13 Lower bound of 32bit < 0 timestamp, no extra bits ok 2 - 1969-12-31 Upper bound of 32bit < 0 timestamp, no extra bits ok 3 - 1970-01-01 Lower bound of 32bit >=0 timestamp, no extra bits ok 4 - 2038-01-19 Upper bound of 32bit >=0 timestamp, no extra bits ok 5 - 2038-01-19 Lower bound of 32bit <0 timestamp, lo extra sec bit on ok 6 - 2106-02-07 Upper bound of 32bit <0 timestamp, lo extra sec bit on ok 7 - 2106-02-07 Lower bound of 32bit >=0 timestamp, lo extra sec bit on ok 8 - 2174-02-25 Upper bound of 32bit >=0 timestamp, lo extra sec bit on ok 9 - 2174-02-25 Lower bound of 32bit <0 timestamp, hi extra sec bit on ok 10 - 2242-03-16 Upper bound of 32bit <0 timestamp, hi extra sec bit on ok 11 - 2242-03-16 Lower bound of 32bit >=0 timestamp, hi extra sec bit on ok 12 - 2310-04-04 Upper bound of 32bit >=0 timestamp, hi extra sec bit on ok 13 - 2310-04-04 Upper bound of 32bit>=0 timestamp, hi extra sec bit 1. 1 ns ok 14 - 2378-04-22 Lower bound of 32bit>= timestamp. Extra sec bits 1. Max ns ok 15 - 2378-04-22 Lower bound of 32bit >=0 timestamp. All extra sec bits on ok 16 - 2446-05-10 Upper bound of 32bit >=0 timestamp. All extra sec bits on # inode_test_xtimestamp_decoding: pass:16 fail:0 skip:0 total:16 ok 1 - inode_test_xtimestamp_decoding Signed-off-by: David Gow Reviewed-by: Daniel Latypov --- Changes since v2: https://lore.kernel.org/linux-kselftest/20211027013702.2039566-4-davidgow@google.com/ - No changes to this patch. lib/kunit/test.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/kunit/test.c b/lib/kunit/test.c index dfe1127aacfd..c66fe1735054 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -520,9 +520,8 @@ int kunit_run_tests(struct kunit_suite *suite) } kunit_log(KERN_INFO, &test, - KUNIT_SUBTEST_INDENT - "# %s: %s %d - %s", - test_case->name, + KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT + "%s %d - %s", kunit_status_to_ok_not_ok(test.status), test.param_index + 1, param_desc);