From patchwork Tue Nov 2 07:30:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gow X-Patchwork-Id: 12598007 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFCE5C433F5 for ; Tue, 2 Nov 2021 07:30:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D519960EB8 for ; Tue, 2 Nov 2021 07:30:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229497AbhKBHc6 (ORCPT ); Tue, 2 Nov 2021 03:32:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229770AbhKBHc5 (ORCPT ); Tue, 2 Nov 2021 03:32:57 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D050C061714 for ; Tue, 2 Nov 2021 00:30:21 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id b126-20020a251b84000000b005bd8aca71a2so29690845ybb.4 for ; Tue, 02 Nov 2021 00:30:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=4TO7cXb2F2AQHNi68Y4UhnFWgCfoNFbr4ZCVJOaX+38=; b=nZ3R7daK2ARwboBEruz7yT3XKV0fn1XsrbiTNK3t7N/+PR1LqhOx2l2vzgm/QxZFOA UGvGqtE0CkCFcPQ3ovL1mr7cSg4qkr8qzFJ0sitC8qcUXH8VBoPfV4riCZx8I5+JkDun Ll4fuLcak6f0hUgyi6SdeLmBlEFKfLqiaxILWxa+WMAU0IUttIJu3NM8lgJNMM49OzRG PcNmXf9pEGA4D8Vuo1wM8y3IiX9NKRuFr+RYdB6u5XDPtNaoRieRRcO4WivPAZVnCew6 ytXbte7+bPj8wCgZ1iUGeSqoVzSHclKHgAMb35HMS91Qp5yfai4RCTFEUuutpCgqVWEq /L5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=4TO7cXb2F2AQHNi68Y4UhnFWgCfoNFbr4ZCVJOaX+38=; b=EMG6yLBzEOR5QivAlHUsZKnDi3LkYPX53uidqC3dCYFUygKcCQrpmNX0E6M6c4i5zE Z2T6iTEvG/Iw8usAnlz+s5HzgV8tfwryf7UV/RULDZ8e/WtQ+WH7V1RspGWGQHJLVjnp v2N+s8dSEA/0DD8RWRnKH5ZGF32lmnMiMng0ukVtkNVJZb9t7ChBaM6ue5k6p/30Brvo 5hbAlCk0/n3JesL3Im+Bhcn2/wKqOFeM+jf34QSsj/+wRPbLCfTxMPLmf37S/Q37AXUx Wzdej2y8qS3c0fFpHcAzJDVoH3wCXVx8/5JQVEeXsKlTCH84t6U84F78QnudWPFrzIuw EcPQ== X-Gm-Message-State: AOAM530lVEfkzlHbNKmhZyG2vgdeBRY8IhwmoRth3BwWTRDmsi3jzb8h mqjTxBv6YeRDKajKPDxph8ur4+GNEx2nhQ== X-Google-Smtp-Source: ABdhPJzzxK4N0+3UkZIX7/cbA0bqfqlwPpFrAUKBBilFhl/vZtMMLFYCw0XJRQnNVClnwP4+DeohLLDs0Viw/Q== X-Received: from spirogrip.svl.corp.google.com ([2620:15c:2cb:201:524f:74d8:b474:2517]) (user=davidgow job=sendgmr) by 2002:a25:2649:: with SMTP id m70mr28997197ybm.21.1635838220057; Tue, 02 Nov 2021 00:30:20 -0700 (PDT) Date: Tue, 2 Nov 2021 00:30:11 -0700 Message-Id: <20211102073014.2901870-1-davidgow@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.1.1089.g2158813163f-goog Subject: [PATCH v4 1/4] kunit: tool: Do not error on tests without test plans From: David Gow To: Brendan Higgins , Rae Moar , Daniel Latypov , Shuah Khan Cc: David Gow , kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org The (K)TAP spec encourages test output to begin with a 'test plan': a count of the number of tests being run of the form: 1..n However, some test suites might not know the number of subtests in advance (for example, KUnit's parameterised tests use a generator function). In this case, it's not possible to print the test plan in advance. kunit_tool already parses test output which doesn't contain a plan, but reports an error. Since we want to use nested subtests with KUnit paramterised tests, remove this error. Signed-off-by: David Gow Reviewed-by: Daniel Latypov Reviewed-by: Brendan Higgins --- No changes since v3: https://lore.kernel.org/linux-kselftest/20211028064154.2301049-1-davidgow@google.com/ Changes since v2: https://lore.kernel.org/linux-kselftest/20211027013702.2039566-1-davidgow@google.com/ - No code changes. - Added Daniel's Reviewed-by. tools/testing/kunit/kunit_parser.py | 5 ++--- tools/testing/kunit/kunit_tool_test.py | 5 ++++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 3355196d0515..50ded55c168c 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -340,8 +340,8 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: """ Parses test plan line and stores the expected number of subtests in test object. Reports an error if expected count is 0. - Returns False and reports missing test plan error if fails to parse - test plan. + Returns False and sets expected_count to None if there is no valid test + plan. Accepted format: - '1..[number of subtests]' @@ -356,7 +356,6 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: match = TEST_PLAN.match(lines.peek()) if not match: test.expected_count = None - test.add_error('missing plan line!') return False test.log.append(lines.pop()) expected_count = int(match.group(1)) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index 9c4126731457..bc8793145713 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -191,7 +191,10 @@ class KUnitParserTest(unittest.TestCase): result = kunit_parser.parse_run_tests( kunit_parser.extract_tap_lines( file.readlines())) - self.assertEqual(2, result.test.counts.errors) + # A missing test plan is not an error. + self.assertEqual(0, result.test.counts.errors) + # All tests should be accounted for. + self.assertEqual(10, result.test.counts.total()) self.assertEqual( kunit_parser.TestStatus.SUCCESS, result.status) From patchwork Tue Nov 2 07:30:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gow X-Patchwork-Id: 12598009 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49F32C433EF for ; Tue, 2 Nov 2021 07:30:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3275D60EB9 for ; Tue, 2 Nov 2021 07:30:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229770AbhKBHc6 (ORCPT ); Tue, 2 Nov 2021 03:32:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229878AbhKBHc5 (ORCPT ); Tue, 2 Nov 2021 03:32:57 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C854C061767 for ; Tue, 2 Nov 2021 00:30:23 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id z65-20020a256544000000b005c21b35717dso12789411ybb.10 for ; Tue, 02 Nov 2021 00:30:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=r67IhOqyj23m1j3onGz6pLQJdGY+9rfgyvIDQST2m8U=; b=Lj7w1D1h/wTVKyHIvCyMUPEkedWGM63l9Yq/LVPAMhKRbLxO7S8AniIHGLxjEnwRuq ym3IkpF8F38ecL6DvC0JeCEhd/kzQDfPaszuBU1ntSYdfo4qJb+V3sh27rNSsLps5pZF Eh8iM1jDcjUfQpOQ/CtAeWQo/XMQhZvT36Y9XThhTn+1V/THyjuL++ZJISKVcFrXkWQ7 NbdSWbhFnXbRrdNmGufI4XGeiYB9kL9y+xOJBlc+WnJs4aRTBQsFD0KUa3dguaNZKQDn 7D4Qvdu3W8ZS0jPCw/O7fPC2ZYV9IxCQXvXLg05tg9rXb1fvh39NkARZ6okME8RRMc/9 3DSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=r67IhOqyj23m1j3onGz6pLQJdGY+9rfgyvIDQST2m8U=; b=0x3AGCNUMZ9IuRrpUCwihUhEswnTyYcI3lEkqAGPqQiiR66EFD6KixTHr01SzOA3eZ VKdpcdo6KmctmFBEYY1cuFCdvEU6dhDGYq6weRRowitQL4+pLjFOXlBkwLyxMoCd/rrI XSSF+8Wec4GRa0ggntPZN8/w+iBPXXhUgO/lWPj2jW8NaBn3901aBoJ68jmkAROumcwf CGQg5/lY1fNJq9NH1EAD2MP33okddHtG5HECvGBlfPxQiQdhRuA1l+UyzO5cc9TYUyjo lS7d8c3/LH271Gp5ZFdtD9RIK7BrRWARe96qBsBcFkkT7K15oBlkjZtM/QEPuSpH2eaR fjiw== X-Gm-Message-State: AOAM530qkf0wPFkEDBz8e3cJAD8xwSGILWbpxAoCU3g7Iorjd3P800Ym Ob9JMnnTxwCzjEv1xaM3L5NxfDEHDAFkTQ== X-Google-Smtp-Source: ABdhPJyj8XA6od3S1FncSK4TcD9GW5PQJl6JIhwoKluHinUhs+iUV0ZapSiA8MQ6VlvGoT0Wr2L3KGSy8q0WhQ== X-Received: from spirogrip.svl.corp.google.com ([2620:15c:2cb:201:524f:74d8:b474:2517]) (user=davidgow job=sendgmr) by 2002:a25:9090:: with SMTP id t16mr20605936ybl.57.1635838222390; Tue, 02 Nov 2021 00:30:22 -0700 (PDT) Date: Tue, 2 Nov 2021 00:30:12 -0700 In-Reply-To: <20211102073014.2901870-1-davidgow@google.com> Message-Id: <20211102073014.2901870-2-davidgow@google.com> Mime-Version: 1.0 References: <20211102073014.2901870-1-davidgow@google.com> X-Mailer: git-send-email 2.33.1.1089.g2158813163f-goog Subject: [PATCH v4 2/4] kunit: tool: Report an error if any test has no subtests From: David Gow To: Brendan Higgins , Rae Moar , Daniel Latypov , Shuah Khan Cc: David Gow , kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org It's possible for a test to have a subtest header, but zero valid subtests. We used to error on this if the test plan had no subtests listed, but it's possible to have subtests without a test plan (indeed, this is how parameterised tests work). Tests with 0 subtests now have the result NO_TESTS, and will report an error (which does not halt test execution, but is printed in a scary red colour and is noted in the results summary). Signed-off-by: David Gow Reviewed-by: Daniel Latypov Reviewed-by: Brendan Higgins --- Changes since v3: https://lore.kernel.org/linux-kselftest/20211028064154.2301049-2-davidgow@google.com/ - Split up and expanded the tests, as suggested by Daniel. Changes since v2: https://lore.kernel.org/linux-kselftest/20211027013702.2039566-2-davidgow@google.com/ - Report NO_TESTS as '[NO TESTS RUN]' in yellow, instead of '[FAILED]' in red, particularly since it doesn't get counted as a failure. tools/testing/kunit/kunit_parser.py | 16 +++++++++++----- tools/testing/kunit/kunit_tool_test.py | 12 ++++++++++++ .../test_is_test_passed-no_tests_no_plan.log | 7 +++++++ 3 files changed, 30 insertions(+), 5 deletions(-) create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 50ded55c168c..68c847e8ca58 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -360,9 +360,6 @@ def parse_test_plan(lines: LineStream, test: Test) -> bool: test.log.append(lines.pop()) expected_count = int(match.group(1)) test.expected_count = expected_count - if expected_count == 0: - test.status = TestStatus.NO_TESTS - test.add_error('0 tests run!') return True TEST_RESULT = re.compile(r'^(ok|not ok) ([0-9]+) (- )?([^#]*)( # .*)?$') @@ -589,6 +586,8 @@ def format_test_result(test: Test) -> str: return (green('[PASSED] ') + test.name) elif test.status == TestStatus.SKIPPED: return (yellow('[SKIPPED] ') + test.name) + elif test.status == TestStatus.NO_TESTS: + return (yellow('[NO TESTS RUN] ') + test.name) elif test.status == TestStatus.TEST_CRASHED: print_log(test.log) return (red('[CRASHED] ') + test.name) @@ -731,6 +730,7 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: # test plan test.name = "main" parse_test_plan(lines, test) + parent_test = True else: # If KTAP/TAP header is not found, test must be subtest # header or test result line so parse attempt to parser @@ -744,7 +744,7 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: expected_count = test.expected_count subtests = [] test_num = 1 - while expected_count is None or test_num <= expected_count: + while parent_test and (expected_count is None or test_num <= expected_count): # Loop to parse any subtests. # Break after parsing expected number of tests or # if expected number of tests is unknown break when test @@ -779,9 +779,15 @@ def parse_test(lines: LineStream, expected_num: int, log: List[str]) -> Test: parse_test_result(lines, test, expected_num) else: test.add_error('missing subtest result line!') + + # Check for there being no tests + if parent_test and len(subtests) == 0: + test.status = TestStatus.NO_TESTS + test.add_error('0 tests run!') + # Add statuses to TestCounts attribute in Test object bubble_up_test_results(test) - if parent_test: + if parent_test and not main: # If test has subtests and is not the main test object, print # footer. print_test_footer(test) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index bc8793145713..9de2072089e6 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -209,6 +209,18 @@ class KUnitParserTest(unittest.TestCase): kunit_parser.TestStatus.NO_TESTS, result.status) + def test_no_tests_no_plan(self): + no_plan_log = test_data_path('test_is_test_passed-no_tests_no_plan.log') + with open(no_plan_log) as file: + result = kunit_parser.parse_run_tests( + kunit_parser.extract_tap_lines(file.readlines())) + self.assertEqual(0, len(result.test.subtests[0].subtests[0].subtests)) + self.assertEqual( + kunit_parser.TestStatus.NO_TESTS, + result.test.subtests[0].subtests[0].status) + self.assertEqual(1, result.test.counts.errors) + + def test_no_kunit_output(self): crash_log = test_data_path('test_insufficient_memory.log') print_mock = mock.patch('builtins.print').start() diff --git a/tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log b/tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log new file mode 100644 index 000000000000..dd873c981108 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-no_tests_no_plan.log @@ -0,0 +1,7 @@ +TAP version 14 +1..1 + # Subtest: suite + 1..1 + # Subtest: case + ok 1 - case # SKIP +ok 1 - suite From patchwork Tue Nov 2 07:30:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gow X-Patchwork-Id: 12598011 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA8A4C433FE for ; Tue, 2 Nov 2021 07:30:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CEF9360F56 for ; Tue, 2 Nov 2021 07:30:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230505AbhKBHdB (ORCPT ); Tue, 2 Nov 2021 03:33:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230407AbhKBHc7 (ORCPT ); Tue, 2 Nov 2021 03:32:59 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 65992C061714 for ; Tue, 2 Nov 2021 00:30:25 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id b15-20020a25ae8f000000b005c20f367790so16095616ybj.2 for ; Tue, 02 Nov 2021 00:30:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=rMM4H9fUIwM33LxbKfiXhTHqgKYzeuUYfM1psLr0v/w=; b=G16Rx7rkvHlwETek+hMCcd3t1Cx51mQJkr83tl3HPZ06497blrWEDCZIJjwKf4neqM wV2Bc8jeUlPFUoGQmhGRap3uqA68vQfigDW8QCq81iQ7vrLaXeXTrVppam6wqFWmqzL8 1iB/xcKK934S7rtEzTja0wcB57FyA8i9wwcHyo+VTA2YXcC1kK/3KnQaDziU6u6wo53a /dXzwys4UpQMPfyRs0QE8nz/oHDoIyC3/nD++0rA06QXidVSaLzeIMYZ/lHVdOKyeKSV M++I2MTmoesPYTXm0pTriO41YlMRyxM2sTRRDCgczHrD2Ba82tAw6yCQMnN/4nK52Wmq KB9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=rMM4H9fUIwM33LxbKfiXhTHqgKYzeuUYfM1psLr0v/w=; b=u+4m11IShbdcSGolSNnKH0m3qKCOu/FfR/fcliZjbq+gPXF5lKQbQI0dv0tcfSw5eY Qx/d49b6yHt3pXSZ6X/+9IT2hgdDNJBv5+wxpWbqgbd1sgK8Xb1D4WibmwqKbvBKEK6j 7nuAe4g/CnbdDGf+gjAILQD2fw3KeiYdyIzoLOKwgkIM4PTVoVOuMBQSPtzUCmbW5ST+ v5IfBBWl0n+NgtzMwzWMRwUZeAdPZ2Gk7xZIPqMK0i5NnKLfqdDWePve/vzX5Txr87DF MP/MncYauulKWe7uO/t2bOfHjVe0pnD2ychlX8DAjbSxiQ0kgRhV2mZSHPCOLU5zDz7n xavQ== X-Gm-Message-State: AOAM533GzdMf2EPswhHmJKeSD1vFRMdZ0/Kjh9/TiqDotQ9Nc1giA8nE jHnd8AYV+O1QLpa2/qRzYSiyv0xy4MAKNA== X-Google-Smtp-Source: ABdhPJyfD8BOlx6vem1NevM/1wKS3tkYqYiqQw/HAnt/slCtW5+A8FyLpwguN04q7xVZgIs8meUEleYFFxELUQ== X-Received: from spirogrip.svl.corp.google.com ([2620:15c:2cb:201:524f:74d8:b474:2517]) (user=davidgow job=sendgmr) by 2002:a05:6902:568:: with SMTP id a8mr19569204ybt.472.1635838224681; Tue, 02 Nov 2021 00:30:24 -0700 (PDT) Date: Tue, 2 Nov 2021 00:30:13 -0700 In-Reply-To: <20211102073014.2901870-1-davidgow@google.com> Message-Id: <20211102073014.2901870-3-davidgow@google.com> Mime-Version: 1.0 References: <20211102073014.2901870-1-davidgow@google.com> X-Mailer: git-send-email 2.33.1.1089.g2158813163f-goog Subject: [PATCH v4 3/4] kunit: Don't crash if no parameters are generated From: David Gow To: Brendan Higgins , Rae Moar , Daniel Latypov , Shuah Khan Cc: David Gow , kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org It's possible that a parameterised test could end up with zero parameters. At the moment, the test function will nevertheless be called with NULL as the parameter. Instead, don't try to run the test code, and just mark the test as SKIPped. Reported-by: Daniel Latypov Signed-off-by: David Gow Reviewed-by: Daniel Latypov Reviewed-by: Brendan Higgins --- Changes since v3: https://lore.kernel.org/linux-kselftest/20211028064154.2301049-3-davidgow@google.com/ - Fix a change which should've been in patch 4. - Reverse the order of the if conditional: handle non-parameterised case first. Changes since v2: https://lore.kernel.org/linux-kselftest/20211027013702.2039566-3-davidgow@google.com/ - Rework to not share the loop between the parameterised and non-parameterised test cases. - Suggested by Daniel Latypov. - Avoids using a magic non-zero pointer value. lib/kunit/test.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 3bd741e50a2d..f96498ede2cc 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -504,16 +504,18 @@ int kunit_run_tests(struct kunit_suite *suite) struct kunit_result_stats param_stats = { 0 }; test_case->status = KUNIT_SKIPPED; - if (test_case->generate_params) { + if (!test_case->generate_params) { + /* Non-parameterised test. */ + kunit_run_case_catch_errors(suite, test_case, &test); + kunit_update_stats(¶m_stats, test.status); + } else { /* Get initial param. */ param_desc[0] = '\0'; test.param_value = test_case->generate_params(NULL, param_desc); - } - do { - kunit_run_case_catch_errors(suite, test_case, &test); + while (test.param_value) { + kunit_run_case_catch_errors(suite, test_case, &test); - if (test_case->generate_params) { if (param_desc[0] == '\0') { snprintf(param_desc, sizeof(param_desc), "param-%d", test.param_index); @@ -530,11 +532,11 @@ int kunit_run_tests(struct kunit_suite *suite) param_desc[0] = '\0'; test.param_value = test_case->generate_params(test.param_value, param_desc); test.param_index++; - } - kunit_update_stats(¶m_stats, test.status); + kunit_update_stats(¶m_stats, test.status); + } + } - } while (test.param_value); kunit_print_test_stats(&test, param_stats); From patchwork Tue Nov 2 07:30:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gow X-Patchwork-Id: 12598013 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF736C433EF for ; Tue, 2 Nov 2021 07:30:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 966C060F02 for ; Tue, 2 Nov 2021 07:30:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230509AbhKBHdD (ORCPT ); Tue, 2 Nov 2021 03:33:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230407AbhKBHdC (ORCPT ); Tue, 2 Nov 2021 03:33:02 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC65BC061764 for ; Tue, 2 Nov 2021 00:30:27 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id f92-20020a25a465000000b005bea37bc0baso29939299ybi.5 for ; Tue, 02 Nov 2021 00:30:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ccic3rr4mdoxbdec06AcmZnoBxRewK3GJUdYuoNnen4=; b=D1CekfftDJe+rtDY9HWYJwucEzP5l6K0C+HneWxHOD3zepmQR/GMV0yegg0KuY31Ji GQXdWGwrGqstc0vOw6oclM/e2Xex53BWfvqSC3qZu13RsITXL2u30USINCOctSICRnX/ Cl/U8dMEvnAexLUEOQAPjxc6Tv3oI4UOkyqFNzC+FcnjUbZeP4/HE/Pr9Hq9YwRE3Jtt IkEP9N2Km/wgmhb+F/FZgss/x1hI1+e/kZzUAjKMRIDZW2WeuAZGWc/VCgjNXajEhOiF HOfzefb36NHsYW/KrKomnusQE9KBx+NcWAwlQb5Z1rwNWDJmkvrZ59Pes+sfYyGYu5Ia DJxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ccic3rr4mdoxbdec06AcmZnoBxRewK3GJUdYuoNnen4=; b=xokNXmp6SJ9CcXFYyZgNPtrMzy6ytoa0zC/JjalkOEqk1htMXUiomQR7uOOnmGZj2I 5Kew7swUMghsnlA6fq4D8TaiYpGf3eA3XbHvcY+tKC/Vch0q3EIYrWt1BWKy1GZKuk+9 hPqWGuMd0jsYno6KX283WECZ6J+0WEa3ASc4/7GvfX4XM072HN/K1uqklJk6A373jysk AJlNyenRKAHS5ZyTjXFCympZHDzvV8twBXmwt6WQ5NtWgB4MtoXC5iQ6jsP7D4plVRLt TmjneET6e8eLls7+EcZlU/TR93gXWbCIFAI5ymJDy3Aqch8qWRgPzQw+0hb+vF4cD3hK Cg+Q== X-Gm-Message-State: AOAM533ash8mLBdJFfYiiXRi/qlI+Tfr8pnLF0pPQIvLl2xMr3sg+VVp fIDUJZh0GMu1dI88bjklQSyGdrXH8bM65g== X-Google-Smtp-Source: ABdhPJy1s2Yko/ka6XxoOb8uiAt2VfEYKhvlH9ZvKrjTWRKBwfamluyKBPgySQrqEVYvpT0H6OaQioq+vPtKwg== X-Received: from spirogrip.svl.corp.google.com ([2620:15c:2cb:201:524f:74d8:b474:2517]) (user=davidgow job=sendgmr) by 2002:a25:cc07:: with SMTP id l7mr26331915ybf.313.1635838226959; Tue, 02 Nov 2021 00:30:26 -0700 (PDT) Date: Tue, 2 Nov 2021 00:30:14 -0700 In-Reply-To: <20211102073014.2901870-1-davidgow@google.com> Message-Id: <20211102073014.2901870-4-davidgow@google.com> Mime-Version: 1.0 References: <20211102073014.2901870-1-davidgow@google.com> X-Mailer: git-send-email 2.33.1.1089.g2158813163f-goog Subject: [PATCH v4 4/4] kunit: Report test parameter results as (K)TAP subtests From: David Gow To: Brendan Higgins , Rae Moar , Daniel Latypov , Shuah Khan Cc: David Gow , kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Currently, the results for individial parameters in a parameterised test are simply output as (K)TAP diagnostic lines. As kunit_tool now supports nested subtests, report each parameter as its own subtest. For example, here's what the output now looks like: # Subtest: inode_test_xtimestamp_decoding ok 1 - 1901-12-13 Lower bound of 32bit < 0 timestamp, no extra bits ok 2 - 1969-12-31 Upper bound of 32bit < 0 timestamp, no extra bits ok 3 - 1970-01-01 Lower bound of 32bit >=0 timestamp, no extra bits ok 4 - 2038-01-19 Upper bound of 32bit >=0 timestamp, no extra bits ok 5 - 2038-01-19 Lower bound of 32bit <0 timestamp, lo extra sec bit on ok 6 - 2106-02-07 Upper bound of 32bit <0 timestamp, lo extra sec bit on ok 7 - 2106-02-07 Lower bound of 32bit >=0 timestamp, lo extra sec bit on ok 8 - 2174-02-25 Upper bound of 32bit >=0 timestamp, lo extra sec bit on ok 9 - 2174-02-25 Lower bound of 32bit <0 timestamp, hi extra sec bit on ok 10 - 2242-03-16 Upper bound of 32bit <0 timestamp, hi extra sec bit on ok 11 - 2242-03-16 Lower bound of 32bit >=0 timestamp, hi extra sec bit on ok 12 - 2310-04-04 Upper bound of 32bit >=0 timestamp, hi extra sec bit on ok 13 - 2310-04-04 Upper bound of 32bit>=0 timestamp, hi extra sec bit 1. 1 ns ok 14 - 2378-04-22 Lower bound of 32bit>= timestamp. Extra sec bits 1. Max ns ok 15 - 2378-04-22 Lower bound of 32bit >=0 timestamp. All extra sec bits on ok 16 - 2446-05-10 Upper bound of 32bit >=0 timestamp. All extra sec bits on # inode_test_xtimestamp_decoding: pass:16 fail:0 skip:0 total:16 ok 1 - inode_test_xtimestamp_decoding Signed-off-by: David Gow Reviewed-by: Daniel Latypov Reviewed-by: Brendan Higgins --- Changes since v3: https://lore.kernel.org/linux-kselftest/20211028064154.2301049-4-davidgow@google.com/ - Fix the missing log line which ended up in patch 3 by mistake. Changes since v2: https://lore.kernel.org/linux-kselftest/20211027013702.2039566-4-davidgow@google.com/ - No changes to this patch. lib/kunit/test.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/kunit/test.c b/lib/kunit/test.c index f96498ede2cc..c7ed4aabec04 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -512,6 +512,8 @@ int kunit_run_tests(struct kunit_suite *suite) /* Get initial param. */ param_desc[0] = '\0'; test.param_value = test_case->generate_params(NULL, param_desc); + kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT + "# Subtest: %s", test_case->name); while (test.param_value) { kunit_run_case_catch_errors(suite, test_case, &test); @@ -522,9 +524,8 @@ int kunit_run_tests(struct kunit_suite *suite) } kunit_log(KERN_INFO, &test, - KUNIT_SUBTEST_INDENT - "# %s: %s %d - %s", - test_case->name, + KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT + "%s %d - %s", kunit_status_to_ok_not_ok(test.status), test.param_index + 1, param_desc);