From patchwork Sat Jun 10 00:51:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 13274640 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC4FFC7EE2E for ; Sat, 10 Jun 2023 00:52:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229480AbjFJAwP (ORCPT ); Fri, 9 Jun 2023 20:52:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52248 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229647AbjFJAwM (ORCPT ); Fri, 9 Jun 2023 20:52:12 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA066E1 for ; Fri, 9 Jun 2023 17:52:09 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-568ab5c813eso33366167b3.2 for ; Fri, 09 Jun 2023 17:52:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686358329; x=1688950329; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ylzIdUa/HdrZ45yd+RRq5LnH+hOmWvIma/I97GpGtAs=; b=Zmr+EkhSeSNvCLPC3fmn2tX9mMT6rgh3uQ/IJsxW1MCezHgJiQdRtFwdXdki72dVD5 CXjnrDMiJEXw566GTSbcLVdlvJmN0o4vXeFJm94iDce0N11wsaJRsjgjDQ9jt1oMOM1N Atyk9QAqumY9ETSWsPdP0GGEPInAebRX8DFrjCTVHtfp2K3le9vulryAW+tjZwfiCl2l tCSrakVDNP/LGpZxVVoNL7MIRcU4bnQOvwNkusbHS6wLTOcSJBgYigUfekT9aXn8yQes Mx9d+Mna3RA/iTXw9GtQS+7tQvd63vIkJTP0UvjgIkbvBSfi0a1jgfk3zyaxGX0koNbO QEtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686358329; x=1688950329; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ylzIdUa/HdrZ45yd+RRq5LnH+hOmWvIma/I97GpGtAs=; b=Ti3EdEbckV0AovNBl0IIlGc3eAJpGid8WtoMFVn9T+Cf/1A9r9D32e8h66xAnsHxBJ wiv0o/uzDycn2KdsHem7oiHNaKHi8w93ph0i1IFE8s0yOikBDkmDWbXE+XBdBwYqUb4d QSRYCIOOh9KS0cOSOhOLpgTZ0tmCRz51zAffLj5/ZbNWuv+VGW08Y3rInycbJatrxEzR Lg2BxLT3Yy/HCYdIpsxc0XQF3rHkj3tldXJj34/xHF5RDvOv4rywnG5PGLMRwLv2jpmv Ya4Ec1+PL/7gVjJpY3WG26j3kxnj3s4/gACaWFJpjViinb9ll2jfB72uYQpfhgkN5TOA ZlMw== X-Gm-Message-State: AC+VfDx4VJQQtnZEEyXki7+/lOQdkpBcoXFzr9pIm/Hno4WRrFAKOvAH /KhNZBHRNx+Iq/xGSqMM87jhP9xlmw== X-Google-Smtp-Source: ACHHUZ6aVbU/xiu8xxQnCIprN1NVtQfosUsTiEtzBVSYhi87pylg/c24Jte89T8fkhjvtHjKHtPnmUIxIw== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a81:8d0d:0:b0:569:8603:577 with SMTP id d13-20020a818d0d000000b0056986030577mr1681187ywg.4.1686358328849; Fri, 09 Jun 2023 17:52:08 -0700 (PDT) Date: Sat, 10 Jun 2023 00:51:44 +0000 In-Reply-To: <20230610005149.1145665-1-rmoar@google.com> Mime-Version: 1.0 References: <20230610005149.1145665-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230610005149.1145665-2-rmoar@google.com> Subject: [RFC v1 1/6] kunit: Add test attributes API structure From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Add the basic structure of the test attribute API to KUnit, which can be used to save and access test associated data. Add attributes.c and attributes.h to hold associated structs and functions for the API. Create a struct that holds a variety of associated helper functions for each test attribute. These helper functions will be used to get the attribute value, convert the value to a string, and filter based on the value. This struct is flexible by design to allow for attributes of numerous types and contexts. Add a method to print test attributes in the format of "# [.]: ". Example for a suite: "# speed: slow" Example for a test case: "# test_case.speed: very_slow" Use this method to report attributes in the KTAP output and _list_tests output. In test.h, add fields and associated helper functions to test cases and suites to hold user-inputted test attributes. Signed-off-by: Rae Moar --- include/kunit/attributes.h | 19 +++++++++++ include/kunit/test.h | 33 +++++++++++++++++++ lib/kunit/Makefile | 3 +- lib/kunit/attributes.c | 65 ++++++++++++++++++++++++++++++++++++++ lib/kunit/executor.c | 10 +++++- lib/kunit/test.c | 17 ++++++---- 6 files changed, 138 insertions(+), 9 deletions(-) create mode 100644 include/kunit/attributes.h create mode 100644 lib/kunit/attributes.c diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h new file mode 100644 index 000000000000..9fcd184cce36 --- /dev/null +++ b/include/kunit/attributes.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * KUnit API to save and access test attributes + * + * Copyright (C) 2023, Google LLC. + * Author: Rae Moar + */ + +#ifndef _KUNIT_ATTRIBUTES_H +#define _KUNIT_ATTRIBUTES_H + +/* + * Print all test attributes for a test case or suite. + * Output format for test cases: "# .: " + * Output format for test suites: "# : " + */ +void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level); + +#endif /* _KUNIT_ATTRIBUTES_H */ diff --git a/include/kunit/test.h b/include/kunit/test.h index 23120d50499e..1fc9155988e9 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -63,12 +63,16 @@ enum kunit_status { KUNIT_SKIPPED, }; +/* Holds attributes for each test case and suite */ +struct kunit_attributes {}; + /** * struct kunit_case - represents an individual test case. * * @run_case: the function representing the actual test case. * @name: the name of the test case. * @generate_params: the generator function for parameterized tests. + * @attr: the attributes associated with the test * * A test case is a function with the signature, * ``void (*)(struct kunit *)`` @@ -104,6 +108,7 @@ struct kunit_case { void (*run_case)(struct kunit *test); const char *name; const void* (*generate_params)(const void *prev, char *desc); + struct kunit_attributes attr; /* private: internal use only. */ enum kunit_status status; @@ -133,6 +138,18 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status) */ #define KUNIT_CASE(test_name) { .run_case = test_name, .name = #test_name } +/** + * KUNIT_CASE_ATTR - A helper for creating a &struct kunit_case + * with attributes + * + * @test_name: a reference to a test case function. + * @attributes: a reference to a struct kunit_attributes object containing + * test attributes + */ +#define KUNIT_CASE_ATTR(test_name, attributes) \ + { .run_case = test_name, .name = #test_name, \ + .attr = attributes } + /** * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case * @@ -154,6 +171,20 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status) { .run_case = test_name, .name = #test_name, \ .generate_params = gen_params } +/** + * KUNIT_CASE_PARAM_ATTR - A helper for creating a parameterized &struct + * kunit_case with attributes + * + * @test_name: a reference to a test case function. + * @gen_params: a reference to a parameter generator function. + * @attributes: a reference to a struct kunit_attributes object containing + * test attributes + */ +#define KUNIT_CASE_PARAM_ATTR(test_name, gen_params, attributes) \ + { .run_case = test_name, .name = #test_name, \ + .generate_params = gen_params, \ + .attr = attributes } + /** * struct kunit_suite - describes a related collection of &struct kunit_case * @@ -163,6 +194,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status) * @init: called before every test case. * @exit: called after every test case. * @test_cases: a null terminated array of test cases. + * @attr: the attributes associated with the test suite * * A kunit_suite is a collection of related &struct kunit_case s, such that * @init is called before every test case and @exit is called after every @@ -182,6 +214,7 @@ struct kunit_suite { int (*init)(struct kunit *test); void (*exit)(struct kunit *test); struct kunit_case *test_cases; + struct kunit_attributes attr; /* private: internal use only */ char status_comment[KUNIT_STATUS_COMMENT_SIZE]; diff --git a/lib/kunit/Makefile b/lib/kunit/Makefile index cb417f504996..46f75f23dfe4 100644 --- a/lib/kunit/Makefile +++ b/lib/kunit/Makefile @@ -6,7 +6,8 @@ kunit-objs += test.o \ string-stream.o \ assert.o \ try-catch.o \ - executor.o + executor.o \ + attributes.o ifeq ($(CONFIG_KUNIT_DEBUGFS),y) kunit-objs += debugfs.o diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c new file mode 100644 index 000000000000..0ea641be795f --- /dev/null +++ b/lib/kunit/attributes.c @@ -0,0 +1,65 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * KUnit API to save and access test attributes + * + * Copyright (C) 2023, Google LLC. + * Author: Rae Moar + */ + +#include +#include + +/** + * struct kunit_attr - represents a test attribute and holds flexible + * helper functions to interact with attribute. + * + * @name: name of test attribute, eg. speed + * @get_attr: function to return attribute value given a test + * @to_string: function to return string representation of given + * attribute value + * @filter: function to indicate whether a given attribute value passes a + * filter + */ +struct kunit_attr { + const char *name; + void *(*get_attr)(void *test_or_suite, bool is_test); + const char *(*to_string)(void *attr, bool *to_free); + int (*filter)(void *attr, const char *input, int *err); + void *attr_default; +}; + +/* List of all Test Attributes */ + +static struct kunit_attr kunit_attr_list[1] = {}; + +/* Helper Functions to Access Attributes */ + +void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level) +{ + int i; + bool to_free; + void *attr; + const char *attr_name, *attr_str; + struct kunit_suite *suite = is_test ? NULL : test_or_suite; + struct kunit_case *test = is_test ? test_or_suite : NULL; + + for (i = 0; i < ARRAY_SIZE(kunit_attr_list); i++) { + attr = kunit_attr_list[i].get_attr(test_or_suite, is_test); + if (attr) { + attr_name = kunit_attr_list[i].name; + attr_str = kunit_attr_list[i].to_string(attr, &to_free); + if (test) { + kunit_log(KERN_INFO, test, "%*s# %s.%s: %s", + KUNIT_INDENT_LEN * test_level, "", test->name, + attr_name, attr_str); + } else { + kunit_log(KERN_INFO, suite, "%*s# %s: %s", + KUNIT_INDENT_LEN * test_level, "", attr_name, attr_str); + } + + /* Free to_string of attribute if needed */ + if (to_free) + kfree(attr_str); + } + } +} diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c index 74982b83707c..767a84e32f06 100644 --- a/lib/kunit/executor.c +++ b/lib/kunit/executor.c @@ -2,6 +2,7 @@ #include #include +#include #include #include @@ -180,10 +181,17 @@ static void kunit_exec_list_tests(struct suite_set *suite_set) /* Hack: print a ktap header so kunit.py can find the start of KUnit output. */ pr_info("KTAP version 1\n"); - for (suites = suite_set->start; suites < suite_set->end; suites++) + for (suites = suite_set->start; suites < suite_set->end; suites++) { + /* Print suite name and suite attributes */ + pr_info("%s\n", (*suites)->name); + kunit_print_attr((void *)(*suites), false, 0); + + /* Print test case name and attributes in suite */ kunit_suite_for_each_test_case((*suites), test_case) { pr_info("%s.%s\n", (*suites)->name, test_case->name); + kunit_print_attr((void *)test_case, true, 0); } + } } int kunit_run_all_tests(void) diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 84e4666555c9..9ee55139ecd1 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -168,6 +169,13 @@ size_t kunit_suite_num_test_cases(struct kunit_suite *suite) } EXPORT_SYMBOL_GPL(kunit_suite_num_test_cases); +/* Currently supported test levels */ +enum { + KUNIT_LEVEL_SUITE = 0, + KUNIT_LEVEL_CASE, + KUNIT_LEVEL_CASE_PARAM, +}; + static void kunit_print_suite_start(struct kunit_suite *suite) { /* @@ -181,17 +189,11 @@ static void kunit_print_suite_start(struct kunit_suite *suite) pr_info(KUNIT_SUBTEST_INDENT "KTAP version 1\n"); pr_info(KUNIT_SUBTEST_INDENT "# Subtest: %s\n", suite->name); + kunit_print_attr((void *)suite, false, KUNIT_LEVEL_CASE); pr_info(KUNIT_SUBTEST_INDENT "1..%zd\n", kunit_suite_num_test_cases(suite)); } -/* Currently supported test levels */ -enum { - KUNIT_LEVEL_SUITE = 0, - KUNIT_LEVEL_CASE, - KUNIT_LEVEL_CASE_PARAM, -}; - static void kunit_print_ok_not_ok(struct kunit *test, unsigned int test_level, enum kunit_status status, @@ -651,6 +653,7 @@ int kunit_run_tests(struct kunit_suite *suite) } } + kunit_print_attr((void *)test_case, true, KUNIT_LEVEL_CASE); kunit_print_test_stats(&test, param_stats); From patchwork Sat Jun 10 00:51:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 13274641 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 706CDC7EE25 for ; Sat, 10 Jun 2023 00:52:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232413AbjFJAwQ (ORCPT ); Fri, 9 Jun 2023 20:52:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52254 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230056AbjFJAwN (ORCPT ); Fri, 9 Jun 2023 20:52:13 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54C791730 for ; Fri, 9 Jun 2023 17:52:11 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-569fee67d9dso51178527b3.1 for ; Fri, 09 Jun 2023 17:52:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686358330; x=1688950330; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kg21mZWTzPOwmlFSsLi8itlal2bzL2xDdn1UhMA3w20=; b=WswYlHq6/ZHkfHrmbeb7arHsRL5XXKC8w3ETG7ZaFzcMXt+WpH/QTKK+jz6PoG+QBq uSLDTqnyS2MGajKMA2iGh6RvicQOPQNCY9hKFYLRHors5KSEWpXfWHeYr5OioTJ9AZaR g0DySrOCSRUfojMwgtpCy9mXPqIxoJEepKcqLZhGTnqA3PQCrSwPR5Hsc5RogDCN6I7w l+YR05Q6aB93INFQnuCul5Nrk+j1ZgV0B8doglxFpMexoWkNgWPW74Kq2jJ5bEy+eIh7 b6O8jeGG3jRv3NtDrrmAR7MMIAZEBNe5HmVSmu2BiBOyacZdvyi+Qyfe7vjBpuIrhgnd Pl8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686358330; x=1688950330; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kg21mZWTzPOwmlFSsLi8itlal2bzL2xDdn1UhMA3w20=; b=Kfwg/htWk3OqupxZOQtCu9HhQlwoWfLYZkbr4cKzBvf/O0i++L5BybkA7zl6eBQNRN 0UMtdCNl5uKW0dCKzbjn2SXV9a6lpXWVmxswDXA3J1n0TC41WgJklOE1q3AvebbRdONA pT9AbUUFoQSsyGbSwxaXYl9j186T+pLDUZo7V/slsU+XvfcyMxaYWUqbD55d69XblDPZ mOecz6oyYFZqwQa+eFulIpVyXpzSnGsSh7T7ILrH47NWdVVfapDMElUd6agWTPXfz5rC iBJ3QLGQmUA0LTHaam53hY15m22kbomIThkxyGBmXa5Ntmv1q+gdOWk8KQ20nkzwtK2i 2gMQ== X-Gm-Message-State: AC+VfDxy0Y2GfX3pCjEZFAIn0ObroQfnd9GdmeaTYM4E5FRsIrrmKvdI ttad/aalob6UH7tI9Hfp4HVO2KdUdw== X-Google-Smtp-Source: ACHHUZ7RszgYDSykFAHyd5I6mFbzO6p8gosaq49TdYgyFazuZ9cddVRnvGvD+puCkZo6ToTL3ooh2Fefzw== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a05:690c:c01:b0:565:d5f6:a390 with SMTP id cl1-20020a05690c0c0100b00565d5f6a390mr2450767ywb.2.1686358330655; Fri, 09 Jun 2023 17:52:10 -0700 (PDT) Date: Sat, 10 Jun 2023 00:51:45 +0000 In-Reply-To: <20230610005149.1145665-1-rmoar@google.com> Mime-Version: 1.0 References: <20230610005149.1145665-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230610005149.1145665-3-rmoar@google.com> Subject: [RFC v1 2/6] kunit: Add speed attribute From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Add speed attribute to the test attribute API. This attribute will allow users to mark tests with a category of speed. Currently the categories of speed proposed are: fast, normal, slow, and very_slow. These are outlined in the enum kunit_speed. Note the speed attribute can also be left as unset and then, will act as the default which is "normal", during filtering. Note speed is intended to be marked based on relative speeds rather than quantitative speeds of KUnit tests. This is because tests may run on various architectures at different speeds. Add the macro KUNIT_CASE_SLOW to set a test as slow, as this is likely a common use of the attributes API. Add an example of marking a slow test to kunit-example-test.c. Signed-off-by: Rae Moar --- include/kunit/test.h | 31 ++++++++++++++++++++++- lib/kunit/attributes.c | 45 +++++++++++++++++++++++++++++++++- lib/kunit/kunit-example-test.c | 9 +++++++ 3 files changed, 83 insertions(+), 2 deletions(-) diff --git a/include/kunit/test.h b/include/kunit/test.h index 1fc9155988e9..3d684723ae57 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -63,8 +63,26 @@ enum kunit_status { KUNIT_SKIPPED, }; +/* Attribute struct/enum definitions */ + +/* + * Speed Attribute is stored as an enum and separated into categories of + * speed: very_slowm, slow, normal, and fast. These speeds are relative + * to other KUnit tests. + */ +enum kunit_speed { + KUNIT_SPEED_UNSET, + KUNIT_SPEED_VERY_SLOW, + KUNIT_SPEED_SLOW, + KUNIT_SPEED_NORMAL, + KUNIT_SPEED_FAST, + KUNIT_SPEED_MAX = KUNIT_SPEED_FAST, +}; + /* Holds attributes for each test case and suite */ -struct kunit_attributes {}; +struct kunit_attributes { + enum kunit_speed speed; +}; /** * struct kunit_case - represents an individual test case. @@ -150,6 +168,17 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status) { .run_case = test_name, .name = #test_name, \ .attr = attributes } +/** + * KUNIT_CASE_SLOW - A helper for creating a &struct kunit_case + * with the slow attribute + * + * @test_name: a reference to a test case function. + */ + +#define KUNIT_CASE_SLOW(test_name) \ + { .run_case = test_name, .name = #test_name, \ + .attr.speed = KUNIT_SPEED_SLOW } + /** * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case * diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c index 0ea641be795f..e17889f94693 100644 --- a/lib/kunit/attributes.c +++ b/lib/kunit/attributes.c @@ -28,9 +28,52 @@ struct kunit_attr { void *attr_default; }; +/* String Lists for enum Attributes */ + +static const char * const speed_str_list[] = {"unset", "very_slow", "slow", "normal", "fast"}; + +/* To String Methods */ + +static const char *attr_enum_to_string(void *attr, const char * const str_list[], bool *to_free) +{ + long val = (long)attr; + + *to_free = false; + if (!val) + return NULL; + return str_list[val]; +} + +static const char *attr_speed_to_string(void *attr, bool *to_free) +{ + return attr_enum_to_string(attr, speed_str_list, to_free); +} + +/* Get Attribute Methods */ + +static void *attr_speed_get(void *test_or_suite, bool is_test) +{ + struct kunit_suite *suite = is_test ? NULL : test_or_suite; + struct kunit_case *test = is_test ? test_or_suite : NULL; + + if (test) + return ((void *) test->attr.speed); + else + return ((void *) suite->attr.speed); +} + +/* Attribute Struct Definitions */ + +static const struct kunit_attr speed_attr = { + .name = "speed", + .get_attr = attr_speed_get, + .to_string = attr_speed_to_string, + .attr_default = (void *)KUNIT_SPEED_NORMAL, +}; + /* List of all Test Attributes */ -static struct kunit_attr kunit_attr_list[1] = {}; +static struct kunit_attr kunit_attr_list[1] = {speed_attr}; /* Helper Functions to Access Attributes */ diff --git a/lib/kunit/kunit-example-test.c b/lib/kunit/kunit-example-test.c index b69b689ea850..01a769f35e1d 100644 --- a/lib/kunit/kunit-example-test.c +++ b/lib/kunit/kunit-example-test.c @@ -220,6 +220,14 @@ static void example_params_test(struct kunit *test) KUNIT_EXPECT_EQ(test, param->value % param->value, 0); } +/* + * This test should always pass. Can be used to practice filtering attributes. + */ +static void example_slow_test(struct kunit *test) +{ + KUNIT_EXPECT_EQ(test, 1 + 1, 2); +} + /* * Here we make a list of all the test cases we want to add to the test suite * below. @@ -237,6 +245,7 @@ static struct kunit_case example_test_cases[] = { KUNIT_CASE(example_all_expect_macros_test), KUNIT_CASE(example_static_stub_test), KUNIT_CASE_PARAM(example_params_test, example_gen_params), + KUNIT_CASE_SLOW(example_slow_test), {} }; From patchwork Sat Jun 10 00:51:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 13274642 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 792E7C8300C for ; Sat, 10 Jun 2023 00:52:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232462AbjFJAwR (ORCPT ); Fri, 9 Jun 2023 20:52:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232349AbjFJAwP (ORCPT ); Fri, 9 Jun 2023 20:52:15 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9690A2717 for ; Fri, 9 Jun 2023 17:52:13 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5692be06cb2so30571687b3.1 for ; Fri, 09 Jun 2023 17:52:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686358332; x=1688950332; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JD9VPXNmaoVBy2gAiYJtNq7wx8MSqrStxnM3XpRxKVA=; b=q21OucQNW4R7TYegSZEEbLLbvxXRHbQGDJ96QH08pyz1J6KQBPHqxc1IazXONp2w81 OM13ovDeJ5G/8Tm1+3t3jdOXgvDr8ol21ow3CnsdVYDIm75MWMPI76HnLsXVgbNQJ+RL rfWba9qn8no74J6QglcrmJCijppFt3g91rp8zDIT7RWkTz6TuGyLn/aMaahbT/GQdx5c RPLvfDR+QI1tRm+v5qLnkRQCVCdNow8zhlZ4xozUhyfzfoiEz2ElIzYcD9+dh5aLjSvV zGAeVaXrxnCxqrEwmdW1abucmorAsp76hmagmxJSeMltB73RVuVr7Zn06wBU/ZLOPj5w lfwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686358332; x=1688950332; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JD9VPXNmaoVBy2gAiYJtNq7wx8MSqrStxnM3XpRxKVA=; b=K91KyIls2ZYsNhSdmc2EtyiPPWAPbwvEZianvjRdr0aLuUuZWbVDgY4o0mjFc04NPr Jdnn5TmPyaXVy1TfloyEJrO7bStX3tL71owqZUAx/4A4HFUVJRH9JRovw17uxfKYX3or QgUJcf4sqkThxJVMrIFEqKCQp6Ih3WrP1EqOEPgkaSyZyjSCW5f/kZrQgIrKn976lF9S S9uEsqP+TBIieaSsUR70WjhaWysTpSTlZV02AzBzLvGraOJLvhxTxfguSnjFv5T8XpPQ kS1e5JHpC5BPnPTmIGnpHiZb/+gyAWt766CAc3Qb8igzuD0GFjo2IElZBX741WlJCp3V 8xiQ== X-Gm-Message-State: AC+VfDyl30xhjO3BZpQr3ZJbVR4QLoVplFZzRRIsX7XLM6XrKuT0JZBT YAiCUbWK++88AxS2lZ+ey21Kyv44qQ== X-Google-Smtp-Source: ACHHUZ5wVXyVqF42eIZPBvG8hcKoYDyK+tpTpA1FekD4DK+IEpUzvBZpSbGfcRJ5leBMjHptgwFvrDbpuA== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a81:af67:0:b0:561:94a8:29c4 with SMTP id x39-20020a81af67000000b0056194a829c4mr1583110ywj.2.1686358332355; Fri, 09 Jun 2023 17:52:12 -0700 (PDT) Date: Sat, 10 Jun 2023 00:51:46 +0000 In-Reply-To: <20230610005149.1145665-1-rmoar@google.com> Mime-Version: 1.0 References: <20230610005149.1145665-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230610005149.1145665-4-rmoar@google.com> Subject: [RFC v1 3/6] kunit: Add ability to filter attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Add filtering of test attributes. Users can filter tests using a module_param_array called "filter". This functionality will be added to kunit.py in the next patch. The filters will be imputed in the format: "" Example: "speed>slow" Operations include: >, <, >=, <=, !=, and =. These operations do not need to act the same for every attribute. Add method to parse inputted filters. Add the process of filtering tests based on attributes. The process of filtering follows these rules: A test case with a set attribute overrides its parent suite's attribute during filtering. Also, if both the test case attribute and suite attribute are unset the test acts as the default attribute value during filtering. Finally, add a "filter" method for the speed attribute to parse and compare enum values of kunit_speed. Signed-off-by: Rae Moar --- include/kunit/attributes.h | 22 +++++ lib/kunit/attributes.c | 172 +++++++++++++++++++++++++++++++++++++ lib/kunit/executor.c | 79 +++++++++++++---- lib/kunit/executor_test.c | 8 +- 4 files changed, 258 insertions(+), 23 deletions(-) diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h index 9fcd184cce36..bca60d1181bb 100644 --- a/include/kunit/attributes.h +++ b/include/kunit/attributes.h @@ -9,6 +9,15 @@ #ifndef _KUNIT_ATTRIBUTES_H #define _KUNIT_ATTRIBUTES_H +/* + * struct kunit_attr_filter - representation of attributes filter with the + * attribute object and string input + */ +struct kunit_attr_filter { + struct kunit_attr *attr; + char *input; +}; + /* * Print all test attributes for a test case or suite. * Output format for test cases: "# .: " @@ -16,4 +25,17 @@ */ void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level); +/* + * Parse attributes filter input and return an object containing the attribute + * object and the string input. + */ +struct kunit_attr_filter kunit_parse_filter_attr(char *input, int *err); + + +/* + * Returns a copy of the suite containing only tests that pass the filter. + */ +struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *const suite, + struct kunit_attr_filter filter, int *err); + #endif /* _KUNIT_ATTRIBUTES_H */ diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c index e17889f94693..4f753a28e4ee 100644 --- a/lib/kunit/attributes.c +++ b/lib/kunit/attributes.c @@ -49,6 +49,66 @@ static const char *attr_speed_to_string(void *attr, bool *to_free) return attr_enum_to_string(attr, speed_str_list, to_free); } +/* Filter Methods */ + +static int int_filter(long val, const char *op, int input, int *err) +{ + if (!strncmp(op, "<=", 2)) + return (val <= input); + else if (!strncmp(op, ">=", 2)) + return (val >= input); + else if (!strncmp(op, "!=", 2)) + return (val != input); + else if (!strncmp(op, ">", 1)) + return (val > input); + else if (!strncmp(op, "<", 1)) + return (val < input); + else if (!strncmp(op, "=", 1)) + return (val == input); + *err = -EINVAL; + pr_err("kunit executor: invalid filter operation: %s\n", op); + return false; +} + +static int attr_enum_filter(void *attr, const char *input, int *err, + const char * const str_list[], int max) +{ + int i, j, input_int; + long test_val = (long)attr; + const char *input_val; + + for (i = 0; input[i]; i++) { + if (!strchr("", input[i])) { + input_val = input + i; + break; + } + } + + if (!input_val) { + *err = -EINVAL; + pr_err("kunit executor: filter operation not found: %s\n", input); + return false; + } + + for (j = 0; j <= max; j++) { + if (!strcmp(input_val, str_list[j])) + input_int = j; + } + + if (!input_int) { + *err = -EINVAL; + pr_err("kunit executor: invalid filter input: %s\n", input); + return false; + } + + return int_filter(test_val, input, input_int, err); +} + +static int attr_speed_filter(void *attr, const char *input, int *err) +{ + return attr_enum_filter(attr, input, err, speed_str_list, KUNIT_SPEED_MAX); +} + /* Get Attribute Methods */ static void *attr_speed_get(void *test_or_suite, bool is_test) @@ -68,6 +128,7 @@ static const struct kunit_attr speed_attr = { .name = "speed", .get_attr = attr_speed_get, .to_string = attr_speed_to_string, + .filter = attr_speed_filter, .attr_default = (void *)KUNIT_SPEED_NORMAL, }; @@ -106,3 +167,114 @@ void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level } } } + +/* Helper Functions to Filter Attributes */ + +struct kunit_attr_filter kunit_parse_filter_attr(char *input, int *err) +{ + struct kunit_attr_filter filter; + int i, j, op_index = 0; + int attr_index = -1; + char op; + + /* Parse input until operation */ + for (i = 0; input[i]; i++) { + if (strchr("<>!=", input[i])) { + op_index = i; + break; + } + if (input[i] == ' ') + break; + } + + if (!op_index) { + *err = -EINVAL; + pr_err("kunit executor: filter operation not found: %s\n", input); + return filter; + } + + op = input[op_index]; + input[op_index] = '\0'; + + /* Find associated kunit_attr object */ + for (j = 0; j < ARRAY_SIZE(kunit_attr_list); j++) { + if (!strcmp(input, kunit_attr_list[j].name)) { + attr_index = j; + break; + } + } + + input[op_index] = op; + filter.input = input + op_index; + + if (attr_index < 0) { + *err = -EINVAL; + pr_err("kunit executor: attribute not found: %s\n", input); + } else { + filter.attr = &kunit_attr_list[attr_index]; + } + + return filter; +} + +struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *const suite, + struct kunit_attr_filter filter, int *err) +{ + int n = 0; + struct kunit_case *filtered, *test_case; + struct kunit_suite *copy; + void *suite_val, *test_val; + bool suite_result, test_result, default_result; + + /* Allocate memory for new copy of suite and list of test cases */ + copy = kmemdup(suite, sizeof(*copy), GFP_KERNEL); + if (!copy) + return ERR_PTR(-ENOMEM); + + kunit_suite_for_each_test_case(suite, test_case) { n++; } + + filtered = kcalloc(n + 1, sizeof(*filtered), GFP_KERNEL); + if (!filtered) { + kfree(copy); + return ERR_PTR(-ENOMEM); + } + + n = 0; + + /* Save filtering result on default value */ + default_result = filter.attr->filter(filter.attr->attr_default, filter.input, err); + + /* Save suite attribute value and filtering result on that value */ + suite_val = filter.attr->get_attr((void *)suite, false); + suite_result = filter.attr->filter(suite_val, filter.input, err); + + /* For each test case, save test case if passes filtering. */ + kunit_suite_for_each_test_case(suite, test_case) { + test_val = filter.attr->get_attr((void *) test_case, true); + test_result = filter.attr->filter(filter.attr->get_attr(test_case, true), + filter.input, err); + /* + * If attribute value of test case is set, filter on that value. + * If not, filter on suite value if set. If not, filter on + * default value. + */ + if (test_val) { + if (test_result) + filtered[n++] = *test_case; + } else if (suite_val) { + if (suite_result) + filtered[n++] = *test_case; + } else if (default_result) { + filtered[n++] = *test_case; + } + } + + if (n == 0) { + kfree(copy); + kfree(filtered); + return NULL; + } + + copy->test_cases = filtered; + return copy; +} diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c index 767a84e32f06..c67657821eec 100644 --- a/lib/kunit/executor.c +++ b/lib/kunit/executor.c @@ -15,8 +15,12 @@ extern struct kunit_suite * const __kunit_suites_end[]; #if IS_BUILTIN(CONFIG_KUNIT) +#define MAX_FILTERS 10 // Limit of number of attribute filters static char *filter_glob_param; static char *action_param; +static int filter_count; +static char *filter_param[MAX_FILTERS]; + module_param_named(filter_glob, filter_glob_param, charp, 0); MODULE_PARM_DESC(filter_glob, @@ -26,15 +30,16 @@ MODULE_PARM_DESC(action, "Changes KUnit executor behavior, valid values are:\n" ": run the tests like normal\n" "'list' to list test names instead of running them.\n"); +module_param_array_named(filter, filter_param, charp, &filter_count, 0); /* glob_match() needs NULL terminated strings, so we need a copy of filter_glob_param. */ -struct kunit_test_filter { +struct kunit_glob_filter { char *suite_glob; char *test_glob; }; /* Split "suite_glob.test_glob" into two. Assumes filter_glob is not empty. */ -static void kunit_parse_filter_glob(struct kunit_test_filter *parsed, +static void kunit_parse_filter_glob(struct kunit_glob_filter *parsed, const char *filter_glob) { const int len = strlen(filter_glob); @@ -56,7 +61,7 @@ static void kunit_parse_filter_glob(struct kunit_test_filter *parsed, /* Create a copy of suite with only tests that match test_glob. */ static struct kunit_suite * -kunit_filter_tests(const struct kunit_suite *const suite, const char *test_glob) +kunit_filter_glob_tests(const struct kunit_suite *const suite, const char *test_glob) { int n = 0; struct kunit_case *filtered, *test_case; @@ -110,12 +115,15 @@ static void kunit_free_suite_set(struct suite_set suite_set) static struct suite_set kunit_filter_suites(const struct suite_set *suite_set, const char *filter_glob, + char **filters, + int filter_count, int *err) { - int i; - struct kunit_suite **copy, *filtered_suite; + int i, j, k; + struct kunit_suite **copy, *filtered_suite, *new_filtered_suite; struct suite_set filtered; - struct kunit_test_filter filter; + struct kunit_glob_filter parsed_glob; + struct kunit_attr_filter parsed_filters[MAX_FILTERS]; const size_t max = suite_set->end - suite_set->start; @@ -126,17 +134,49 @@ static struct suite_set kunit_filter_suites(const struct suite_set *suite_set, return filtered; } - kunit_parse_filter_glob(&filter, filter_glob); + if (filter_glob) + kunit_parse_filter_glob(&parsed_glob, filter_glob); - for (i = 0; &suite_set->start[i] != suite_set->end; i++) { - if (!glob_match(filter.suite_glob, suite_set->start[i]->name)) - continue; + /* Parse attribute filters */ + if (filter_count) { + for (j = 0; j < filter_count; j++) { + parsed_filters[j] = kunit_parse_filter_attr(filters[j], err); + if (*err) + return filtered; + } + } - filtered_suite = kunit_filter_tests(suite_set->start[i], filter.test_glob); - if (IS_ERR(filtered_suite)) { - *err = PTR_ERR(filtered_suite); - return filtered; + for (i = 0; &suite_set->start[i] != suite_set->end; i++) { + filtered_suite = suite_set->start[i]; + if (filter_glob) { + if (!glob_match(parsed_glob.suite_glob, filtered_suite->name)) + continue; + filtered_suite = kunit_filter_glob_tests(filtered_suite, + parsed_glob.test_glob); + if (IS_ERR(filtered_suite)) { + *err = PTR_ERR(filtered_suite); + return filtered; + } } + if (filter_count) { + for (k = 0; k < filter_count; k++) { + new_filtered_suite = kunit_filter_attr_tests(filtered_suite, + parsed_filters[k], err); + + /* Free previous copy of suite */ + if (k > 0 || filter_glob) + kfree(filtered_suite); + filtered_suite = new_filtered_suite; + + if (*err) + return filtered; + if (IS_ERR(filtered_suite)) { + *err = PTR_ERR(filtered_suite); + return filtered; + } + } + } + if (!filtered_suite) continue; @@ -144,8 +184,8 @@ static struct suite_set kunit_filter_suites(const struct suite_set *suite_set, } filtered.end = copy; - kfree(filter.suite_glob); - kfree(filter.test_glob); + kfree(parsed_glob.suite_glob); + kfree(parsed_glob.test_glob); return filtered; } @@ -203,8 +243,9 @@ int kunit_run_all_tests(void) goto out; } - if (filter_glob_param) { - suite_set = kunit_filter_suites(&suite_set, filter_glob_param, &err); + if (filter_glob_param || filter_count) { + suite_set = kunit_filter_suites(&suite_set, filter_glob_param, + filter_param, filter_count, &err); if (err) { pr_err("kunit executor: error filtering suites: %d\n", err); goto out; @@ -218,7 +259,7 @@ int kunit_run_all_tests(void) else pr_err("kunit executor: unknown action '%s'\n", action_param); - if (filter_glob_param) { /* a copy was made of each suite */ + if (filter_glob_param || filter_count) { /* a copy was made of each suite */ kunit_free_suite_set(suite_set); } diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c index ce6749af374d..4c8cb46857b2 100644 --- a/lib/kunit/executor_test.c +++ b/lib/kunit/executor_test.c @@ -24,7 +24,7 @@ static struct kunit_case dummy_test_cases[] = { static void parse_filter_test(struct kunit *test) { - struct kunit_test_filter filter = {NULL, NULL}; + struct kunit_glob_filter filter = {NULL, NULL}; kunit_parse_filter_glob(&filter, "suite"); KUNIT_EXPECT_STREQ(test, filter.suite_glob, "suite"); @@ -50,7 +50,7 @@ static void filter_suites_test(struct kunit *test) subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases); /* Want: suite1, suite2, NULL -> suite2, NULL */ - got = kunit_filter_suites(&suite_set, "suite2", &err); + got = kunit_filter_suites(&suite_set, "suite2", NULL, 0, &err); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start); KUNIT_ASSERT_EQ(test, err, 0); kfree_at_end(test, got.start); @@ -74,7 +74,7 @@ static void filter_suites_test_glob_test(struct kunit *test) subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases); /* Want: suite1, suite2, NULL -> suite2 (just test1), NULL */ - got = kunit_filter_suites(&suite_set, "suite2.test2", &err); + got = kunit_filter_suites(&suite_set, "suite2.test2", NULL, 0, &err); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start); KUNIT_ASSERT_EQ(test, err, 0); kfree_at_end(test, got.start); @@ -100,7 +100,7 @@ static void filter_suites_to_empty_test(struct kunit *test) subsuite[0] = alloc_fake_suite(test, "suite1", dummy_test_cases); subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases); - got = kunit_filter_suites(&suite_set, "not_found", &err); + got = kunit_filter_suites(&suite_set, "not_found", NULL, 0, &err); KUNIT_ASSERT_EQ(test, err, 0); kfree_at_end(test, got.start); /* just in case */ From patchwork Sat Jun 10 00:51:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 13274643 X-Patchwork-Delegate: brendanhiggins@google.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2280C7EE2E for ; Sat, 10 Jun 2023 00:52:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229818AbjFJAwS (ORCPT ); Fri, 9 Jun 2023 20:52:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231921AbjFJAwR (ORCPT ); Fri, 9 Jun 2023 20:52:17 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC95D30E5 for ; Fri, 9 Jun 2023 17:52:14 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-bb39aebdd87so5208453276.0 for ; Fri, 09 Jun 2023 17:52:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686358334; x=1688950334; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=uBsX1t2HFlz4ikZu05s+Ij9XO4LQtrN6NyLytDkTSA4=; b=Sn2Dd/x48qKkeIH4X8r0VPCdNgCpAfZC4xE9ujjDXRue4B+pfHp1HmaywFTo0K3+6o lIMdQITumf52CzC5mIO9rOY+PwTnj+07s0iqrmtJgtRE2Z8oTSd6T6sNPkPhlZnY9osq nfZiY0EuUD6jsHoET9sCuAxNiGJ9Hf9JR+wKeOS4o+tt+EjPPcW891tA3Mq4kGdCXoVL SnYw+lK2w+y7vTZIL5IyuiT+SEO32BbAhL6HWhvz289unvwaZ7++RyMiK1UTmhtSO8bG 3aBMxvhnYHTlNasVxX5xJjtP5RRCRCcYd4kAdxCQjIhdVpM4D8LZF/q+hgEfBSnUqHeR 7qvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686358334; x=1688950334; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=uBsX1t2HFlz4ikZu05s+Ij9XO4LQtrN6NyLytDkTSA4=; b=IxGpTjfOwRq7tYjhFXA7XOa0FiZHn1l8O3mFYXs71Dy0MDb9sw6lvMnup4GG7it1WV 1bH007TTnXkmy7hwSq8HdlN6mrPqIN9EqjXVfwWWpXENNa386+qIypcdY6TCGOhcibe1 mr5KkIPipgwQwE8ODarzoV3tu/Qv1hVnXQ62gP/VSfp7FlUGOb7uZajCdeL84qeFbB7e uxDzPQfpEWU3Je4NbMerHfGcinMswyFby5EZpWfAvGIeABDsmRUXlWGL/wqrtpnj29mc fFv7j51Y2DP+wnTKkb1HqdsdCnnfS3D4kKC2BDmNPiMJ4Oh2JfRZhmZQPYlVwc7YEXti /J+g== X-Gm-Message-State: AC+VfDz/USH44pqiazuwEj2z4NXUU4ubqlq3k0KftiwOB5rw+KU9ewmN A3pKHO9bGC0RT+gQRWOa2BDwNPjGYA== X-Google-Smtp-Source: ACHHUZ7SEar8r2ciV3/3VZaJyuZQBccjH4OUT8kuimqEuwg1KKuOjTcXWu1yCDpCVeSKtPjG3+Q6ZcY4OQ== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a25:8c07:0:b0:bad:2b06:da3 with SMTP id k7-20020a258c07000000b00bad2b060da3mr2079249ybl.3.1686358333943; Fri, 09 Jun 2023 17:52:13 -0700 (PDT) Date: Sat, 10 Jun 2023 00:51:47 +0000 In-Reply-To: <20230610005149.1145665-1-rmoar@google.com> Mime-Version: 1.0 References: <20230610005149.1145665-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230610005149.1145665-5-rmoar@google.com> Subject: [RFC v1 4/6] kunit: tool: Add command line interface to filter and report attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Add ability to use kunit.py to filter attributes and to report a list of tests including attributes without running tests. Add flag "--filter" to input filters on test attributes. Tests will be filtered out if they do not match all inputted filters. Example: --filter speed=slow This filter would run only the tests that are marked as slow. Note there cannot be spaces within a filter. As said in the previous patch, filters can have different operations: <, >, <=, >=, !=, and =. Note that the characters < and > are often interpreted by the shell, so they may need to be quoted or escaped. Example: --filter "speed>=normal" or –filter speed\>=normal This filter would run only the tests that have the speed faster than or equal to normal. Add flag "--list_tests" to output a list of tests and their attributes without running tests. This will be useful to see test attributes and which tests will run with given filters. Example of the output of these tests: example example.test_1 example.test_2 # example.test_2.speed: slow This output includes a suite, example, with two test cases, test_1 and test_2. And in this instance test_2 has been marked as slow. Signed-off-by: Rae Moar --- tools/testing/kunit/kunit.py | 34 +++++++++++++++++---- tools/testing/kunit/kunit_kernel.py | 6 ++-- tools/testing/kunit/kunit_tool_test.py | 41 +++++++++++++------------- 3 files changed, 54 insertions(+), 27 deletions(-) diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 3905c43369c3..661c39f7acf5 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -55,8 +55,10 @@ class KunitExecRequest(KunitParseRequest): build_dir: str timeout: int filter_glob: str + filter: Optional[List[str]] kernel_args: Optional[List[str]] run_isolated: Optional[str] + list_tests: Optional[bool] @dataclass class KunitRequest(KunitExecRequest, KunitBuildRequest): @@ -100,7 +102,7 @@ def config_and_build_tests(linux: kunit_kernel.LinuxSourceTree, return build_tests(linux, request) -def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> List[str]: +def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> Iterable[str]: args = ['kunit.action=list'] if request.kernel_args: args.extend(request.kernel_args) @@ -108,13 +110,17 @@ def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) output = linux.run_kernel(args=args, timeout=request.timeout, filter_glob=request.filter_glob, + filter=request.filter, build_dir=request.build_dir) lines = kunit_parser.extract_tap_lines(output) # Hack! Drop the dummy TAP version header that the executor prints out. lines.pop() # Filter out any extraneous non-test output that might have gotten mixed in. - return [l for l in lines if re.match(r'^[^\s.]+\.[^\s.]+$', l)] + return output + +def _get_tests(output: Iterable[str]) -> List[str]: + return [l for l in output if re.match(r'^[^\s.]+\.[^\s.]+$', l)] def _suites_from_test_list(tests: List[str]) -> List[str]: """Extracts all the suites from an ordered list of tests.""" @@ -132,8 +138,14 @@ def _suites_from_test_list(tests: List[str]) -> List[str]: def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> KunitResult: filter_globs = [request.filter_glob] + if request.list_tests: + output = _list_tests(linux, request) + for line in output: + print(line.rstrip()) + return KunitResult(status=KunitStatus.SUCCESS, elapsed_time=0.0) if request.run_isolated: - tests = _list_tests(linux, request) + output = _list_tests(linux, request) + tests = _get_tests(output) if request.run_isolated == 'test': filter_globs = tests elif request.run_isolated == 'suite': @@ -155,6 +167,7 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) - args=request.kernel_args, timeout=request.timeout, filter_glob=filter_glob, + filter=request.filter, build_dir=request.build_dir) _, test_result = parse_tests(request, metadata, run_result) @@ -341,6 +354,11 @@ def add_exec_opts(parser: argparse.ArgumentParser) -> None: nargs='?', default='', metavar='filter_glob') + parser.add_argument('--filter', + help='Filter which KUnit tests run by attributes' + 'e.g. speed=fast or speed=>low', + type=str, + nargs='*') parser.add_argument('--kernel_args', help='Kernel command-line parameters. Maybe be repeated', action='append', metavar='') @@ -350,6 +368,8 @@ def add_exec_opts(parser: argparse.ArgumentParser) -> None: 'what ran before it.', type=str, choices=['suite', 'test']) + parser.add_argument('--list_tests', help='If set, list all tests and attributes.', + action='store_true') def add_parse_opts(parser: argparse.ArgumentParser) -> None: parser.add_argument('--raw_output', help='If set don\'t parse output from kernel. ' @@ -398,8 +418,10 @@ def run_handler(cli_args: argparse.Namespace) -> None: json=cli_args.json, timeout=cli_args.timeout, filter_glob=cli_args.filter_glob, + filter=cli_args.filter, kernel_args=cli_args.kernel_args, - run_isolated=cli_args.run_isolated) + run_isolated=cli_args.run_isolated, + list_tests=cli_args.list_tests) result = run_tests(linux, request) if result.status != KunitStatus.SUCCESS: sys.exit(1) @@ -441,8 +463,10 @@ def exec_handler(cli_args: argparse.Namespace) -> None: json=cli_args.json, timeout=cli_args.timeout, filter_glob=cli_args.filter_glob, + filter=cli_args.filter, kernel_args=cli_args.kernel_args, - run_isolated=cli_args.run_isolated) + run_isolated=cli_args.run_isolated, + list_tests=cli_args.list_tests) result = exec_tests(linux, exec_request) stdout.print_with_timestamp(( 'Elapsed time: %.3fs\n') % (result.elapsed_time)) diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py index 7f648802caf6..62cb8200f60e 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -330,11 +330,13 @@ class LinuxSourceTree: return False return self.validate_config(build_dir) - def run_kernel(self, args: Optional[List[str]]=None, build_dir: str='', filter_glob: str='', timeout: Optional[int]=None) -> Iterator[str]: + def run_kernel(self, args: Optional[List[str]]=None, build_dir: str='', filter_glob: str='', filter: Optional[List[str]]=None, timeout: Optional[int]=None) -> Iterator[str]: if not args: args = [] if filter_glob: - args.append('kunit.filter_glob='+filter_glob) + args.append('kunit.filter_glob=' + filter_glob) + if filter: + args.append('kunit.filter=' + (','.join(filter))) args.append('kunit.enable=1') process = self._ops.start(args, build_dir) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index be35999bb84f..4a7f3112d06c 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -14,6 +14,7 @@ import tempfile, shutil # Handling test_tmpdir import itertools import json import os +import re import signal import subprocess from typing import Iterable @@ -597,7 +598,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 0) self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir='.kunit', filter_glob='', timeout=300) + args=None, build_dir='.kunit', filter_glob='', filter=None, timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_run_passes_args_pass(self): @@ -605,7 +606,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir='.kunit', filter_glob='', timeout=300) + args=None, build_dir='.kunit', filter_glob='', filter=None, timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_exec_passes_args_fail(self): @@ -629,7 +630,7 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run']) self.assertEqual(e.exception.code, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir='.kunit', filter_glob='', timeout=300) + args=None, build_dir='.kunit', filter_glob='', filter=None, timeout=300) self.print_mock.assert_any_call(StrContains(' 0 tests run!')) def test_exec_raw_output(self): @@ -670,13 +671,13 @@ class KUnitMainTest(unittest.TestCase): self.linux_source_mock.run_kernel = mock.Mock(return_value=[]) kunit.main(['run', '--raw_output', 'filter_glob']) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir='.kunit', filter_glob='filter_glob', timeout=300) + args=None, build_dir='.kunit', filter_glob='filter_glob', filter=None, timeout=300) def test_exec_timeout(self): timeout = 3453 kunit.main(['exec', '--timeout', str(timeout)]) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir='.kunit', filter_glob='', timeout=timeout) + args=None, build_dir='.kunit', filter_glob='', filter=None, timeout=timeout) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_run_timeout(self): @@ -684,7 +685,7 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run', '--timeout', str(timeout)]) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir='.kunit', filter_glob='', timeout=timeout) + args=None, build_dir='.kunit', filter_glob='', filter=None, timeout=timeout) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_run_builddir(self): @@ -692,7 +693,7 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run', '--build_dir=.kunit']) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir=build_dir, filter_glob='', timeout=300) + args=None, build_dir=build_dir, filter_glob='', filter=None, timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_config_builddir(self): @@ -710,7 +711,7 @@ class KUnitMainTest(unittest.TestCase): build_dir = '.kunit' kunit.main(['exec', '--build_dir', build_dir]) self.linux_source_mock.run_kernel.assert_called_once_with( - args=None, build_dir=build_dir, filter_glob='', timeout=300) + args=None, build_dir=build_dir, filter_glob='', filter=None, timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_run_kunitconfig(self): @@ -786,7 +787,7 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run', '--kernel_args=a=1', '--kernel_args=b=2']) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=['a=1','b=2'], build_dir='.kunit', filter_glob='', timeout=300) + args=['a=1','b=2'], build_dir='.kunit', filter_glob='', filter=None, timeout=300) self.print_mock.assert_any_call(StrContains('Testing complete.')) def test_list_tests(self): @@ -794,12 +795,13 @@ class KUnitMainTest(unittest.TestCase): self.linux_source_mock.run_kernel.return_value = ['TAP version 14', 'init: random output'] + want got = kunit._list_tests(self.linux_source_mock, - kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'suite')) + kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, None, 'suite', False)) + tests = kunit._get_tests(got) - self.assertEqual(got, want) + self.assertEqual(tests, want) # Should respect the user's filter glob when listing tests. self.linux_source_mock.run_kernel.assert_called_once_with( - args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', timeout=300) + args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', filter=None, timeout=300) @mock.patch.object(kunit, '_list_tests') @@ -809,10 +811,10 @@ class KUnitMainTest(unittest.TestCase): # Should respect the user's filter glob when listing tests. mock_tests.assert_called_once_with(mock.ANY, - kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*', None, 'suite')) + kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*', None, None, 'suite', False)) self.linux_source_mock.run_kernel.assert_has_calls([ - mock.call(args=None, build_dir='.kunit', filter_glob='suite.test*', timeout=300), - mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test*', timeout=300), + mock.call(args=None, build_dir='.kunit', filter_glob='suite.test*', filter=None, timeout=300), + mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test*', filter=None, timeout=300), ]) @mock.patch.object(kunit, '_list_tests') @@ -822,13 +824,12 @@ class KUnitMainTest(unittest.TestCase): # Should respect the user's filter glob when listing tests. mock_tests.assert_called_once_with(mock.ANY, - kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'test')) + kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, None, 'test', False)) self.linux_source_mock.run_kernel.assert_has_calls([ - mock.call(args=None, build_dir='.kunit', filter_glob='suite.test1', timeout=300), - mock.call(args=None, build_dir='.kunit', filter_glob='suite.test2', timeout=300), - mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test1', timeout=300), + mock.call(args=None, build_dir='.kunit', filter_glob='suite.test1', filter=None, timeout=300), + mock.call(args=None, build_dir='.kunit', filter_glob='suite.test2', filter=None, timeout=300), + mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test1', filter=None, timeout=300), ]) - if __name__ == '__main__': unittest.main() From patchwork Sat Jun 10 00:51:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 13274644 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 626DCC7EE25 for ; Sat, 10 Jun 2023 00:52:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232829AbjFJAwW (ORCPT ); Fri, 9 Jun 2023 20:52:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232531AbjFJAwR (ORCPT ); Fri, 9 Jun 2023 20:52:17 -0400 Received: from mail-qt1-x849.google.com (mail-qt1-x849.google.com [IPv6:2607:f8b0:4864:20::849]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 46F5EE1 for ; Fri, 9 Jun 2023 17:52:16 -0700 (PDT) Received: by mail-qt1-x849.google.com with SMTP id d75a77b69052e-3f9a9df0b85so19523411cf.1 for ; Fri, 09 Jun 2023 17:52:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686358335; x=1688950335; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=T3zbnVqAY087jc6zf5zGMbDLx4xURbX3P9p7/Tuiobs=; b=et/MOGBK43VQyQvYIB7WrQSc2K4IoXPVkm0UVfNUNy96HPcVZba6zLNT5urtOdGcBs P5jt2t2alfdalvP+MXwOg5CshT5nrkNH4RWZjKB7EdlIykKTWXG/akynbg/MxRcvIali cR6E0dGflT2FGLplmwq9iV7KLl5YiqcNNNeJqrHLkpgbofZ845oFq7BFKXlpp3VAZhyB 0anqrIvE8hlk1cks2OHMBgZmMdPUQYGvDXR5uzuiNXd1k6vEEfIPVZWzwFBkXVAFjir+ T7UvI1CHBZDi+GBLKQfZrLguMJw+Mcmyrl5IDqx5fM6Oa4UHrYFHXc04eE87IVZpjdve mT4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686358335; x=1688950335; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=T3zbnVqAY087jc6zf5zGMbDLx4xURbX3P9p7/Tuiobs=; b=NsklKKsaSj5BvP1lJMLXDxvFWmL1Ofc7C5HtJTQFidOt7JgSfFmiEUV1yeKaB0L0XC NXWSUkEtItSglj6OaXTPTHvXGHc7Js6ahExJXu8W0SqpvMweFh/BYFWGjamz0BM8gMQT +D+a6Fxuswp54j7W6Kr2Di4R4aJuqn3zj9Z3M8GKQG14CfXlp/8ch9TkMGiBxwTn/53K iBJfh3XFeIK0qMYVE0BXWsWj+SEwf8NJAyqgpf/thuzD3hctLNA6cu6RJOL+X5uSyCf+ LRMsxAokUbtt9ihMKcFgOFIJhWkLheI0UKmVnX2F4qdcjATxbSryxY5cVBUVbzBsUY2E CoVw== X-Gm-Message-State: AC+VfDzi1uwMJjTmSEovl923AR6w/FeoTkYtOkNo0bZLNRqiwkObmo5L +oTJyzo3ixNalrVYk9qkhx20zbWASw== X-Google-Smtp-Source: ACHHUZ6dnvhynsP+qbvMSlLVpDX2UXxgBwmpdFpa0jMBlxgs6rhuAeeLQxvZlPYg2eocE1JYwxLuOIy3Sg== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a05:622a:1485:b0:3f7:469b:91a with SMTP id t5-20020a05622a148500b003f7469b091amr1337196qtx.6.1686358335447; Fri, 09 Jun 2023 17:52:15 -0700 (PDT) Date: Sat, 10 Jun 2023 00:51:48 +0000 In-Reply-To: <20230610005149.1145665-1-rmoar@google.com> Mime-Version: 1.0 References: <20230610005149.1145665-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230610005149.1145665-6-rmoar@google.com> Subject: [RFC v1 5/6] kunit: memcpy: Mark tests as slow using test attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Mark slow memcpy KUnit tests using test attributes. Tests marked as slow are as follows: memcpy_large_test, memmove_test, memmove_large_test, and memmove_overlap_test. These tests were the slowest of the memcpy tests and relatively slower to most other KUnit tests. Most of these tests are already skipped when CONFIG_MEMCPY_SLOW_KUNIT_TEST is not enabled. These tests can now be filtered on using the KUnit test attribute filtering feature. Example: --filter "speed>slow". This will run only the tests that have speeds faster than slow. The slow attribute will also be outputted in KTAP. Signed-off-by: Rae Moar --- lib/memcpy_kunit.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/memcpy_kunit.c b/lib/memcpy_kunit.c index 887926f04731..440aee705ccc 100644 --- a/lib/memcpy_kunit.c +++ b/lib/memcpy_kunit.c @@ -551,10 +551,10 @@ static void strtomem_test(struct kunit *test) static struct kunit_case memcpy_test_cases[] = { KUNIT_CASE(memset_test), KUNIT_CASE(memcpy_test), - KUNIT_CASE(memcpy_large_test), - KUNIT_CASE(memmove_test), - KUNIT_CASE(memmove_large_test), - KUNIT_CASE(memmove_overlap_test), + KUNIT_CASE_SLOW(memcpy_large_test), + KUNIT_CASE_SLOW(memmove_test), + KUNIT_CASE_SLOW(memmove_large_test), + KUNIT_CASE_SLOW(memmove_overlap_test), KUNIT_CASE(strtomem_test), {} }; From patchwork Sat Jun 10 00:51:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rae Moar X-Patchwork-Id: 13274645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCE0BC8300C for ; Sat, 10 Jun 2023 00:52:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232471AbjFJAwX (ORCPT ); Fri, 9 Jun 2023 20:52:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232735AbjFJAwT (ORCPT ); Fri, 9 Jun 2023 20:52:19 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 42A4630E8 for ; Fri, 9 Jun 2023 17:52:18 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-564fb1018bcso33561397b3.0 for ; Fri, 09 Jun 2023 17:52:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686358337; x=1688950337; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KcmdgdtrOaPERzGMLLSgLUzQ9hHmsIkGLHU1LmoZfH8=; b=22Hb/1PuiZ6DCILf0AAvPSn7RCceVmjHUMkry1uEt7SnQWVqm0GdauDWyayZNLHePk treeEyNSuxwZnzIj1JdAE3QODBhN0LUH3X7GPduL7LWybzUeWhV2S0tIMW917YmYmdrk uCGy0JI5whT0RAjzzYZI0O7rngpvEVwH8KfHyApXpcfFK57jGWcFz/1shb4TvOaF+QpR vAuhIqxiS520cOmhta6uqhv5XKIBro6kTWqQK9CLNO9YICKTNoh0xDQt8QfcoR1zHTKM UMLJhNgR4gn2Sewwd4+sflVmXIgvriDSlcI5c4hGOIumWhCEMZYgJTIbeFXcYhKJIYIj 5zbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686358337; x=1688950337; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KcmdgdtrOaPERzGMLLSgLUzQ9hHmsIkGLHU1LmoZfH8=; b=jBI7rEcGxpICACHSRHzxqc3skmJwuQ88u1MWPCVeReTIPcBx0Ks7mCvflMWrN34HSD ulqE5zV9YRmmls1GtEIJZXDm0Vy4+1tLed9BOmYONI8+TSSuKQ7PEMFPYB8784g/ZcoD 60SG7+cL3U+g538G/UM6sbVfq6GAFzZSIb0kOOOru3RHlEqtUGNhIIj/8en4jYOYUC/J dad1rSFydh0nlWrfgix2bQviB+cu4zlyBwHlkErs+SZHgtoitTDDSt6paXTyqBZUqSqJ lWyUPtwwaTtVH7udO6zICL0Qy6oWTlYwPuf0CTb0n2nETSrr7cBF9BU1Pjv7Y5TSl+9W S1Tg== X-Gm-Message-State: AC+VfDwqOKpNfC7lZA2XCGGfm3yKSI/atkroIHygiJtgEUXXVRaGysQz 90CBr0U5RBffmD6nBvn1Be8zpbySiw== X-Google-Smtp-Source: ACHHUZ50X9vGPEaRfFB8Zyfp7hAGjAAFOLWkJmhz9m0q7Z+GgI9UAkMD03XJMau7OkFzqamRls5gtUVd/A== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a81:c14b:0:b0:561:b7fc:7445 with SMTP id e11-20020a81c14b000000b00561b7fc7445mr1706591ywl.1.1686358337108; Fri, 09 Jun 2023 17:52:17 -0700 (PDT) Date: Sat, 10 Jun 2023 00:51:49 +0000 In-Reply-To: <20230610005149.1145665-1-rmoar@google.com> Mime-Version: 1.0 References: <20230610005149.1145665-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230610005149.1145665-7-rmoar@google.com> Subject: [RFC v1 6/6] kunit: time: Mark test as slow using test attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Mark the time KUnit test, time64_to_tm_test_date_range, as slow using test attributes. This test ran relatively much slower than most other KUnit tests. By marking this test as slow, the test can now be filtered on using the KUnit test attribute filtering feature. Example: --filter "speed>slow". This will run only the tests that have speeds faster than slow. The slow attribute will also be outputted in KTAP. Signed-off-by: Rae Moar --- kernel/time/time_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/time/time_test.c b/kernel/time/time_test.c index 831e8e779ace..ca058c8af6ba 100644 --- a/kernel/time/time_test.c +++ b/kernel/time/time_test.c @@ -86,7 +86,7 @@ static void time64_to_tm_test_date_range(struct kunit *test) } static struct kunit_case time_test_cases[] = { - KUNIT_CASE(time64_to_tm_test_date_range), + KUNIT_CASE_SLOW(time64_to_tm_test_date_range), {} };