From patchwork Tue Jul 16 09:42:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Higgins X-Patchwork-Id: 11045761 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7B33D746 for ; Tue, 16 Jul 2019 09:44:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6744128451 for ; Tue, 16 Jul 2019 09:44:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 58C6C285A9; Tue, 16 Jul 2019 09:44:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58990285B0 for ; Tue, 16 Jul 2019 09:44:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1733096AbfGPJnk (ORCPT ); Tue, 16 Jul 2019 05:43:40 -0400 Received: from mail-pl1-f202.google.com ([209.85.214.202]:34602 "EHLO mail-pl1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732997AbfGPJnf (ORCPT ); Tue, 16 Jul 2019 05:43:35 -0400 Received: by mail-pl1-f202.google.com with SMTP id 71so9914163pld.1 for ; Tue, 16 Jul 2019 02:43:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=jCUV64Lerg+AOWflh1KKV24AQjxHpkOWaQRDVvGmlX0=; b=OR825/37af2RRCmj6CjGYrG/dd5iQHcpetOszQVAk3qUx9pqwJ6o3a9UlFa5jB7bo1 9igNTyshrmW8kPFIKhW+CAu3F8J5jb7BjTZCtrdkA42dK84M5De7+eqUQId2erjof/+N WTorHVGOdbJI8D35YtU8B1h+EwRZ2RNSZo6nPlni4HKeavOG10xkkE1jJN6kmQWFbmTb 1gl9kLdLqtd8+mcPrfFjhxgEDwzrmSYtFm5GJ3yvzXdS/hREvln920amtwPXpnfKY5aj CYgB1Smcx4FVl0XiokZt6yHxwRseYUXWxuwK1ymfWvbFdwQUPUSLv20rL1HZSgVDNtAN tLOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=jCUV64Lerg+AOWflh1KKV24AQjxHpkOWaQRDVvGmlX0=; b=ZFBVByzRIqPIEH8FN1wB4PImqkIdqcDkfHlUs1XmzQYhTP7/ser3mxYhpbJNhW2FGU 4Dmmb/hIakCgk1roV+Bk6pF62T/sLxT8fpZI/GfgUkwy2WQxiIuuIKp7wGX9yXulKlBA 2QT7Yn6M0mvEU+3a3cyC36rQ+TBr+rljDceAy2kacsQtbMNzI7FiAnqikZnKbWMEyh2k XtZBz9BLkXKYiI1aFrbbsEQZIDz0SA2+K09wpio7t8WS1NxpqcI5MIoF8zXXEPbHlh7w 4yX6JLE/BuQ5hZcvHoA9buUOMSj5P54Elx1ChYqahG8DCLLPw5cmRvbvkwKd2CqrnQv6 r6/A== X-Gm-Message-State: APjAAAWtdiuskgQER/dzztHKzwjU1yFIW+71uCS9hAfJ7qL3O1PkO8xb WHWTJxt+MTJsyNVyES3qHcc81TRJKOOtp3Y5teTRkQ== X-Google-Smtp-Source: APXvYqwPTgtr+5hZqjB53EbBUK4IJHrOgh1jJwGhK6sWYrC3H9sQ8IuR1INif3yfX6ma3QMF0kdUUG8dgjfy1/1x2Z/W3A== X-Received: by 2002:a63:3203:: with SMTP id y3mr32699223pgy.191.1563270214123; Tue, 16 Jul 2019 02:43:34 -0700 (PDT) Date: Tue, 16 Jul 2019 02:42:53 -0700 In-Reply-To: <20190716094302.180360-1-brendanhiggins@google.com> Message-Id: <20190716094302.180360-10-brendanhiggins@google.com> Mime-Version: 1.0 References: <20190716094302.180360-1-brendanhiggins@google.com> X-Mailer: git-send-email 2.22.0.510.g264f2c817a-goog Subject: [PATCH v10 09/18] kunit: test: add support for test abort From: Brendan Higgins To: frowand.list@gmail.com, gregkh@linuxfoundation.org, jpoimboe@redhat.com, keescook@google.com, kieran.bingham@ideasonboard.com, mcgrof@kernel.org, peterz@infradead.org, robh@kernel.org, sboyd@kernel.org, shuah@kernel.org, tytso@mit.edu, yamada.masahiro@socionext.com Cc: devicetree@vger.kernel.org, dri-devel@lists.freedesktop.org, kunit-dev@googlegroups.com, linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-nvdimm@lists.01.org, linux-um@lists.infradead.org, Alexander.Levin@microsoft.com, Tim.Bird@sony.com, amir73il@gmail.com, dan.carpenter@oracle.com, daniel@ffwll.ch, jdike@addtoit.com, joel@jms.id.au, julia.lawall@lip6.fr, khilman@baylibre.com, knut.omang@oracle.com, logang@deltatee.com, mpe@ellerman.id.au, pmladek@suse.com, rdunlap@infradead.org, richard@nod.at, rientjes@google.com, rostedt@goodmis.org, wfg@linux.intel.com, Brendan Higgins Sender: linux-kbuild-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kbuild@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add support for aborting/bailing out of test cases, which is needed for implementing assertions. An assertion is like an expectation, but bails out of the test case early if the assertion is not met. The idea with assertions is that you use them to state all the preconditions for your test. Logically speaking, these are the premises of the test case, so if a premise isn't true, there is no point in continuing the test case because there are no conclusions that can be drawn without the premises. Whereas, the expectation is the thing you are trying to prove. Signed-off-by: Brendan Higgins Reviewed-by: Greg Kroah-Hartman Reviewed-by: Logan Gunthorpe --- include/kunit/test.h | 16 ++++ include/kunit/try-catch.h | 69 +++++++++++++++ kunit/Makefile | 3 +- kunit/test.c | 176 ++++++++++++++++++++++++++++++++++---- kunit/try-catch.c | 95 ++++++++++++++++++++ 5 files changed, 343 insertions(+), 16 deletions(-) create mode 100644 include/kunit/try-catch.h create mode 100644 kunit/try-catch.c diff --git a/include/kunit/test.h b/include/kunit/test.h index 4cc94f09389d7..1d35f542a30c9 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -13,6 +13,7 @@ #include #include #include +#include struct kunit_resource; @@ -167,6 +168,7 @@ struct kunit { /* private: internal use only. */ const char *name; /* Read only after initialization! */ + struct kunit_try_catch try_catch; /* * success starts as true, and may only be set to false during a test * case; thus, it is safe to update this across multiple threads using @@ -176,6 +178,11 @@ struct kunit { */ bool success; /* Read only after test_case finishes! */ spinlock_t lock; /* Gaurds all mutable test state. */ + /* + * death_test may be both set and unset from multiple threads in a test + * case. + */ + bool death_test; /* Protected by lock. */ /* * Because resources is a list that may be updated multiple times (with * new resources) from any thread associated with a test case, we must @@ -184,10 +191,19 @@ struct kunit { struct list_head resources; /* Protected by lock. */ }; +static inline void kunit_set_death_test(struct kunit *test, bool death_test) +{ + spin_lock(&test->lock); + test->death_test = death_test; + spin_unlock(&test->lock); +} + void kunit_init_test(struct kunit *test, const char *name); void kunit_fail(struct kunit *test, struct kunit_stream *stream); +void kunit_abort(struct kunit *test); + int kunit_run_tests(struct kunit_suite *suite); /** diff --git a/include/kunit/try-catch.h b/include/kunit/try-catch.h new file mode 100644 index 0000000000000..8a414a9af0b64 --- /dev/null +++ b/include/kunit/try-catch.h @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * An API to allow a function, that may fail, to be executed, and recover in a + * controlled manner. + * + * Copyright (C) 2019, Google LLC. + * Author: Brendan Higgins + */ + +#ifndef _KUNIT_TRY_CATCH_H +#define _KUNIT_TRY_CATCH_H + +#include + +typedef void (*kunit_try_catch_func_t)(void *); + +struct kunit; + +/* + * struct kunit_try_catch - provides a generic way to run code which might fail. + * @context: used to pass user data to the try and catch functions. + * + * kunit_try_catch provides a generic, architecture independent way to execute + * an arbitrary function of type kunit_try_catch_func_t which may bail out by + * calling kunit_try_catch_throw(). If kunit_try_catch_throw() is called, @try + * is stopped at the site of invocation and @catch is catch is called. + * + * struct kunit_try_catch provides a generic interface for the functionality + * needed to implement kunit->abort() which in turn is needed for implementing + * assertions. Assertions allow stating a precondition for a test simplifying + * how test cases are written and presented. + * + * Assertions are like expectations, except they abort (call + * kunit_try_catch_throw()) when the specified condition is not met. This is + * useful when you look at a test case as a logical statement about some piece + * of code, where assertions are the premises for the test case, and the + * conclusion is a set of predicates, rather expectations, that must all be + * true. If your premises are violated, it does not makes sense to continue. + */ +struct kunit_try_catch { + /* private: internal use only. */ + struct kunit *test; + struct completion *try_completion; + int try_result; + kunit_try_catch_func_t try; + kunit_try_catch_func_t catch; + void *context; +}; + +void kunit_try_catch_init(struct kunit_try_catch *try_catch, + struct kunit *test, + kunit_try_catch_func_t try, + kunit_try_catch_func_t catch); + +void kunit_try_catch_run(struct kunit_try_catch *try_catch, void *context); + +void __noreturn kunit_try_catch_throw(struct kunit_try_catch *try_catch); + +static inline int kunit_try_catch_get_result(struct kunit_try_catch *try_catch) +{ + return try_catch->try_result; +} + +/* + * Exposed for testing only. + */ +void kunit_generic_try_catch_init(struct kunit_try_catch *try_catch); + +#endif /* _KUNIT_TRY_CATCH_H */ diff --git a/kunit/Makefile b/kunit/Makefile index 60a9ea6cb4697..1f7680cfa11ad 100644 --- a/kunit/Makefile +++ b/kunit/Makefile @@ -1,6 +1,7 @@ obj-$(CONFIG_KUNIT) += test.o \ string-stream.o \ - kunit-stream.o + kunit-stream.o \ + try-catch.o obj-$(CONFIG_KUNIT_TEST) += string-stream-test.o diff --git a/kunit/test.c b/kunit/test.c index 1f94a9224b03e..12db50b221781 100644 --- a/kunit/test.c +++ b/kunit/test.c @@ -7,13 +7,26 @@ */ #include +#include #include +#include static void kunit_set_failure(struct kunit *test) { WRITE_ONCE(test->success, false); } +static bool kunit_get_death_test(struct kunit *test) +{ + bool death_test; + + spin_lock(&test->lock); + death_test = test->death_test; + spin_unlock(&test->lock); + + return death_test; +} + static int kunit_vprintk_emit(int level, const char *fmt, va_list args) { return vprintk_emit(0, level, NULL, 0, fmt, args); @@ -126,42 +139,175 @@ void kunit_fail(struct kunit *test, struct kunit_stream *stream) kunit_stream_commit(stream); } +void __noreturn kunit_abort(struct kunit *test) +{ + kunit_set_death_test(test, true); + + kunit_try_catch_throw(&test->try_catch); + + /* + * Throw could not abort from test. + * + * XXX: we should never reach this line! As kunit_try_catch_throw is + * marked __noreturn. + */ + WARN_ONCE(true, "Throw could not abort from test!\n"); +} + void kunit_init_test(struct kunit *test, const char *name) { spin_lock_init(&test->lock); INIT_LIST_HEAD(&test->resources); test->name = name; test->success = true; + test->death_test = false; } /* - * Performs all logic to run a test case. + * Initializes and runs test case. Does not clean up or do post validations. */ -static void kunit_run_case(struct kunit_suite *suite, - struct kunit_case *test_case) +static void kunit_run_case_internal(struct kunit *test, + struct kunit_suite *suite, + struct kunit_case *test_case) { - struct kunit test; - - kunit_init_test(&test, test_case->name); - if (suite->init) { int ret; - ret = suite->init(&test); + ret = suite->init(test); if (ret) { - kunit_err(&test, "failed to initialize: %d\n", ret); - kunit_set_failure(&test); - test_case->success = test.success; + kunit_err(test, "failed to initialize: %d\n", ret); + kunit_set_failure(test); return; } } - test_case->run_case(&test); + test_case->run_case(test); +} + +static void kunit_case_internal_cleanup(struct kunit *test) +{ + kunit_cleanup(test); +} +/* + * Performs post validations and cleanup after a test case was run. + * XXX: Should ONLY BE CALLED AFTER kunit_run_case_internal! + */ +static void kunit_run_case_cleanup(struct kunit *test, + struct kunit_suite *suite) +{ if (suite->exit) - suite->exit(&test); + suite->exit(test); + + kunit_case_internal_cleanup(test); +} + +/* + * Handles an unexpected crash in a test case. + */ +static void kunit_handle_test_crash(struct kunit *test, + struct kunit_suite *suite, + struct kunit_case *test_case) +{ + kunit_err(test, "kunit test case crashed!"); + /* + * TODO(brendanhiggins@google.com): This prints the stack trace up + * through this frame, not up to the frame that caused the crash. + */ + show_stack(NULL, NULL); + + kunit_case_internal_cleanup(test); +} + +struct kunit_try_catch_context { + struct kunit *test; + struct kunit_suite *suite; + struct kunit_case *test_case; +}; + +static void kunit_try_run_case(void *data) +{ + struct kunit_try_catch_context *ctx = data; + struct kunit *test = ctx->test; + struct kunit_suite *suite = ctx->suite; + struct kunit_case *test_case = ctx->test_case; + + /* + * kunit_run_case_internal may encounter a fatal error; if it does, + * abort will be called, this thread will exit, and finally the parent + * thread will resume control and handle any necessary clean up. + */ + kunit_run_case_internal(test, suite, test_case); + /* This line may never be reached. */ + kunit_run_case_cleanup(test, suite); +} + +static void kunit_catch_run_case(void *data) +{ + struct kunit_try_catch_context *ctx = data; + struct kunit *test = ctx->test; + struct kunit_suite *suite = ctx->suite; + struct kunit_case *test_case = ctx->test_case; + int try_exit_code = kunit_try_catch_get_result(&test->try_catch); + + if (try_exit_code) { + kunit_set_failure(test); + /* + * Test case could not finish, we have no idea what state it is + * in, so don't do clean up. + */ + if (try_exit_code == -ETIMEDOUT) + kunit_err(test, "test case timed out\n"); + /* + * Unknown internal error occurred preventing test case from + * running, so there is nothing to clean up. + */ + else + kunit_err(test, "internal error occurred preventing test case from running: %d\n", + try_exit_code); + return; + } + + if (kunit_get_death_test(test)) { + /* + * EXPECTED DEATH: kunit_run_case_internal encountered + * anticipated fatal error. Everything should be in a safe + * state. + */ + kunit_run_case_cleanup(test, suite); + } else { + /* + * UNEXPECTED DEATH: kunit_run_case_internal encountered an + * unanticipated fatal error. We have no idea what the state of + * the test case is in. + */ + kunit_handle_test_crash(test, suite, test_case); + kunit_set_failure(test); + } +} + +/* + * Performs all logic to run a test case. It also catches most errors that + * occurs in a test case and reports them as failures. + */ +static void kunit_run_case_catch_errors(struct kunit_suite *suite, + struct kunit_case *test_case) +{ + struct kunit_try_catch_context context; + struct kunit_try_catch *try_catch; + struct kunit test; + + kunit_init_test(&test, test_case->name); + try_catch = &test.try_catch; - kunit_cleanup(&test); + kunit_try_catch_init(try_catch, + &test, + kunit_try_run_case, + kunit_catch_run_case); + context.test = &test; + context.suite = suite; + context.test_case = test_case; + kunit_try_catch_run(try_catch, &context); test_case->success = test.success; } @@ -174,7 +320,7 @@ int kunit_run_tests(struct kunit_suite *suite) kunit_print_subtest_start(suite); for (test_case = suite->test_cases; test_case->run_case; test_case++) { - kunit_run_case(suite, test_case); + kunit_run_case_catch_errors(suite, test_case); kunit_print_test_case_ok_not_ok(test_case, test_case_count++); } diff --git a/kunit/try-catch.c b/kunit/try-catch.c new file mode 100644 index 0000000000000..de580f074387b --- /dev/null +++ b/kunit/try-catch.c @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * An API to allow a function, that may fail, to be executed, and recover in a + * controlled manner. + * + * Copyright (C) 2019, Google LLC. + * Author: Brendan Higgins + */ + +#include +#include +#include +#include + +void __noreturn kunit_try_catch_throw(struct kunit_try_catch *try_catch) +{ + try_catch->try_result = -EFAULT; + complete_and_exit(try_catch->try_completion, -EFAULT); +} + +static int kunit_generic_run_threadfn_adapter(void *data) +{ + struct kunit_try_catch *try_catch = data; + + try_catch->try(try_catch->context); + + complete_and_exit(try_catch->try_completion, 0); +} + +void kunit_try_catch_run(struct kunit_try_catch *try_catch, void *context) +{ + DECLARE_COMPLETION_ONSTACK(try_completion); + struct kunit *test = try_catch->test; + struct task_struct *task_struct; + int exit_code, status; + + try_catch->context = context; + try_catch->try_completion = &try_completion; + try_catch->try_result = 0; + task_struct = kthread_run(kunit_generic_run_threadfn_adapter, + try_catch, + "kunit_try_catch_thread"); + if (IS_ERR(task_struct)) { + try_catch->catch(try_catch->context); + return; + } + + /* + * TODO(brendanhiggins@google.com): We should probably have some type of + * variable timeout here. The only question is what that timeout value + * should be. + * + * The intention has always been, at some point, to be able to label + * tests with some type of size bucket (unit/small, integration/medium, + * large/system/end-to-end, etc), where each size bucket would get a + * default timeout value kind of like what Bazel does: + * https://docs.bazel.build/versions/master/be/common-definitions.html#test.size + * There is still some debate to be had on exactly how we do this. (For + * one, we probably want to have some sort of test runner level + * timeout.) + * + * For more background on this topic, see: + * https://mike-bland.com/2011/11/01/small-medium-large.html + */ + status = wait_for_completion_timeout(&try_completion, + 300 * MSEC_PER_SEC); /* 5 min */ + if (status < 0) { + kunit_err(test, "try timed out\n"); + try_catch->try_result = -ETIMEDOUT; + } + + exit_code = try_catch->try_result; + + if (!exit_code) + return; + + if (exit_code == -EFAULT) + try_catch->try_result = 0; + else if (exit_code == -EINTR) + kunit_err(test, "wake_up_process() was never called\n"); + else if (exit_code) + kunit_err(test, "Unknown error: %d\n", exit_code); + + try_catch->catch(try_catch->context); +} + +void kunit_try_catch_init(struct kunit_try_catch *try_catch, + struct kunit *test, + kunit_try_catch_func_t try, + kunit_try_catch_func_t catch) +{ + try_catch->test = test; + try_catch->try = try; + try_catch->catch = catch; +}