From patchwork Wed Jun 24 20:55:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Higgins X-Patchwork-Id: 11624371 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D6D76C1 for ; Wed, 24 Jun 2020 21:00:43 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 654B4206A5 for ; Wed, 24 Jun 2020 21:00:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="jeG0bHjp"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="igHQvbvp" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 654B4206A5 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:Mime-Version:Message-Id:Date: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=VJwXpk9xWDysaJLg++jxctnbFynq1W7AuET04SZZXDU=; b=jeG0bHjpSGpktwNJkBZPAdbtuP kz5XBnVy4aHw41Uk1VG6bpyqMABYhT+2umSamXvaX7b3TT39HOgjs+dbimuOO+ujUY6Dsn/Ejlbx8 h2mbIoxuTlErgEgWIPIRUQLSNb2x0SprPnWlMHH8J9X7YYWjr+TrGesmsi4Zg2YX14FZLxxkgkG6u IhjZt19QcC0oAtqe2RgrTsXtx+GHD6UUyXqbj1slTBxSPuG6cky1jtSf5Vt4MfIobl4xHHc5pKxsd NJZUXemqXVmz7yBsOKCXt+0KgvjkW1f34JjP6q5KSyn585+8X5wUI2ULekaXxjj5OA/HOVpX+uLkV NGUoV8BA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1joCTK-0001yR-5t; Wed, 24 Jun 2020 20:58:18 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1joCTE-0001u7-QK for linux-arm-kernel@lists.infradead.org; Wed, 24 Jun 2020 20:58:13 +0000 Received: by mail-yb1-xb49.google.com with SMTP id c3so3580881ybp.8 for ; Wed, 24 Jun 2020 13:58:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=z9RsMjfeXSae3iGDejA5AkwFlm1N1ZEabjVKLWLCH5k=; b=igHQvbvp05csXRFVNMsKy1ZebRhTuY3NP+zZ2HjV0T+/gI+jb+eKqCp93ukxeHPtVx vKexLly0v1uOT9X2cqIVd+g6jyd+7RTziC3rFr14BBMqIOg5QQKxu00/Rn339E+lmS/5 2xB33r447ROtLvWtq74Kg2n1OtxOnNYLbuojY+vL3V53NhRkZBTuGh6xp0FRtIwngav9 4hjWt/Vw8CwOzsDdviIQKbzX77bVV+KJXcjugw+Qa7sObPC9NZF7GUxIUSRc4G/re9eu nUorVxhww/S4gHa/YjkivVv2X0BStXm8mnwiQDOR/mq56sH+aMHCVSs/MvxaO0a+pOAO pr+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=z9RsMjfeXSae3iGDejA5AkwFlm1N1ZEabjVKLWLCH5k=; b=TEJqE1qC4/A9EkyIqSNdNtMR7urq5mCtDWkCHtMom+avJfDiO2dI2bFWWQWGunKwx+ m/lzFi3b8lQ4si+TuKNW2pxfi1vkyWrKQovAbDpnYOGAnpKQ8783YHiYIr0/wQMFL/EV M0EIfsZP2zf6TC+pQZC9z7L15l0YULPVH0SIqfHhviFiKOmmOgdL7F6s/FO8GupP+4ON 689kQe9FxFU/vJzZJrBZK/dC0EtqxUiDpMV0mppbZg0/e0RdfapY8pJ6Vi53NVwbpTHF obyfRK1tJZyMVQvQMEZEWsL6AZqS6UIERT97z+P12mVmRWp2ZIk1+Gk52EdEHeANCKn2 7LAA== X-Gm-Message-State: AOAM532SEhQBRaemKeNbB4r0bmt+fZ3cu0m4z91i/UzvWpjptLoRZtdT NWgf75fyqm5Gnf5Hc+KGds9II791x/5QSj7Fi55shQ== X-Google-Smtp-Source: ABdhPJwbCCxlHAdc2b6DVi9T8MLEsrJ1Mw0re6va2SY63A6gZs31gniWqbvj3OZKwy9TJdgHL9Ts4hghLYRC1CcMQOZgFg== X-Received: by 2002:a25:9345:: with SMTP id g5mr46614006ybo.485.1593032288632; Wed, 24 Jun 2020 13:58:08 -0700 (PDT) Date: Wed, 24 Jun 2020 13:55:39 -0700 Message-Id: <20200624205550.215599-1-brendanhiggins@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.27.0.212.ge8ba1cc988-goog Subject: [PATCH v4 00/11] kunit: create a centralized executor to dispatch all KUnit tests From: Brendan Higgins To: jdike@addtoit.com, richard@nod.at, anton.ivanov@cambridgegreys.com, arnd@arndb.de, keescook@chromium.org, skhan@linuxfoundation.org, alan.maguire@oracle.com, yzaikin@google.com, davidgow@google.com, akpm@linux-foundation.org, rppt@linux.ibm.com, frowand.list@gmail.com, catalin.marinas@arm.com, will@kernel.org, monstr@monstr.eu, mpe@ellerman.id.au, benh@kernel.crashing.org, paulus@samba.org, chris@zankel.net, jcmvbkbc@gmail.com X-Spam-Note: CRM114 invocation failed X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:b49 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, linux-xtensa@linux-xtensa.org, linux-doc@vger.kernel.org, sboyd@kernel.org, gregkh@linuxfoundation.org, linuxppc-dev@lists.ozlabs.org, linux-um@lists.infradead.org, linux-kernel@vger.kernel.org, Brendan Higgins , mcgrof@kernel.org, linux-kselftest@vger.kernel.org, logang@deltatee.com, linux-arm-kernel@lists.infradead.org, kunit-dev@googlegroups.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org ## TL;DR This patchset adds a centralized executor to dispatch tests rather than relying on late_initcall to schedule each test suite separately along with a couple of new features that depend on it. Also, sorry for the extreme delay in getting this out. Part of the delay came from finding that there were actually several architectures that the previous revision of this patchset didn't work on, so I went through and attempted to test this patchset on every architecture - more on that later. ## What am I trying to do? Conceptually, I am trying to provide a mechanism by which test suites can be grouped together so that they can be reasoned about collectively. The last two of three patches in this series add features which depend on this: PATCH 8/11 Prints out a test plan[1] right before KUnit tests are run; this is valuable because it makes it possible for a test harness to detect whether the number of tests run matches the number of tests expected to be run, ensuring that no tests silently failed. The test plan includes a count of tests that will run. With the centralized executor, the tests are located in a single data structure and thus can be counted. PATCH 9/11 Add a new kernel command-line option which allows the user to specify that the kernel poweroff, halt, or reboot after completing all KUnit tests; this is very handy for running KUnit tests on UML or a VM so that the UML/VM process exits cleanly immediately after running all tests without needing a special initramfs. The centralized executor provides a definitive point when all tests have completed and the poweroff, halt, or reboot could occur. In addition, by dispatching tests from a single location, we can guarantee that all KUnit tests run after late_init is complete, which was a concern during the initial KUnit patchset review (this has not been a problem in practice, but resolving with certainty is nevertheless desirable). Other use cases for this exist, but the above features should provide an idea of the value that this could provide. ## Changes since last revision: - On the last revision I got some messages from 0day that showed that this patchset didn't work on several architectures, one issue that this patchset addresses is that we were aligning both memory segments as well as structures in the segments to specific byte boundaries which was incorrect. - The issue mentioned above also caused me to test on additional architectures which revealed that some architectures other than UML do not use the default init linker section macro that most architectures use. There are now several new patches (2, 3, 4, and 6). - Fixed a formatting consistency issue in the kernel params documentation patch (9/9). - Add a brief blurb on how and when the kunit_test_suite macro works. ## Remaining work to be done: The only architecture for which I was able to get a compiler, but was apparently unable to get KUnit into a section that the executor to see was m68k - not sure why. Alan Maguire (1): kunit: test: create a single centralized executor for all tests Brendan Higgins (10): vmlinux.lds.h: add linker section for KUnit test suites arch: arm64: add linker section for KUnit test suites arch: microblaze: add linker section for KUnit test suites arch: powerpc: add linker section for KUnit test suites arch: um: add linker section for KUnit test suites arch: xtensa: add linker section for KUnit test suites init: main: add KUnit to kernel init kunit: test: add test plan to KUnit TAP format Documentation: Add kunit_shutdown to kernel-parameters.txt Documentation: kunit: add a brief blurb about kunit_test_suite .../admin-guide/kernel-parameters.txt | 8 ++ Documentation/dev-tools/kunit/usage.rst | 5 ++ arch/arm64/kernel/vmlinux.lds.S | 3 + arch/microblaze/kernel/vmlinux.lds.S | 4 + arch/powerpc/kernel/vmlinux.lds.S | 4 + arch/um/include/asm/common.lds.S | 4 + arch/xtensa/kernel/vmlinux.lds.S | 4 + include/asm-generic/vmlinux.lds.h | 8 ++ include/kunit/test.h | 73 ++++++++++++----- init/main.c | 4 + lib/kunit/Makefile | 3 +- lib/kunit/executor.c | 63 +++++++++++++++ lib/kunit/test.c | 13 +-- tools/testing/kunit/kunit_kernel.py | 2 +- tools/testing/kunit/kunit_parser.py | 74 +++++++++++++++--- .../test_is_test_passed-all_passed.log | Bin 1562 -> 1567 bytes .../test_data/test_is_test_passed-crash.log | Bin 3016 -> 3021 bytes .../test_data/test_is_test_passed-failure.log | Bin 1700 -> 1705 bytes 18 files changed, 226 insertions(+), 46 deletions(-) create mode 100644 lib/kunit/executor.c base-commit: 4333a9b0b67bb4e8bcd91bdd80da80b0ec151162 prerequisite-patch-id: 2d4b5aa9fa8ada9ae04c8584b47c299a822b9455 prerequisite-patch-id: 582b6d9d28ce4b71628890ec832df6522ca68de0 These patches are available for download with dependencies here: https://kunit-review.googlesource.com/c/linux/+/3829 [1] https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan [2] https://patchwork.kernel.org/patch/11383635/