From patchwork Wed Apr 24 23:12:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10915889 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 81A2E161F for ; Wed, 24 Apr 2019 23:13:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 71BD228BD0 for ; Wed, 24 Apr 2019 23:13:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 65E6428BE9; Wed, 24 Apr 2019 23:13:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D0CE928BD0 for ; Wed, 24 Apr 2019 23:13:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727613AbfDXXNT (ORCPT ); Wed, 24 Apr 2019 19:13:19 -0400 Received: from mail-pf1-f195.google.com ([209.85.210.195]:35080 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727381AbfDXXMo (ORCPT ); Wed, 24 Apr 2019 19:12:44 -0400 Received: by mail-pf1-f195.google.com with SMTP id t21so10119184pfh.2 for ; Wed, 24 Apr 2019 16:12:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=XnQBDo9k2oLEZ5F3zS0pOxbIMI+0vYGMgEt5HE+i3VY=; b=cnPSxqo2s6pQjFMgb1bvtkRrgeGU3L4fKJGDRn8gM/+CBe1AikmWM1GOiZrXP4HlFX EVrq3QGX+jFufgJ9cI6lxPWe2dIajCGADpYdjndB6N9ArmNpfPR6XyH8U/+QhBOGkKIW cT6+Fvh6BSaaylJhBQM8P5zsRSCpCzewqcfcw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=XnQBDo9k2oLEZ5F3zS0pOxbIMI+0vYGMgEt5HE+i3VY=; b=e1C4leBtD6HLQCbeT33S+fTBQ8X/PIGhrf0YhNp2PPJdFkNDRjVIn8T6Y4fQU+Gxql 2gsysTeVGwmFYyqlZAaIRJtSrDjUVXaSNlCZx3eN0IIzwTZfX4npGU5puze3y/1n8BTU Jt0zHrQER/Dt4xxqhroHVXAS3giK5LNKo4GU9BCrSDIRMKtaI3/XtSZs/TgxEmJUALI+ LGieO5pcaqol6KDFPp0AswrDgkqkfjsCT4q4LPW2YMi6NMQu1RLYFJHpMAZZSJZP8GNm xpRt0CWbd7xv753uyo2d8XAZETXAelnTnTnt04mdfuliDAr9t16kB70YfzktSj4aBfHW IyPg== X-Gm-Message-State: APjAAAU7YIGpYjE73l1vaGikUXlajXOIOVboRRepAX/27ffdVH+/0r8P sOd0IQ4goZJvWz5vI9j5emZ4MQ== X-Google-Smtp-Source: APXvYqwCSb1YskOnavo70LGN7TD5SJE1/RNSuA9Or7U0zIF27H9lIssGxFJGpmJbSYavJjt6acz10Q== X-Received: by 2002:a62:4e86:: with SMTP id c128mr35880525pfb.39.1556147563094; Wed, 24 Apr 2019 16:12:43 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id 13sm31138871pfi.172.2019.04.24.16.12.40 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 24 Apr 2019 16:12:40 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , Thomas Gleixner , Ingo Molnar , Peter Zijlstra , Darren Hart , Christian Brauner , Tycho Andersen , Serge Hallyn , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/8] selftests: Extract single-test shell logic from lib.mk Date: Wed, 24 Apr 2019 16:12:30 -0700 Message-Id: <20190424231237.14776-2-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190424231237.14776-1-keescook@chromium.org> References: <20190424231237.14776-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In order to improve the reusability of the kselftest test running logic, this extracts the single-test logic from lib.mk into kselftest/runner.sh which lib.mk can call directly. No changes in output. As part of the change, this moves the "summary" Makefile logic around to set a new "logfile" output. This will be used again in the future "emit_tests" target as well. Signed-off-by: Kees Cook --- tools/testing/selftests/.gitignore | 1 - tools/testing/selftests/kselftest/runner.sh | 32 ++++++++++++++++++ tools/testing/selftests/lib.mk | 37 ++++----------------- 3 files changed, 39 insertions(+), 31 deletions(-) create mode 100644 tools/testing/selftests/kselftest/runner.sh diff --git a/tools/testing/selftests/.gitignore b/tools/testing/selftests/.gitignore index 91750352459d..8059ce834247 100644 --- a/tools/testing/selftests/.gitignore +++ b/tools/testing/selftests/.gitignore @@ -1,4 +1,3 @@ -kselftest gpiogpio-event-mon gpiogpio-hammer gpioinclude/ diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh new file mode 100644 index 000000000000..e1117d703887 --- /dev/null +++ b/tools/testing/selftests/kselftest/runner.sh @@ -0,0 +1,32 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# +# Runs a set of tests in a given subdirectory. +export skip_rc=4 +export logfile=/dev/stdout + +run_one() +{ + TEST="$1" + NUM="$2" + + BASENAME_TEST=$(basename $TEST) + + TEST_HDR_MSG="selftests: "`basename $PWD`:" $BASENAME_TEST" + echo "$TEST_HDR_MSG" + echo "========================================" + if [ ! -x "$TEST" ]; then + echo "$TEST_HDR_MSG: Warning: file $TEST is not executable, correct this." + echo "not ok 1..$test_num $TEST_HDR_MSG [FAIL]" + else + cd `dirname $TEST` > /dev/null + (./$BASENAME_TEST >> "$logfile" 2>&1 && + echo "ok 1..$test_num $TEST_HDR_MSG [PASS]") || + (if [ $? -eq $skip_rc ]; then \ + echo "not ok 1..$test_num $TEST_HDR_MSG [SKIP]" + else + echo "not ok 1..$test_num $TEST_HDR_MSG [FAIL]" + fi) + cd - >/dev/null + fi +} diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index 5979fdc4f36c..9d2b3c303bfa 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -14,6 +14,7 @@ ifeq (0,$(MAKELEVEL)) endif endif endif +selfdir = $(realpath $(dir $(filter %/lib.mk,$(MAKEFILE_LIST)))) # The following are built by lib.mk common compile rules. # TEST_CUSTOM_PROGS should be used by tests that require @@ -65,43 +66,19 @@ all: $(TEST_GEN_PROGS) $(TEST_GEN_PROGS_EXTENDED) $(TEST_GEN_FILES) endif .ONESHELL: -define RUN_TEST_PRINT_RESULT - TEST_HDR_MSG="selftests: "`basename $$PWD`:" $$BASENAME_TEST"; \ - echo $$TEST_HDR_MSG; \ - echo "========================================"; \ - if [ ! -x $$TEST ]; then \ - echo "$$TEST_HDR_MSG: Warning: file $$BASENAME_TEST is not executable, correct this.";\ - echo "not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]"; \ - else \ - cd `dirname $$TEST` > /dev/null; \ - if [ "X$(summary)" != "X" ]; then \ - (./$$BASENAME_TEST > /tmp/$$BASENAME_TEST 2>&1 && \ - echo "ok 1..$$test_num $$TEST_HDR_MSG [PASS]") || \ - (if [ $$? -eq $$skip ]; then \ - echo "not ok 1..$$test_num $$TEST_HDR_MSG [SKIP]"; \ - else echo "not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]"; \ - fi;) \ - else \ - (./$$BASENAME_TEST && \ - echo "ok 1..$$test_num $$TEST_HDR_MSG [PASS]") || \ - (if [ $$? -eq $$skip ]; then \ - echo "not ok 1..$$test_num $$TEST_HDR_MSG [SKIP]"; \ - else echo "not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]"; \ - fi;) \ - fi; \ - cd - > /dev/null; \ - fi; -endef - define RUN_TESTS @export KSFT_TAP_LEVEL=`echo 1`; \ test_num=`echo 0`; \ - skip=`echo 4`; \ + . $(selfdir)/kselftest/runner.sh; \ echo "TAP version 13"; \ for TEST in $(1); do \ BASENAME_TEST=`basename $$TEST`; \ test_num=`echo $$test_num+1 | bc`; \ - $(call RUN_TEST_PRINT_RESULT,$(TEST),$(BASENAME_TEST),$(test_num),$(skip)) \ + if [ "X$(summary)" != "X" ]; then \ + logfile="/tmp/$$BASENAME_TEST"; \ + cat /dev/null > "$$logfile"; \ + fi; \ + run_one "$$BASENAME_TEST" "$$test_num"; \ done; endef From patchwork Wed Apr 24 23:12:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10915881 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1AFBE161F for ; Wed, 24 Apr 2019 23:13:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0941C28BD0 for ; Wed, 24 Apr 2019 23:13:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EE8BE28BE9; Wed, 24 Apr 2019 23:13:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 81CA828BD0 for ; Wed, 24 Apr 2019 23:13:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727565AbfDXXNE (ORCPT ); Wed, 24 Apr 2019 19:13:04 -0400 Received: from mail-pl1-f194.google.com ([209.85.214.194]:34648 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727429AbfDXXMo (ORCPT ); Wed, 24 Apr 2019 19:12:44 -0400 Received: by mail-pl1-f194.google.com with SMTP id y6so10082147plt.1 for ; Wed, 24 Apr 2019 16:12:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZuCugwZb+/GRAp2HfwR/1ziBqDVM1XVAZ/RmaQUhBGo=; b=msuNws63xTwU3xDReEFIk60xeSadqexwYZRxp18ms1vdZ5UDDoVCHXjBjBW5ssG4e7 E16EVNFkdb5/tOLeML3XR/kNcV4g2M60Ag60RMJJjEhe27DYnpf0ul9MAyWwPkhVSzyn qhFKdLvmXqElCYnHQaqK5aW6g8o+jMEgmUjto= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ZuCugwZb+/GRAp2HfwR/1ziBqDVM1XVAZ/RmaQUhBGo=; b=QZeBBo6my85sMuhmXth/iiotrtDWaYupPjgwRc39RzMUeI4K0Z4Mzx49b1U99PXaYF a+4C12oP6El6cUW+xxbvam56hYquhB/hc8i7Rqh+39IHC5utHHZ0EHtbBqplMO7YtpdR DS1I+0a5b8OAybrA2wmm6sMo6WYcXQ1fvkz3FlX7DRu0KKXMx9GUTsyEqWJHNMVI5b6U 28H87wBxT9Rqy1nQYzqVwY/lco9JKTXNjb/03eORlDxf1lUCgD5lLVLlJP/jfhc8+Gar OXQ6f+oZDsVgBfMAhMRu6lyqqAO7kSf5oScT9KNhtdC+4YQruzIx0/BOadx3ZNCDujmY VTpQ== X-Gm-Message-State: APjAAAU2giyK8c5kIuza6WfwRgvEieZpExBiqNNgX14BN/GqK8n8mg3+ Jbry9Re3z5bExjntkXNA953lqg== X-Google-Smtp-Source: APXvYqxGjazCTufdmjMBo+HpHsm2QiS4rwC/7Wqx3kyp3G6niUjUe9D3Why2pNQyCXTJaD8lbvB99w== X-Received: by 2002:a17:902:684a:: with SMTP id f10mr9678565pln.286.1556147563691; Wed, 24 Apr 2019 16:12:43 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id q80sm37177424pfa.66.2019.04.24.16.12.40 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 24 Apr 2019 16:12:40 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , Thomas Gleixner , Ingo Molnar , Peter Zijlstra , Darren Hart , Christian Brauner , Tycho Andersen , Serge Hallyn , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/8] selftests: Use runner.sh for emit targets Date: Wed, 24 Apr 2019 16:12:31 -0700 Message-Id: <20190424231237.14776-3-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190424231237.14776-1-keescook@chromium.org> References: <20190424231237.14776-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This reuses the new runner.sh for the emit targets instead of manually running each test via run_kselftest.sh. Signed-off-by: Kees Cook --- tools/testing/selftests/Makefile | 11 +++++------ tools/testing/selftests/lib.mk | 15 ++------------- 2 files changed, 7 insertions(+), 19 deletions(-) diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index f2ebf8cf4686..c5f9f736cdbd 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -176,7 +176,8 @@ ALL_SCRIPT := $(INSTALL_PATH)/run_kselftest.sh install: ifdef INSTALL_PATH @# Ask all targets to install their files - mkdir -p $(INSTALL_PATH) + mkdir -p $(INSTALL_PATH)/kselftest + install -m 744 kselftest/runner.sh $(INSTALL_PATH)/kselftest/ @for TARGET in $(TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ make OUTPUT=$$BUILD_TARGET -C $$TARGET INSTALL_PATH=$(INSTALL_PATH)/$$TARGET install; \ @@ -186,15 +187,13 @@ ifdef INSTALL_PATH echo "#!/bin/sh" > $(ALL_SCRIPT) echo "BASE_DIR=\$$(realpath \$$(dirname \$$0))" >> $(ALL_SCRIPT) echo "cd \$$BASE_DIR" >> $(ALL_SCRIPT) + echo ". ./kselftest/runner.sh" >> $(ALL_SCRIPT) echo "ROOT=\$$PWD" >> $(ALL_SCRIPT) echo "if [ \"\$$1\" = \"--summary\" ]; then" >> $(ALL_SCRIPT) - echo " OUTPUT=\$$BASE_DIR/output.log" >> $(ALL_SCRIPT) - echo " cat /dev/null > \$$OUTPUT" >> $(ALL_SCRIPT) - echo "else" >> $(ALL_SCRIPT) - echo " OUTPUT=/dev/stdout" >> $(ALL_SCRIPT) + echo " logfile=\$$BASE_DIR/output.log" >> $(ALL_SCRIPT) + echo " cat /dev/null > \$$logfile" >> $(ALL_SCRIPT) echo "fi" >> $(ALL_SCRIPT) echo "export KSFT_TAP_LEVEL=1" >> $(ALL_SCRIPT) - echo "export skip=4" >> $(ALL_SCRIPT) for TARGET in $(TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index 9d2b3c303bfa..6b2d026a94ea 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -116,24 +116,13 @@ else $(error Error: set INSTALL_PATH to use install) endif -define EMIT_TESTS +emit_tests: @test_num=`echo 0`; \ for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \ BASENAME_TEST=`basename $$TEST`; \ test_num=`echo $$test_num+1 | bc`; \ - TEST_HDR_MSG="selftests: "`basename $$PWD`:" $$BASENAME_TEST"; \ - echo "echo $$TEST_HDR_MSG"; \ - if [ ! -x $$TEST ]; then \ - echo "echo \"$$TEST_HDR_MSG: Warning: file $$BASENAME_TEST is not executable, correct this.\""; \ - echo "echo \"not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]\""; \ - else - echo "(./$$BASENAME_TEST >> \$$OUTPUT 2>&1 && echo \"ok 1..$$test_num $$TEST_HDR_MSG [PASS]\") || (if [ \$$? -eq \$$skip ]; then echo \"not ok 1..$$test_num $$TEST_HDR_MSG [SKIP]\"; else echo \"not ok 1..$$test_num $$TEST_HDR_MSG [FAIL]\"; fi;)"; \ - fi; \ + echo "run_one \"$$BASENAME_TEST\" \"$$test_num\""; \ done; -endef - -emit_tests: - $(EMIT_TESTS) # define if isn't already. It is undefined in make O= case. ifeq ($(RM),) From patchwork Wed Apr 24 23:12:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10915887 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0C443161F for ; Wed, 24 Apr 2019 23:13:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EF46628BD0 for ; Wed, 24 Apr 2019 23:13:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E396828BE9; Wed, 24 Apr 2019 23:13:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 680E328BD0 for ; Wed, 24 Apr 2019 23:13:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727616AbfDXXNT (ORCPT ); Wed, 24 Apr 2019 19:13:19 -0400 Received: from mail-pl1-f195.google.com ([209.85.214.195]:34648 "EHLO mail-pl1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727411AbfDXXMn (ORCPT ); Wed, 24 Apr 2019 19:12:43 -0400 Received: by mail-pl1-f195.google.com with SMTP id y6so10082124plt.1 for ; Wed, 24 Apr 2019 16:12:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MWO4KenSHR25slNpTCUIhefVFREevGFRFzkC3RHE20M=; b=PrZ7aZVfgBiuZtYe4PrbOKtp4idfOdZwylzuSDPuhfaByEedQaMRp7lp+Pd4iBhZQ7 DVmaJzF1UIzikdCp4jleLb55o8m8Kn5fZ/sj6boyac+PReSOkQfTT7vRwm+gHfVOIe0F DTY6AUYfSPYHDKkeQ+fc6tlPHng3B5NQdxg8o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MWO4KenSHR25slNpTCUIhefVFREevGFRFzkC3RHE20M=; b=eW3HH+ktjh+VjuMxo8Uxq6awHUTwJlTn5SQsYCqDzzeJmjv5HtzCw9V0JWHd7H20ex JX0g7fsB1yctUsysW06ZYej4lWHZ+0wJW0pcaoOIpYMeOteYJJCL83tTESMUMzfiaZIq Xyxi3L2+i0PDaSXoxNCRAQahLU4ZsQJBl9jlvZ/xSgbbgGbRhww3jBvHz9XPjSVFENTk UM37+zwCm7hFyF7la/cG+chySScdaSDN9zQmcy4KSR8f1Ax4CglqU1xD5CCQVTxLj5WD 8nU3+JEKx3gJ7cfJiGg3RKcCEILNcSFyeCHKW9JdaiZZCCslUtQ0ePSSgjOckY3M8WJd 9f2w== X-Gm-Message-State: APjAAAXOkKZAsOmYHxj+0oR2vuO1sZcljpebDifXoXZZiB/g6JnfKxeH 8HJ1S72o23rLsKoAiLwGtMFnxA== X-Google-Smtp-Source: APXvYqxkX+7q5cK6Y/F4x3Q++0IUyKNree3PZVatnmatqo691HD28fZot/p2tmE8gNTbnpiJCJ3Abg== X-Received: by 2002:a17:902:bd94:: with SMTP id q20mr5134764pls.334.1556147562468; Wed, 24 Apr 2019 16:12:42 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id o81sm34003162pfa.156.2019.04.24.16.12.40 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 24 Apr 2019 16:12:40 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , Thomas Gleixner , Ingo Molnar , Peter Zijlstra , Darren Hart , Christian Brauner , Tycho Andersen , Serge Hallyn , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 3/8] selftests: Extract logic for multiple test runs Date: Wed, 24 Apr 2019 16:12:32 -0700 Message-Id: <20190424231237.14776-4-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190424231237.14776-1-keescook@chromium.org> References: <20190424231237.14776-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This moves the logic for running multiple tests into a single "run_many" function of runner.sh. Both "run_tests" and "emit_tests" are modified to use it. Summary handling is now controlled by the "per_test_logging" shell flag. Signed-off-by: Kees Cook --- tools/testing/selftests/Makefile | 6 ++--- tools/testing/selftests/kselftest/runner.sh | 25 ++++++++++++++++++--- tools/testing/selftests/lib.mk | 25 +++++++-------------- 3 files changed, 32 insertions(+), 24 deletions(-) diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index c5f9f736cdbd..4ac1d1c7ce5b 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -193,16 +193,14 @@ ifdef INSTALL_PATH echo " logfile=\$$BASE_DIR/output.log" >> $(ALL_SCRIPT) echo " cat /dev/null > \$$logfile" >> $(ALL_SCRIPT) echo "fi" >> $(ALL_SCRIPT) - echo "export KSFT_TAP_LEVEL=1" >> $(ALL_SCRIPT) for TARGET in $(TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ - echo "echo ; echo TAP version 13" >> $(ALL_SCRIPT); \ - echo "echo Running tests in $$TARGET" >> $(ALL_SCRIPT); \ - echo "echo ========================================" >> $(ALL_SCRIPT); \ echo "[ -w /dev/kmsg ] && echo \"kselftest: Running tests in $$TARGET\" >> /dev/kmsg" >> $(ALL_SCRIPT); \ echo "cd $$TARGET" >> $(ALL_SCRIPT); \ + echo -n "run_many" >> $(ALL_SCRIPT); \ make -s --no-print-directory OUTPUT=$$BUILD_TARGET -C $$TARGET emit_tests >> $(ALL_SCRIPT); \ + echo "" >> $(ALL_SCRIPT); \ echo "cd \$$ROOT" >> $(ALL_SCRIPT); \ done; diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index e1117d703887..f12b0a631273 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -2,17 +2,20 @@ # SPDX-License-Identifier: GPL-2.0 # # Runs a set of tests in a given subdirectory. +export KSFT_TAP_LEVEL=1 export skip_rc=4 export logfile=/dev/stdout +export per_test_logging= run_one() { - TEST="$1" - NUM="$2" + DIR="$1" + TEST="$2" + NUM="$3" BASENAME_TEST=$(basename $TEST) - TEST_HDR_MSG="selftests: "`basename $PWD`:" $BASENAME_TEST" + TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST" echo "$TEST_HDR_MSG" echo "========================================" if [ ! -x "$TEST" ]; then @@ -30,3 +33,19 @@ run_one() cd - >/dev/null fi } + +run_many() +{ + echo "TAP version 13" + DIR=$(basename "$PWD") + test_num=0 + for TEST in "$@"; do + BASENAME_TEST=$(basename $TEST) + test_num=$(( test_num + 1 )) + if [ -n "$per_test_logging" ]; then + logfile="/tmp/$BASENAME_TEST" + cat /dev/null > "$logfile" + fi + run_one "$DIR" "$TEST" "$test_num" + done +} diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index 6b2d026a94ea..28b8ffedfdf1 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -67,19 +67,11 @@ endif .ONESHELL: define RUN_TESTS - @export KSFT_TAP_LEVEL=`echo 1`; \ - test_num=`echo 0`; \ - . $(selfdir)/kselftest/runner.sh; \ - echo "TAP version 13"; \ - for TEST in $(1); do \ - BASENAME_TEST=`basename $$TEST`; \ - test_num=`echo $$test_num+1 | bc`; \ - if [ "X$(summary)" != "X" ]; then \ - logfile="/tmp/$$BASENAME_TEST"; \ - cat /dev/null > "$$logfile"; \ - fi; \ - run_one "$$BASENAME_TEST" "$$test_num"; \ - done; + @. $(selfdir)/kselftest/runner.sh; \ + if [ "X$(summary)" != "X" ]; then \ + per_test_logging=1; \ + fi; \ + run_many $(1) endef run_tests: all @@ -117,12 +109,11 @@ else endif emit_tests: - @test_num=`echo 0`; \ for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \ BASENAME_TEST=`basename $$TEST`; \ - test_num=`echo $$test_num+1 | bc`; \ - echo "run_one \"$$BASENAME_TEST\" \"$$test_num\""; \ - done; + echo " \\"; \ + echo -n " \"$$BASENAME_TEST\""; \ + done; \ # define if isn't already. It is undefined in make O= case. ifeq ($(RM),) From patchwork Wed Apr 24 23:12:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10915885 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 75A3E922 for ; Wed, 24 Apr 2019 23:13:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 64D5828BD0 for ; Wed, 24 Apr 2019 23:13:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5936B28BE9; Wed, 24 Apr 2019 23:13:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EF1BC28BE1 for ; Wed, 24 Apr 2019 23:13:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727556AbfDXXNE (ORCPT ); Wed, 24 Apr 2019 19:13:04 -0400 Received: from mail-pg1-f194.google.com ([209.85.215.194]:46818 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727450AbfDXXMp (ORCPT ); Wed, 24 Apr 2019 19:12:45 -0400 Received: by mail-pg1-f194.google.com with SMTP id n2so1018245pgg.13 for ; Wed, 24 Apr 2019 16:12:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=iTzC2LknEGq+dSiv23e3s6QdQpr5UW6imYhTJ6A60aI=; b=KjrT1fxDoT0McG3ptEjcZjwYX/M2l3/ZOctARJxQcFagSvVVB/jkPqLKNlamhaNS6h M0mbC5ceW4bdap8uq1JnMPinYl4aPpAjkxlZkEKua5P3ZdDU9QQg/6hn752sJJ9PBeba 2cPSgtD7SDtZTj0niI/dyeMmj1Mhj3VnDfgRU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=iTzC2LknEGq+dSiv23e3s6QdQpr5UW6imYhTJ6A60aI=; b=LjEXpaEl9aZT/+c71RPERzNDZALdCOtb1nzIaEc7rYY2EIdKoB9wZNQMpJv6l6GtJW WBgmb5NhyeAfDnXkuW2mMNuRcVE/TevDcnwMfcRrkKUxBhsQvVuwQ4WHB/HNHVQ0qFt3 mBFHrPQRCyYLNb9tRtOlRhF1jvrY+ahzBK18URfpULYH8dqxY/5ye+oXBu++LdhSUlYO Px4R71DbXuzxUUJqtnVqxBCNoY9+sVhiVJ1WN518+3fQK98Fni5RcgzGXfFVk7mNZFrx kD38vB3KqETwLUu6tLRJEPZVqRHSH+Q0EiUcw9Jrm44MmuqKFOWzLHbjCzthI6XTB6I3 mvHw== X-Gm-Message-State: APjAAAXRYeL/KsdDRpcnXQjuHex+2w1S6e62hG1LmAnB/uyIlMyvdz08 XrVOWCmwLV47g8Ycz+ZkmdnhcA== X-Google-Smtp-Source: APXvYqx0reCRfsMOvXF9jnXzGq1YqJDFDpM9afbkbk08bZjtkLPY3rbMc8ftRSFPNH7mhTjh8gagNg== X-Received: by 2002:a63:5041:: with SMTP id q1mr10368034pgl.386.1556147564334; Wed, 24 Apr 2019 16:12:44 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id p6sm25076749pfd.122.2019.04.24.16.12.40 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 24 Apr 2019 16:12:40 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , Thomas Gleixner , Ingo Molnar , Peter Zijlstra , Darren Hart , Christian Brauner , Tycho Andersen , Serge Hallyn , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 4/8] selftests: Add plan line and fix result line syntax Date: Wed, 24 Apr 2019 16:12:33 -0700 Message-Id: <20190424231237.14776-5-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190424231237.14776-1-keescook@chromium.org> References: <20190424231237.14776-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The TAP version 13 spec requires a "plan" line, which has been missing. Since we always know how many tests we're going to run, emit the count on the plan line. This also fixes the result lines to remove the "1.." prefix which is against spec, and to mark skips with the correct "# SKIP" suffix. Signed-off-by: Kees Cook --- tools/testing/selftests/kselftest.h | 4 ++-- tools/testing/selftests/kselftest/runner.sh | 10 ++++++---- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/kselftest.h b/tools/testing/selftests/kselftest.h index 47e1d995c182..9f4147a6fdbc 100644 --- a/tools/testing/selftests/kselftest.h +++ b/tools/testing/selftests/kselftest.h @@ -111,7 +111,7 @@ static inline void ksft_test_result_skip(const char *msg, ...) ksft_cnt.ksft_xskip++; va_start(args, msg); - printf("ok %d # skip ", ksft_test_num()); + printf("not ok %d # SKIP ", ksft_test_num()); vprintf(msg, args); va_end(args); } @@ -172,7 +172,7 @@ static inline int ksft_exit_skip(const char *msg, ...) va_list args; va_start(args, msg); - printf("1..%d # Skipped: ", ksft_test_num()); + printf("not ok %d # SKIP ", ksft_test_num()); vprintf(msg, args); va_end(args); } else { diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index f12b0a631273..e0621974e01e 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -20,15 +20,15 @@ run_one() echo "========================================" if [ ! -x "$TEST" ]; then echo "$TEST_HDR_MSG: Warning: file $TEST is not executable, correct this." - echo "not ok 1..$test_num $TEST_HDR_MSG [FAIL]" + echo "not ok $test_num $TEST_HDR_MSG" else cd `dirname $TEST` > /dev/null (./$BASENAME_TEST >> "$logfile" 2>&1 && - echo "ok 1..$test_num $TEST_HDR_MSG [PASS]") || + echo "ok $test_num $TEST_HDR_MSG") || (if [ $? -eq $skip_rc ]; then \ - echo "not ok 1..$test_num $TEST_HDR_MSG [SKIP]" + echo "not ok $test_num $TEST_HDR_MSG # SKIP" else - echo "not ok 1..$test_num $TEST_HDR_MSG [FAIL]" + echo "not ok $test_num $TEST_HDR_MSG" fi) cd - >/dev/null fi @@ -39,6 +39,8 @@ run_many() echo "TAP version 13" DIR=$(basename "$PWD") test_num=0 + total=$(echo "$@" | wc -w) + echo "1..$total" for TEST in "$@"; do BASENAME_TEST=$(basename $TEST) test_num=$(( test_num + 1 )) From patchwork Wed Apr 24 23:12:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10915883 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 45A76161F for ; Wed, 24 Apr 2019 23:13:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3573728BD0 for ; Wed, 24 Apr 2019 23:13:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 29F4228BEA; Wed, 24 Apr 2019 23:13:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C695C28BD0 for ; Wed, 24 Apr 2019 23:13:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727450AbfDXXNE (ORCPT ); Wed, 24 Apr 2019 19:13:04 -0400 Received: from mail-pg1-f194.google.com ([209.85.215.194]:45086 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727452AbfDXXMp (ORCPT ); Wed, 24 Apr 2019 19:12:45 -0400 Received: by mail-pg1-f194.google.com with SMTP id y3so10139412pgk.12 for ; Wed, 24 Apr 2019 16:12:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=KIorirH95SzVwrqSDe1yVALnBy6zYImlloz9Cy/V4l4=; b=H4dUFSl30RUmaew5ui9/relos6unEywoW37la27lK9cd1+Ze/AxlvlKLRxj24we6C+ tZ201qsAlJK3WTntDFI8qhBgybh1CO4s3IXFBqF3sZT578Eds4gkxzEm6LGg2dH9Q6Yj 8dFQKXkBerCOUfVPHqPa56Yv+QHQE7hE8VjOw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=KIorirH95SzVwrqSDe1yVALnBy6zYImlloz9Cy/V4l4=; b=lQ/T2JXJ6qrzfQFBNKci0n4O9Hb8xn+iZY2/n1UX/KO+BKt17ZX1gNsRGFm3WO4VwS SA/rHmU11qiZ4xEdw6JWS7XRYhfNRT1Eff4XD5Z36U9SmvCN6V8ItXbhLJI8UHP7XSGa 0xsbyE57uitjvMpp/HyxRrgP7ng+xwO+QWLjyLiqdC+iMMOkEE/a9z0t8MBOaXuOO+3F +7QcUXVylYmpkRX59arnqPzaSQcdMefhvEDrwO/bLAbM6EWH9HUeJath9/ZxS+sAY8AW imE+tLZcFsmDaquPU350faXeNX26Ss/eyZh1Wpga86DMZ/SUWziJyyhg8KH1R3jlWeUF Y/sg== X-Gm-Message-State: APjAAAVmXI306oMW581E/5qZq9Wk9BwyZyoDluI0A+dL+xI5IreDuLKr S/bgw88gh2rxPKSexLiyVJPITg== X-Google-Smtp-Source: APXvYqwL78F7iJIpuPRIG7HbqT8j/8lHbv+3i0+DrPZ4uXrUKcpW0lwg3z7IZ71eb6qyC4QLPtcwkQ== X-Received: by 2002:a63:d908:: with SMTP id r8mr186591pgg.268.1556147565115; Wed, 24 Apr 2019 16:12:45 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id f87sm32642931pff.56.2019.04.24.16.12.42 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 24 Apr 2019 16:12:44 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , Thomas Gleixner , Ingo Molnar , Peter Zijlstra , Darren Hart , Christian Brauner , Tycho Andersen , Serge Hallyn , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 5/8] selftests: Distinguish between missing and non-executable Date: Wed, 24 Apr 2019 16:12:34 -0700 Message-Id: <20190424231237.14776-6-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190424231237.14776-1-keescook@chromium.org> References: <20190424231237.14776-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If a test was missing (e.g. wrong architecture, etc), the test runner would incorrectly claim the test was non-executable. This adds an existence check to report correctly. Signed-off-by: Kees Cook --- tools/testing/selftests/kselftest/runner.sh | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index e0621974e01e..a66fb64e61e9 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -19,7 +19,12 @@ run_one() echo "$TEST_HDR_MSG" echo "========================================" if [ ! -x "$TEST" ]; then - echo "$TEST_HDR_MSG: Warning: file $TEST is not executable, correct this." + echo -n "$TEST_HDR_MSG: Warning: file $TEST is " + if [ ! -e "$TEST" ]; then + echo "missing!" + else + echo "not executable, correct this." + fi echo "not ok $test_num $TEST_HDR_MSG" else cd `dirname $TEST` > /dev/null From patchwork Wed Apr 24 23:12:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10915875 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 99244922 for ; Wed, 24 Apr 2019 23:12:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8543F28BE1 for ; Wed, 24 Apr 2019 23:12:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 77F7D28BEC; Wed, 24 Apr 2019 23:12:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D799528BE1 for ; Wed, 24 Apr 2019 23:12:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727499AbfDXXMs (ORCPT ); Wed, 24 Apr 2019 19:12:48 -0400 Received: from mail-pl1-f195.google.com ([209.85.214.195]:36823 "EHLO mail-pl1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727463AbfDXXMr (ORCPT ); Wed, 24 Apr 2019 19:12:47 -0400 Received: by mail-pl1-f195.google.com with SMTP id ck15so10072565plb.3 for ; Wed, 24 Apr 2019 16:12:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LLIDRHHL5EHjKhGJrHYjqstCVGlQWoi012j6Tt+ZkY4=; b=eOtE+N0wZVLaDwB/A4k/a0q/b9sZYHHnmWWb9fIGjnbYSOLrifwUlD5RsWodLfTWRj 9MAKVsxERfAPEQtBv+17N58usVjYxgaH/8uj5z4Y1/haLUWKZrZmqlxkEeA3IQi807Q3 5bjIcOFVh1oM3hMEhBu3rSVctxsQnKV7YEBzo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=LLIDRHHL5EHjKhGJrHYjqstCVGlQWoi012j6Tt+ZkY4=; b=MannbodmM1ch41I5CsQudMxt4LfjvB67jqed+GbU3dhL6ASV6CZ473AF+K5mMxAc70 e28fs9+sdae6nXi6Jq09yM19c64yjXKqrM6/ESz/DabMTYFwxSRjV7Tw9EoA4I14PSi3 y+ROb1EQGC4pFL34D6wkqZ64qt9pCsmZNdMYMU79i1NL7eGLztb5Y/LqeuOEzsCPIX8r ZF7iHXT/jqzWqJ/uKTcga/TFaUg5cQiDyQdcOaxGIDiWRhIXT8Mxf+gjPDkPbNemzXUw 3vzTThtHqxvdVdsYIc3Sddew8B7ndLV+NP3YFaDxKnaJANjlzSXXEf62iLYa8KAu/vzt hHsQ== X-Gm-Message-State: APjAAAXqHFnMBWzU0ZZXUMAcyTD2Njm2rIKPH/vh24MxFWk6QXF/vlrG a/1NTaPXV1zmIJiXV2ypHsr7lw== X-Google-Smtp-Source: APXvYqyfprOWMCVssgGTdzMisOSgF25PUjM9K31waTxT42gCRt2KNaYzKW2FadwROfobnBuQMUE8gA== X-Received: by 2002:a17:902:8a8b:: with SMTP id p11mr34776188plo.227.1556147566226; Wed, 24 Apr 2019 16:12:46 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id w3sm35226676pfn.179.2019.04.24.16.12.42 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 24 Apr 2019 16:12:44 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , Thomas Gleixner , Ingo Molnar , Peter Zijlstra , Darren Hart , Christian Brauner , Tycho Andersen , Serge Hallyn , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 6/8] selftests: Move test output to diagnostic lines Date: Wed, 24 Apr 2019 16:12:35 -0700 Message-Id: <20190424231237.14776-7-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190424231237.14776-1-keescook@chromium.org> References: <20190424231237.14776-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This changes the selftest output so that each test's output is prefixed with "# " as a TAP "diagnostic line". This creates a bit of a kernel-specific TAP dialect where the diagnostics precede the results. The TAP spec isn't entirely clear about this, though, so I think it's the correct solution so as to keep interactive runs making sense. If the output _followed_ the result line in the spec-suggested YAML form, each test would dump all of its output at once instead of as it went, making debugging harder. This does, however, solve the recursive TAP output problem, as sub-tests will simply be prefixed by "# ". Parsing sub-tests becomes a simple problem of just removing the first two characters of a given top-level test's diagnostic output, and parsing the results. Note that the shell construct needed to both get an exit code from the first command in a pipe and still filter the pipe (to add the "# " prefix) uses a POSIX solution rather than the bash "pipefail" option which is not supported by dash. Since some test environments may have a very minimal set of utilities available, the new prefixing code will fall back to doing line-at-a-time prefixing if perl and/or stdbuf are not available. Signed-off-by: Kees Cook --- tools/testing/selftests/Makefile | 1 + tools/testing/selftests/kselftest.h | 2 +- tools/testing/selftests/kselftest/prefix.pl | 23 +++++++++++++ tools/testing/selftests/kselftest/runner.sh | 37 ++++++++++++++++++--- tools/testing/selftests/lib.mk | 3 +- 5 files changed, 60 insertions(+), 6 deletions(-) create mode 100755 tools/testing/selftests/kselftest/prefix.pl diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 4ac1d1c7ce5b..64699f59b95f 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -178,6 +178,7 @@ ifdef INSTALL_PATH @# Ask all targets to install their files mkdir -p $(INSTALL_PATH)/kselftest install -m 744 kselftest/runner.sh $(INSTALL_PATH)/kselftest/ + install -m 744 kselftest/prefix.pl $(INSTALL_PATH)/kselftest/ @for TARGET in $(TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ make OUTPUT=$$BUILD_TARGET -C $$TARGET INSTALL_PATH=$(INSTALL_PATH)/$$TARGET install; \ diff --git a/tools/testing/selftests/kselftest.h b/tools/testing/selftests/kselftest.h index 9f4147a6fdbc..7f078e79a9fa 100644 --- a/tools/testing/selftests/kselftest.h +++ b/tools/testing/selftests/kselftest.h @@ -63,7 +63,7 @@ static inline void ksft_print_header(void) static inline void ksft_print_cnts(void) { - printf("Pass %d Fail %d Xfail %d Xpass %d Skip %d Error %d\n", + printf("# Pass %d Fail %d Xfail %d Xpass %d Skip %d Error %d\n", ksft_cnt.ksft_pass, ksft_cnt.ksft_fail, ksft_cnt.ksft_xfail, ksft_cnt.ksft_xpass, ksft_cnt.ksft_xskip, ksft_cnt.ksft_error); diff --git a/tools/testing/selftests/kselftest/prefix.pl b/tools/testing/selftests/kselftest/prefix.pl new file mode 100755 index 000000000000..ec7e48118183 --- /dev/null +++ b/tools/testing/selftests/kselftest/prefix.pl @@ -0,0 +1,23 @@ +#!/usr/bin/perl +# SPDX-License-Identifier: GPL-2.0 +# Prefix all lines with "# ", unbuffered. Command being piped in may need +# to have unbuffering forced with "stdbuf -i0 -o0 -e0 $cmd". +use strict; + +binmode STDIN; +binmode STDOUT; + +STDOUT->autoflush(1); + +my $needed = 1; +while (1) { + my $char; + my $bytes = sysread(STDIN, $char, 1); + exit 0 if ($bytes == 0); + if ($needed) { + print "# "; + $needed = 0; + } + print $char; + $needed = 1 if ($char eq "\n"); +} diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index a66fb64e61e9..b9f74e5a2ee5 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -7,6 +7,34 @@ export skip_rc=4 export logfile=/dev/stdout export per_test_logging= +# There isn't a shell-agnostic way to find the path of a sourced file, +# so we must rely on BASE_DIR being set to find other tools. +if [ -z "$BASE_DIR" ]; then + echo "Error: BASE_DIR must be set before sourcing." >&2 + exit 1 +fi + +# If Perl is unavailable, we must fall back to line-at-a-time prefixing +# with sed instead of unbuffered output. +tap_prefix() +{ + if [ ! -x /usr/bin/perl ]; then + sed -e 's/^/# /' + else + "$BASE_DIR"/kselftest/prefix.pl + fi +} + +# If stdbuf is unavailable, we must fall back to line-at-a-time piping. +tap_unbuffer() +{ + if ! which stdbuf >/dev/null ; then + "$@" + else + stdbuf -i0 -o0 -e0 "$@" + fi +} + run_one() { DIR="$1" @@ -16,10 +44,9 @@ run_one() BASENAME_TEST=$(basename $TEST) TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST" - echo "$TEST_HDR_MSG" - echo "========================================" + echo "# $TEST_HDR_MSG" if [ ! -x "$TEST" ]; then - echo -n "$TEST_HDR_MSG: Warning: file $TEST is " + echo -n "# Warning: file $TEST is " if [ ! -e "$TEST" ]; then echo "missing!" else @@ -28,7 +55,9 @@ run_one() echo "not ok $test_num $TEST_HDR_MSG" else cd `dirname $TEST` > /dev/null - (./$BASENAME_TEST >> "$logfile" 2>&1 && + (((((tap_unbuffer ./$BASENAME_TEST 2>&1; echo $? >&3) | + tap_prefix >&4) 3>&1) | + (read xs; exit $xs)) 4>>"$logfile" && echo "ok $test_num $TEST_HDR_MSG") || (if [ $? -eq $skip_rc ]; then \ echo "not ok $test_num $TEST_HDR_MSG # SKIP" diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index 28b8ffedfdf1..098dd0065fb1 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -67,7 +67,8 @@ endif .ONESHELL: define RUN_TESTS - @. $(selfdir)/kselftest/runner.sh; \ + @BASE_DIR="$(selfdir)"; \ + . $(selfdir)/kselftest/runner.sh; \ if [ "X$(summary)" != "X" ]; then \ per_test_logging=1; \ fi; \ From patchwork Wed Apr 24 23:12:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10915879 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BD9B7922 for ; Wed, 24 Apr 2019 23:13:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ABE4728BD0 for ; Wed, 24 Apr 2019 23:13:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A06B628BE9; Wed, 24 Apr 2019 23:13:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 43A7128BD0 for ; Wed, 24 Apr 2019 23:13:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727474AbfDXXMq (ORCPT ); Wed, 24 Apr 2019 19:12:46 -0400 Received: from mail-pf1-f196.google.com ([209.85.210.196]:41724 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727463AbfDXXMq (ORCPT ); Wed, 24 Apr 2019 19:12:46 -0400 Received: by mail-pf1-f196.google.com with SMTP id 188so10100487pfd.8 for ; Wed, 24 Apr 2019 16:12:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=USyjebKhKCZCLrhUyqlX69vkoqbsFxWhHCmLAtiPTjk=; b=LrjzKFxVODV8Pr1dnRea7pyPGeBJmS9eBnfVmJ67fomco/hgGamwbbGddXrpUQ5OQ1 cdhpoLVzPGviAFCFtMaF3mNFXVCW5MWqQCcNTRs9OtEyYC710xTyziMDzWkwNiE6mZol KhVfqc9y7DRORxOLfn28xAAdbUTtcUbEkoax0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=USyjebKhKCZCLrhUyqlX69vkoqbsFxWhHCmLAtiPTjk=; b=m/XXYJ9Is/22UXKKPoKizQJWFNkYrG5yHpWPuoFms548BXZHv2ZV2B7eD6hLg5GoQS 2iL9heRUJteD1hGr49JvLmOxBggHhndcbXYrxjUzsJhchh5ydLvgnibeEkJtrScJ/kGW RdSXBreGWd4zcKGtmZpsZvV7Kxx46fs9iD4lOE5g+xvSs1hQe7c8HVh5D8NpYWK6GwI3 COCIayWrUP3ztwBim4RkZxL9gkyoQklmQI45oKrr7BDeucTbSsoPe2kSac2FQv5sG2Rs klILMy+Bmsatnp9NTFg9geGXziKJssw+K/coy22wAf9YE7KRVj2pq96RsqEcZQ4ZYgia uVOA== X-Gm-Message-State: APjAAAWU6QFZWAw0+7GEc1x1hfTPqx+j/cGF1Xm8kWQdRU6yYyqMlvcA U/9SmlxS5cGegUTF1fY1fDddXA== X-Google-Smtp-Source: APXvYqycbCdQbeLJP1q3+Fn6T26GnvkkzbCWr10Sjc48ytdPZn5nYqG/blhh4GIlO94nIyHI3nBTEw== X-Received: by 2002:a62:474a:: with SMTP id u71mr35333692pfa.87.1556147565647; Wed, 24 Apr 2019 16:12:45 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id a6sm39947726pgd.67.2019.04.24.16.12.43 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 24 Apr 2019 16:12:44 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , Thomas Gleixner , Ingo Molnar , Peter Zijlstra , Darren Hart , Christian Brauner , Tycho Andersen , Serge Hallyn , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 7/8] selftests: Remove KSFT_TAP_LEVEL Date: Wed, 24 Apr 2019 16:12:36 -0700 Message-Id: <20190424231237.14776-8-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190424231237.14776-1-keescook@chromium.org> References: <20190424231237.14776-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since sub-testing can now be detected by indentation level, this removes KSFT_TAP_LEVEL so that subtests report their TAP header for later parsing. Signed-off-by: Kees Cook --- tools/testing/selftests/Makefile | 6 ------ tools/testing/selftests/kselftest/runner.sh | 1 - 2 files changed, 7 deletions(-) diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 64699f59b95f..9f05448e5e4b 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -86,12 +86,6 @@ else endif endif -# KSFT_TAP_LEVEL is used from KSFT framework to prevent nested TAP header -# printing from tests. Applicable to run_tests case where run_tests adds -# TAP header prior running tests and when a test program invokes another -# with system() call. Export it here to cover override RUN_TESTS defines. -export KSFT_TAP_LEVEL=`echo 1` - # Prepare for headers install top_srcdir ?= ../../.. include $(top_srcdir)/scripts/subarch.include diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh index b9f74e5a2ee5..eff3ee303d0d 100644 --- a/tools/testing/selftests/kselftest/runner.sh +++ b/tools/testing/selftests/kselftest/runner.sh @@ -2,7 +2,6 @@ # SPDX-License-Identifier: GPL-2.0 # # Runs a set of tests in a given subdirectory. -export KSFT_TAP_LEVEL=1 export skip_rc=4 export logfile=/dev/stdout export per_test_logging= From patchwork Wed Apr 24 23:12:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 10915877 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 81A12922 for ; Wed, 24 Apr 2019 23:12:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7019428BD0 for ; Wed, 24 Apr 2019 23:12:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6471D28BE9; Wed, 24 Apr 2019 23:12:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8C9AA28BD0 for ; Wed, 24 Apr 2019 23:12:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727463AbfDXXMs (ORCPT ); Wed, 24 Apr 2019 19:12:48 -0400 Received: from mail-pf1-f195.google.com ([209.85.210.195]:36264 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727492AbfDXXMs (ORCPT ); Wed, 24 Apr 2019 19:12:48 -0400 Received: by mail-pf1-f195.google.com with SMTP id z5so10109228pfn.3 for ; Wed, 24 Apr 2019 16:12:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0xopY1lmZ8N5OeEKtF78e6u48ARcCoPF6av4+vwYGRo=; b=fuH9kVme655YR5+nnR2nihLgcQ86AQlQoJdRaP1bm40T75402xyGkug5CGctC0ndSq h/ZAvfTG9ylfp3Oun4SpdYTeppMljuo9s9wlJucapoqouh9hmeZhZSBt7dNIFWqYscTP L+EYYBsQWUzVrkFdFMfe9YGvfPmsm+eH8sYwY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0xopY1lmZ8N5OeEKtF78e6u48ARcCoPF6av4+vwYGRo=; b=NsT6xjgdBW6WxP971O7JQ1/Q3Z7W2O8GYJ0N+90ml9lMFj94VPWqwJj3vAOPAg2oa2 bvtlBT6ZH/YIAbAlcCyQRh9TozgFWZFI6In4+j8N4u2iqqaHAeM+82pqDfR/vfQUAU3p p+RPZCWWC/tph9iuoUPJq5r/bc0UNy3mDo2CsuhZW5r1EUL++ah+6uFd2UrFVJxsXRW1 wQg1FXgVPRp3zi9WRCKvfhC/0i7b2MNM/d0x80uNtBzh0hIy8CEIVSmkChzHCNKaUI/x hDUKuewtBIgUMHF0IwMfiOLGuMWI9JqlUOqksXRRpT5it8Xf9EGEq1DWHR2TYUyzSi/e iSzQ== X-Gm-Message-State: APjAAAXsOT8UGSgM+JrnCexxJdn5mGC/cL5kGxvgef4UJucfZA7849Hi wuVHDR6nB/2wDz5FEsvhr23Rvg== X-Google-Smtp-Source: APXvYqzcqsWTHWTUtQU4LHsPUM4bh0obWKd8JT1CxFHpaiQ+cRNdj4bl4HkHA6mXfHdwDDcFt/NJZA== X-Received: by 2002:a65:518d:: with SMTP id h13mr33759514pgq.259.1556147566777; Wed, 24 Apr 2019 16:12:46 -0700 (PDT) Received: from www.outflux.net (173-164-112-133-Oregon.hfc.comcastbusiness.net. [173.164.112.133]) by smtp.gmail.com with ESMTPSA id j22sm27387266pfn.129.2019.04.24.16.12.42 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 24 Apr 2019 16:12:44 -0700 (PDT) From: Kees Cook To: Shuah Khan Cc: Kees Cook , Thomas Gleixner , Ingo Molnar , Peter Zijlstra , Darren Hart , Christian Brauner , Tycho Andersen , Serge Hallyn , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 8/8] selftests: Add test plan API to kselftest.h and adjust callers Date: Wed, 24 Apr 2019 16:12:37 -0700 Message-Id: <20190424231237.14776-9-keescook@chromium.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190424231237.14776-1-keescook@chromium.org> References: <20190424231237.14776-1-keescook@chromium.org> Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The test plan for TAP needs to be declared immediately after the header. This adds the test plan API to kselftest.h and updates all callers to declare their expected test counts. Signed-off-by: Kees Cook --- .../selftests/breakpoints/breakpoint_test.c | 15 ++++++++++++--- .../selftests/breakpoints/breakpoint_test_arm64.c | 3 ++- .../breakpoints/step_after_suspend_test.c | 8 ++++++++ .../testing/selftests/capabilities/test_execve.c | 6 ++++-- .../selftests/futex/functional/futex_requeue_pi.c | 1 + .../functional/futex_requeue_pi_mismatched_ops.c | 1 + .../functional/futex_requeue_pi_signal_restart.c | 1 + .../functional/futex_wait_private_mapped_file.c | 1 + .../futex/functional/futex_wait_timeout.c | 1 + .../functional/futex_wait_uninitialized_heap.c | 1 + .../futex/functional/futex_wait_wouldblock.c | 1 + tools/testing/selftests/kselftest.h | 13 +++++++++++-- .../selftests/membarrier/membarrier_test.c | 1 + tools/testing/selftests/pidfd/pidfd_test.c | 1 + tools/testing/selftests/sigaltstack/sas.c | 1 + tools/testing/selftests/sync/sync_test.c | 1 + 16 files changed, 48 insertions(+), 8 deletions(-) diff --git a/tools/testing/selftests/breakpoints/breakpoint_test.c b/tools/testing/selftests/breakpoints/breakpoint_test.c index 901b85ea6a59..8f3655e59020 100644 --- a/tools/testing/selftests/breakpoints/breakpoint_test.c +++ b/tools/testing/selftests/breakpoints/breakpoint_test.c @@ -21,6 +21,8 @@ #include "../kselftest.h" +#define COUNT_ISN_BPS 4 +#define COUNT_WPS 4 /* Breakpoint access modes */ enum { @@ -220,7 +222,7 @@ static void trigger_tests(void) if (!local && !global) continue; - for (i = 0; i < 4; i++) { + for (i = 0; i < COUNT_ISN_BPS; i++) { dummy_funcs[i](); check_trapped(); } @@ -292,7 +294,7 @@ static void launch_instruction_breakpoints(char *buf, int local, int global) { int i; - for (i = 0; i < 4; i++) { + for (i = 0; i < COUNT_ISN_BPS; i++) { set_breakpoint_addr(dummy_funcs[i], i); toggle_breakpoint(i, BP_X, 1, local, global, 1); ptrace(PTRACE_CONT, child_pid, NULL, 0); @@ -314,7 +316,7 @@ static void launch_watchpoints(char *buf, int mode, int len, else mode_str = "read"; - for (i = 0; i < 4; i++) { + for (i = 0; i < COUNT_WPS; i++) { set_breakpoint_addr(&dummy_var[i], i); toggle_breakpoint(i, mode, len, local, global, 1); ptrace(PTRACE_CONT, child_pid, NULL, 0); @@ -330,8 +332,15 @@ static void launch_watchpoints(char *buf, int mode, int len, static void launch_tests(void) { char buf[1024]; + unsigned int tests = 0; int len, local, global, i; + tests += 3 * COUNT_ISN_BPS; + tests += sizeof(long) / 2 * 3 * COUNT_WPS; + tests += sizeof(long) / 2 * 3 * COUNT_WPS; + tests += 2; + ksft_set_plan(tests); + /* Instruction breakpoints */ for (local = 0; local < 2; local++) { for (global = 0; global < 2; global++) { diff --git a/tools/testing/selftests/breakpoints/breakpoint_test_arm64.c b/tools/testing/selftests/breakpoints/breakpoint_test_arm64.c index 2d95e5adde72..ab59d814341a 100644 --- a/tools/testing/selftests/breakpoints/breakpoint_test_arm64.c +++ b/tools/testing/selftests/breakpoints/breakpoint_test_arm64.c @@ -118,7 +118,7 @@ static bool set_watchpoint(pid_t pid, int size, int wp) return false; } -static bool run_test(int wr_size, int wp_size, int wr, int wp) +static bool arun_test(int wr_size, int wp_size, int wr, int wp) { int status; siginfo_t siginfo; @@ -214,6 +214,7 @@ int main(int argc, char **argv) bool result; ksft_print_header(); + ksft_set_plan(213); act.sa_handler = sigalrm; sigemptyset(&act.sa_mask); diff --git a/tools/testing/selftests/breakpoints/step_after_suspend_test.c b/tools/testing/selftests/breakpoints/step_after_suspend_test.c index f82dcc1f8841..cf868b5e00f7 100644 --- a/tools/testing/selftests/breakpoints/step_after_suspend_test.c +++ b/tools/testing/selftests/breakpoints/step_after_suspend_test.c @@ -173,6 +173,7 @@ int main(int argc, char **argv) int opt; bool do_suspend = true; bool succeeded = true; + unsigned int tests = 0; cpu_set_t available_cpus; int err; int cpu; @@ -191,6 +192,13 @@ int main(int argc, char **argv) } } + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) { + if (!CPU_ISSET(cpu, &available_cpus)) + continue; + tests++; + } + ksft_set_plan(tests); + if (do_suspend) suspend(); diff --git a/tools/testing/selftests/capabilities/test_execve.c b/tools/testing/selftests/capabilities/test_execve.c index 3ab39a61b95b..df0ef02b4036 100644 --- a/tools/testing/selftests/capabilities/test_execve.c +++ b/tools/testing/selftests/capabilities/test_execve.c @@ -430,8 +430,6 @@ int main(int argc, char **argv) { char *tmp1, *tmp2, *our_path; - ksft_print_header(); - /* Find our path */ tmp1 = strdup(argv[0]); if (!tmp1) @@ -445,6 +443,8 @@ int main(int argc, char **argv) mpid = getpid(); if (fork_wait()) { + ksft_print_header(); + ksft_set_plan(12); ksft_print_msg("[RUN]\t+++ Tests with uid == 0 +++\n"); return do_tests(0, our_path); } @@ -452,6 +452,8 @@ int main(int argc, char **argv) ksft_print_msg("==================================================\n"); if (fork_wait()) { + ksft_print_header(); + ksft_set_plan(9); ksft_print_msg("[RUN]\t+++ Tests with uid != 0 +++\n"); return do_tests(1, our_path); } diff --git a/tools/testing/selftests/futex/functional/futex_requeue_pi.c b/tools/testing/selftests/futex/functional/futex_requeue_pi.c index 54cd5c414e82..8d20957f7586 100644 --- a/tools/testing/selftests/futex/functional/futex_requeue_pi.c +++ b/tools/testing/selftests/futex/functional/futex_requeue_pi.c @@ -395,6 +395,7 @@ int main(int argc, char *argv[]) } ksft_print_header(); + ksft_set_plan(1); ksft_print_msg("%s: Test requeue functionality\n", basename(argv[0])); ksft_print_msg( "\tArguments: broadcast=%d locked=%d owner=%d timeout=%ldns\n", diff --git a/tools/testing/selftests/futex/functional/futex_requeue_pi_mismatched_ops.c b/tools/testing/selftests/futex/functional/futex_requeue_pi_mismatched_ops.c index 08187a16507f..742624c59ba7 100644 --- a/tools/testing/selftests/futex/functional/futex_requeue_pi_mismatched_ops.c +++ b/tools/testing/selftests/futex/functional/futex_requeue_pi_mismatched_ops.c @@ -79,6 +79,7 @@ int main(int argc, char *argv[]) } ksft_print_header(); + ksft_set_plan(1); ksft_print_msg("%s: Detect mismatched requeue_pi operations\n", basename(argv[0])); diff --git a/tools/testing/selftests/futex/functional/futex_requeue_pi_signal_restart.c b/tools/testing/selftests/futex/functional/futex_requeue_pi_signal_restart.c index f0542a344d95..a0f5934707ff 100644 --- a/tools/testing/selftests/futex/functional/futex_requeue_pi_signal_restart.c +++ b/tools/testing/selftests/futex/functional/futex_requeue_pi_signal_restart.c @@ -144,6 +144,7 @@ int main(int argc, char *argv[]) } ksft_print_header(); + ksft_set_plan(1); ksft_print_msg("%s: Test signal handling during requeue_pi\n", basename(argv[0])); ksft_print_msg("\tArguments: \n"); diff --git a/tools/testing/selftests/futex/functional/futex_wait_private_mapped_file.c b/tools/testing/selftests/futex/functional/futex_wait_private_mapped_file.c index 6216de828093..a458d42ff86e 100644 --- a/tools/testing/selftests/futex/functional/futex_wait_private_mapped_file.c +++ b/tools/testing/selftests/futex/functional/futex_wait_private_mapped_file.c @@ -98,6 +98,7 @@ int main(int argc, char **argv) } ksft_print_header(); + ksft_set_plan(1); ksft_print_msg( "%s: Test the futex value of private file mappings in FUTEX_WAIT\n", basename(argv[0])); diff --git a/tools/testing/selftests/futex/functional/futex_wait_timeout.c b/tools/testing/selftests/futex/functional/futex_wait_timeout.c index bab3dfe1787f..04b95478059c 100644 --- a/tools/testing/selftests/futex/functional/futex_wait_timeout.c +++ b/tools/testing/selftests/futex/functional/futex_wait_timeout.c @@ -69,6 +69,7 @@ int main(int argc, char *argv[]) } ksft_print_header(); + ksft_set_plan(1); ksft_print_msg("%s: Block on a futex and wait for timeout\n", basename(argv[0])); ksft_print_msg("\tArguments: timeout=%ldns\n", timeout_ns); diff --git a/tools/testing/selftests/futex/functional/futex_wait_uninitialized_heap.c b/tools/testing/selftests/futex/functional/futex_wait_uninitialized_heap.c index 26975322545b..3a1d12a14921 100644 --- a/tools/testing/selftests/futex/functional/futex_wait_uninitialized_heap.c +++ b/tools/testing/selftests/futex/functional/futex_wait_uninitialized_heap.c @@ -100,6 +100,7 @@ int main(int argc, char **argv) } ksft_print_header(); + ksft_set_plan(1); ksft_print_msg("%s: Test the uninitialized futex value in FUTEX_WAIT\n", basename(argv[0])); diff --git a/tools/testing/selftests/futex/functional/futex_wait_wouldblock.c b/tools/testing/selftests/futex/functional/futex_wait_wouldblock.c index da15a63269b4..a34a6bbc30ce 100644 --- a/tools/testing/selftests/futex/functional/futex_wait_wouldblock.c +++ b/tools/testing/selftests/futex/functional/futex_wait_wouldblock.c @@ -65,6 +65,7 @@ int main(int argc, char *argv[]) } ksft_print_header(); + ksft_set_plan(1); ksft_print_msg("%s: Test the unexpected futex value in FUTEX_WAIT\n", basename(argv[0])); diff --git a/tools/testing/selftests/kselftest.h b/tools/testing/selftests/kselftest.h index 7f078e79a9fa..ec15c4f6af55 100644 --- a/tools/testing/selftests/kselftest.h +++ b/tools/testing/selftests/kselftest.h @@ -33,6 +33,7 @@ struct ksft_count { }; static struct ksft_count ksft_cnt; +static unsigned int ksft_plan; static inline int ksft_test_num(void) { @@ -61,13 +62,21 @@ static inline void ksft_print_header(void) printf("TAP version 13\n"); } +static inline void ksft_set_plan(unsigned int plan) +{ + ksft_plan = plan; + printf("1..%d\n", ksft_plan); +} + static inline void ksft_print_cnts(void) { + if (ksft_plan != ksft_test_num()) + printf("# Planned tests != run tests (%u != %u)\n", + ksft_plan, ksft_test_num()); printf("# Pass %d Fail %d Xfail %d Xpass %d Skip %d Error %d\n", ksft_cnt.ksft_pass, ksft_cnt.ksft_fail, ksft_cnt.ksft_xfail, ksft_cnt.ksft_xpass, ksft_cnt.ksft_xskip, ksft_cnt.ksft_error); - printf("1..%d\n", ksft_test_num()); } static inline void ksft_print_msg(const char *msg, ...) @@ -172,7 +181,7 @@ static inline int ksft_exit_skip(const char *msg, ...) va_list args; va_start(args, msg); - printf("not ok %d # SKIP ", ksft_test_num()); + printf("not ok %d # SKIP ", 1 + ksft_test_num()); vprintf(msg, args); va_end(args); } else { diff --git a/tools/testing/selftests/membarrier/membarrier_test.c b/tools/testing/selftests/membarrier/membarrier_test.c index 6793f8ecc8e7..70b4ddbf126b 100644 --- a/tools/testing/selftests/membarrier/membarrier_test.c +++ b/tools/testing/selftests/membarrier/membarrier_test.c @@ -304,6 +304,7 @@ static int test_membarrier_query(void) int main(int argc, char **argv) { ksft_print_header(); + ksft_set_plan(13); test_membarrier_query(); test_membarrier(); diff --git a/tools/testing/selftests/pidfd/pidfd_test.c b/tools/testing/selftests/pidfd/pidfd_test.c index d59378a93782..5bae1792e3d6 100644 --- a/tools/testing/selftests/pidfd/pidfd_test.c +++ b/tools/testing/selftests/pidfd/pidfd_test.c @@ -371,6 +371,7 @@ static int test_pidfd_send_signal_syscall_support(void) int main(int argc, char **argv) { ksft_print_header(); + ksft_set_plan(4); test_pidfd_send_signal_syscall_support(); test_pidfd_send_signal_simple_success(); diff --git a/tools/testing/selftests/sigaltstack/sas.c b/tools/testing/selftests/sigaltstack/sas.c index 228c2ae47687..ad0f8df2ca0a 100644 --- a/tools/testing/selftests/sigaltstack/sas.c +++ b/tools/testing/selftests/sigaltstack/sas.c @@ -109,6 +109,7 @@ int main(void) int err; ksft_print_header(); + ksft_set_plan(3); sigemptyset(&act.sa_mask); act.sa_flags = SA_ONSTACK | SA_SIGINFO; diff --git a/tools/testing/selftests/sync/sync_test.c b/tools/testing/selftests/sync/sync_test.c index 7f7938263c5c..3824b66f41a0 100644 --- a/tools/testing/selftests/sync/sync_test.c +++ b/tools/testing/selftests/sync/sync_test.c @@ -86,6 +86,7 @@ int main(void) int err; ksft_print_header(); + ksft_set_plan(3 + 7); sync_api_supported();