From patchwork Thu Sep 26 17:52:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cristian Marussi X-Patchwork-Id: 11163125 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A7C2713B1 for ; Thu, 26 Sep 2019 17:53:07 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8160C21D56 for ; Thu, 26 Sep 2019 17:53:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Ioe8teVC" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8160C21D56 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=BWPg1jbva/RPRBFKOy76+Yilfi8K5QMs9iE55gubGIE=; b=Ioe8teVCJsS5YkjWL2v8JARpFX AytoT+GmEGGYx4YRkC/eJHkb14+M+I5SAP5j3+IKN+/0ltCNlDGk+2ODl8nI3HhkRGiPC0btTZoUe hTI3H0VN39WqhE6rLjubThPw3GMgrP42FmAuTitmakpjLzBKF5TI7glFuUSlLXLk8Gdz8SdSHnznl 9wUkca5Prwn/JBbcPBQFKgaIJ+TmaAfLsMZaq0DlKW4kfiAKL8Ze1ksrj9fHCcz/S4JCDV5JGG6W/ dhUJBWJ/r4tLrj5TWcxDvGEbqy3a06lUycaQO5V02X4YhE07JRdIvHXDpoNQ4KAn2SmZQwwvMp0Zi TCu2hbog==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.2 #3 (Red Hat Linux)) id 1iDXww-00045h-4V; Thu, 26 Sep 2019 17:53:06 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.2 #3 (Red Hat Linux)) id 1iDXwh-0003vG-IW for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2019 17:52:52 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A0D861570; Thu, 26 Sep 2019 10:52:49 -0700 (PDT) Received: from e120937-lin.cambridge.arm.com (e120937-lin.cambridge.arm.com [10.1.197.50]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D48AE3F67D; Thu, 26 Sep 2019 10:52:48 -0700 (PDT) From: Cristian Marussi To: linux-kselftest@vger.kernel.org, linux-arm-kernel@lists.infradead.org, shuah@kernel.org Subject: [PATCH v2 1/2] kselftest: add capability to skip chosen TARGETS Date: Thu, 26 Sep 2019 18:52:18 +0100 Message-Id: <20190926175219.29805-2-cristian.marussi@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190926175219.29805-1-cristian.marussi@arm.com> References: <20190926175219.29805-1-cristian.marussi@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190926_105251_653731_02FD557C X-CRM114-Status: GOOD ( 10.58 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tim.Bird@sony.com, dave.martin@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Let the user specify an optional TARGETS skiplist through the new optional SKIP_TARGETS Makefile variable. It is easier to skip at will using a reduced and well defined list of possibly problematic targets with SKIP_TARGETS than to provide a partially stripped down list of good targets using the usual TARGETS variable. Signed-off-by: Cristian Marussi --- v1 --> v2 - fixed commit msg - added documentation fix - added forced override to allow TARGETS= and SKIP_TARGETS being both passed on the cmdline --- Documentation/dev-tools/kselftest.rst | 16 ++++++++++++++++ tools/testing/selftests/Makefile | 7 +++++++ 2 files changed, 23 insertions(+) diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst index 25604904fa6e..ecdfdc9d4b03 100644 --- a/Documentation/dev-tools/kselftest.rst +++ b/Documentation/dev-tools/kselftest.rst @@ -89,6 +89,22 @@ To build, save output files in a separate directory with KBUILD_OUTPUT :: $ export KBUILD_OUTPUT=/tmp/kselftest; make TARGETS="size timers" kselftest +Additionally you can use the "SKIP_TARGETS" variable on the make command +line to specify one or more targets to exclude from the TARGETS list. + +To run all tests but a single subsystem:: + + $ make -C tools/testing/selftests SKIP_TARGETS=ptrace run_tests + +You can specify multiple tests to skip:: + + $ make SKIP_TARGETS="size timers" kselftest + +You can also specify a restricted list of tests to run together with a +dedicated skiplist:: + + $ make TARGETS="bpf breakpoints size timers" SKIP_TARGETS=bpf kselftest + See the top-level tools/testing/selftests/Makefile for the list of all possible targets. diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 25b43a8c2b15..9621b26ee433 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -63,6 +63,13 @@ TARGETS += zram TARGETS_HOTPLUG = cpu-hotplug TARGETS_HOTPLUG += memory-hotplug +# User can optionally provide a TARGETS skiplist. +SKIP_TARGETS ?= +ifneq ($(SKIP_TARGETS),) + TMP := $(filter-out $(SKIP_TARGETS), $(TARGETS)) + override TARGETS := $(TMP) +endif + # Clear LDFLAGS and MAKEFLAGS if called from main # Makefile to avoid test build failures when test # Makefile doesn't have explicit build rules. From patchwork Thu Sep 26 17:52:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cristian Marussi X-Patchwork-Id: 11163127 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC13714ED for ; Thu, 26 Sep 2019 17:53:25 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8A71221D56 for ; Thu, 26 Sep 2019 17:53:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="eOe6p+j8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8A71221D56 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=G2jOc3KbLB4gP/r5YYIgZATdt+zzz98IsstoviGfiIQ=; b=eOe6p+j8dDvsR4xkCuK9e4XwHu mqiPG5IO5RigrpAc/EDyBfe6NG3PCPVOEQcMvXJdG9bj7vZiEkH9maZcaxGoDRhd67xY5rXkYcuP9 c7n5F/48K0XgiAZQygaqF7iPWokamXW4Z1zVV1mE4yoWW3lJK6TkJnScD5Z5XDnBx4EfsvpYERYrL 37V677jlDhVwB65HwmDj2ZDrr+lozHLpnuVDQRf0qtyNGLxE5+8VDhxtWe8Y6LG0L+UMEOmr02/Gg qIU88aNbYH1GAGFlhtOE8jXOiPNzaRhfxRYgYbqvkXV258Roeh8X5/ZoddK6BVFEDsvQnuha9hdh8 /W1QyVng==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.2 #3 (Red Hat Linux)) id 1iDXxC-0004L4-TG; Thu, 26 Sep 2019 17:53:22 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.2 #3 (Red Hat Linux)) id 1iDXwi-0003vF-Pq for linux-arm-kernel@lists.infradead.org; Thu, 26 Sep 2019 17:52:54 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A14561596; Thu, 26 Sep 2019 10:52:50 -0700 (PDT) Received: from e120937-lin.cambridge.arm.com (e120937-lin.cambridge.arm.com [10.1.197.50]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D4E1C3F67D; Thu, 26 Sep 2019 10:52:49 -0700 (PDT) From: Cristian Marussi To: linux-kselftest@vger.kernel.org, linux-arm-kernel@lists.infradead.org, shuah@kernel.org Subject: [PATCH v2 2/2] kselftest: exclude failed TARGETS from runlist Date: Thu, 26 Sep 2019 18:52:19 +0100 Message-Id: <20190926175219.29805-3-cristian.marussi@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190926175219.29805-1-cristian.marussi@arm.com> References: <20190926175219.29805-1-cristian.marussi@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190926_105252_878755_1BFA0418 X-CRM114-Status: UNSURE ( 8.07 ) X-CRM114-Notice: Please train this message. X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tim.Bird@sony.com, dave.martin@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org A TARGET which failed to be built/installed should not be included in the runlist generated inside the run_kselftest.sh script. Signed-off-by: Cristian Marussi --- As an example, here BPF failed to compile but was included in the runlist and attempted to run anyway: ... ./KSFT_LAST_2/run_kselftest.sh: 37: cd: can't cd to bpf TAP version 13 1..49 \# selftests: KSFT_LAST_2: test_verifier \# Warning: file test_verifier is missing! not ok 1 selftests: KSFT_LAST_2: test_verifier \# selftests: KSFT_LAST_2: test_tag \# Warning: file test_tag is missing! --- tools/testing/selftests/Makefile | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile index 9621b26ee433..d7a9df186837 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -205,8 +205,12 @@ ifdef INSTALL_PATH echo " cat /dev/null > \$$logfile" >> $(ALL_SCRIPT) echo "fi" >> $(ALL_SCRIPT) + @# While building run_kselftest.sh skip also non-existent TARGET dirs: + @# they could be the result of a build failure and should NOT be + @# included in the generated runlist. for TARGET in $(TARGETS); do \ BUILD_TARGET=$$BUILD/$$TARGET; \ + [ ! -d $$INSTALL_PATH/$$TARGET ] && echo "Skipping non-existent dir: $$TARGET" && continue; \ echo "[ -w /dev/kmsg ] && echo \"kselftest: Running tests in $$TARGET\" >> /dev/kmsg" >> $(ALL_SCRIPT); \ echo "cd $$TARGET" >> $(ALL_SCRIPT); \ echo -n "run_many" >> $(ALL_SCRIPT); \