From patchwork Sat May 4 08:14:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shin'ichiro Kawasaki X-Patchwork-Id: 13653814 Received: from esa6.hgst.iphmx.com (esa6.hgst.iphmx.com [216.71.154.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8B393FBF0 for ; Sat, 4 May 2024 08:14:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=216.71.154.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714810501; cv=none; b=OkTEJNndiA4pKI6FEdmdkyLGhl1t76VnDAC0TFFg7kmcnCoBZFlCEAmahSbIXeSE9ZVzYXcZwOENua+akLPAzCsT6pTHiHpKQ8DFqxnY4OMCevfyxVP8SuvOn4zrQx/QJKOyeobN36xObu2vVikjcn+eRXAE7kwnP248FHcq3p0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714810501; c=relaxed/simple; bh=eJKXXTmmutvILsJm/2yBesDZIUjxWUBFtkZckYioXbo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IBoUiS9P1qFYRtsGEGXoqSmWoDWAUJ48RXQFEM4DU9uxhzluwjH2y2ZIe8lWDDj2hGgFXaS0FUDMAuPb40Fci6QoYo7smBVk9c2tB6qINZQXsjXDDmuhrCKGMpIgerdqhjOmWxVJV8K+lxUviQj0eOWLqi9MkL9xn7AVijzANLU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=wdc.com; spf=pass smtp.mailfrom=wdc.com; dkim=pass (2048-bit key) header.d=wdc.com header.i=@wdc.com header.b=P9hebJdp; arc=none smtp.client-ip=216.71.154.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=wdc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=wdc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=wdc.com header.i=@wdc.com header.b="P9hebJdp" DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1714810499; x=1746346499; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=eJKXXTmmutvILsJm/2yBesDZIUjxWUBFtkZckYioXbo=; b=P9hebJdpmC/WO/PiNdeE9Evrp5kZe70FWUZOWoskHsggc3HDGmw9BjIv CFAAqxwXuJNL00p6WbmozZ2+qSYN5xZMMpl2Zi/2FpKAEORXfGSsLx7/s I7UH0Gp3A8RTSbvoRQLg3KMDwdE3ce3XlcZLqi7+8kS72SQ6l+Yz11p4i 8Sqx6gkF7bN1vYvChmkhXNIl1p/GX39iWNF5ys6+5asPjwBTQjLtnCEcn YAnGvSqsOneWWbIg3WvUrtqQaZvDqzxRFGnG/6fhPJmeK1/EqUDB6RHQj g6LjoydsxcPeUM8es0mTxUkhO2y6wCKSs5sw6BuNCZpOpRjEdYg3Y8Fno g==; X-CSE-ConnectionGUID: 9zTRmP4LSUq3QohK9+a4HQ== X-CSE-MsgGUID: 0gWRgdYQT/iGrhZS1Nz6fA== X-IronPort-AV: E=Sophos;i="6.07,253,1708358400"; d="scan'208";a="15540296" Received: from h199-255-45-14.hgst.com (HELO uls-op-cesaep01.wdc.com) ([199.255.45.14]) by ob1.hgst.iphmx.com with ESMTP; 04 May 2024 16:14:52 +0800 IronPort-SDR: bKeqy23lvB8whJJRsd+SzQgO+WWV7bAQLgvdUH7wfN/lYUp46ABAqgT6T2/v/WVMsMMjPkxWYO xLXHhTgfrhCjhg3vkrDNbX1odtWRH1Wuk38Q9x9vqrN4V2/xrZr9aAutJIfYz0r3UFUqkkXbK1 AXe4jjXTD84U7/RqeFadpNLzyK9bfV8Z4kOuUtGYF1EtsvLfrjesT38PTcmSWi2+Y0JiN0QCE3 8Tn+iGJCorsIv+gn8nBYjy6XaRZpeqOxGjNI9luD6pRTWtxpWbSBwUF+YEBuCYYHRrosYg+Vi9 tfY= Received: from uls-op-cesaip01.wdc.com ([10.248.3.36]) by uls-op-cesaep01.wdc.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 04 May 2024 00:22:50 -0700 IronPort-SDR: /9Fo3YiPgAPrZ5iB4p7vAbI0p2npSot9JCaMuZWmmgl2KTZZGMlccEoVn04aMwbc60n6+7Quet BIWMfD6CaRRA8dt1/nC7MPjNLvpwnP2UcNmAVGVub3H9KptzSvl9Iw9Y8dZsj75mbrOmxN8uwu St3S6NFTb27Ff+o2uI2fYubSv5ll/NMaHupVQznklDjspjbQgLxG9YOjyaEI/JWAkLFWGwkM/h qDBrf52etE4ZGLr7PKYPNaNuLf5uVjLj2ZLq/0NdufxYF00QBFjc6F6co4YmtRuvI2aAuuYdqj InQ= WDCIronportException: Internal Received: from unknown (HELO shinhome.flets-east.jp) ([10.225.163.38]) by uls-op-cesaip01.wdc.com with ESMTP; 04 May 2024 01:14:51 -0700 From: Shin'ichiro Kawasaki To: linux-block@vger.kernel.org Cc: linux-nvme@lists.infradead.org, Daniel Wagner , Chaitanya Kulkarni , Sagi Grimberg , Shin'ichiro Kawasaki Subject: [PATCH blktests v4 02/17] check: support test case repeat by different conditions Date: Sat, 4 May 2024 17:14:33 +0900 Message-ID: <20240504081448.1107562-3-shinichiro.kawasaki@wdc.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240504081448.1107562-1-shinichiro.kawasaki@wdc.com> References: <20240504081448.1107562-1-shinichiro.kawasaki@wdc.com> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 It is often required to run the same test with slightly different test conditions. If we create each test case for each test condition, those test cases are almost same and have code duplication. Such duplication is seen in many of the nvme test cases that set up nvme transport. To avoid the code duplication, introduce a new feature to support test case repetition with different conditions. When a test case implements the function set_conditions(), blktests repeat the test case. When set_conditions() is called without an argument, it returns how many times the test case is to be repeated. Before each test case run, blktests calls set_conditions() with an argument number from 0 to the number of repetitions minus 1. set_conditions() sets up the condition for each test run referring to the argument as the index of the condition to set up. set_conditions() also sets up a short string in the COND_DESC variable. This string is printed to stdout to identify the condition of each run. It is also used as the directory path name to hold result files. Document the usage of set_conditions() in the new script. Separate out shellcheck command line for the new script to avoid a false-positive warning unique to the file. Reviewed-by: Daniel Wagner Reviewed-by: Chaitanya Kulkarni Acked-by: Nitesh Shetty Signed-off-by: Shin'ichiro Kawasaki --- Makefile | 3 ++- check | 57 ++++++++++++++++++++++++++++++++++------------- common/shellcheck | 2 +- new | 21 +++++++++++++++++ 4 files changed, 65 insertions(+), 18 deletions(-) diff --git a/Makefile b/Makefile index 43f2ab0..1c685fe 100644 --- a/Makefile +++ b/Makefile @@ -18,8 +18,9 @@ install: SHELLCHECK_EXCLUDE := SC2119 check: - shellcheck -x -e $(SHELLCHECK_EXCLUDE) -f gcc check new common/* \ + shellcheck -x -e $(SHELLCHECK_EXCLUDE) -f gcc check common/* \ tests/*/rc tests/*/[0-9]*[0-9] src/*.sh + shellcheck --exclude=$(SHELLCHECK_EXCLUDE),SC2154 --format=gcc new ! grep TODO tests/*/rc tests/*/[0-9]*[0-9] ! find -name '*.out' -perm /u=x+g=x+o=x -printf '%p is executable\n' | grep . diff --git a/check b/check index 7d09ec0..edc421d 100755 --- a/check +++ b/check @@ -17,7 +17,9 @@ _found_test() { local test_name="$1" local explicit="$2" - unset CAN_BE_ZONED DESCRIPTION QUICK TIMED requires device_requires test test_device fallback_device cleanup_fallback_device + unset CAN_BE_ZONED DESCRIPTION QUICK TIMED requires device_requires \ + test test_device fallback_device cleanup_fallback_device \ + set_conditions # shellcheck disable=SC1090 if ! . "tests/${test_name}"; then @@ -190,15 +192,12 @@ _write_test_run() { _output_status() { local test="$1" local status="$2" - local zoned=" " + local str="${test} " - if (( RUN_FOR_ZONED )); then zoned=" (zoned) "; fi - - if [[ "${DESCRIPTION:-}" ]]; then - printf '%-60s' "${test}${zoned}($DESCRIPTION)" - else - printf '%-60s' "${test}${zoned}" - fi + (( RUN_FOR_ZONED )) && str="$str(zoned) " + [[ ${COND_DESC:-} ]] && str="$str(${COND_DESC}) " + [[ ${DESCRIPTION:-} ]] && str="$str(${DESCRIPTION})" + printf '%-60s' "${str}" if [[ -z $status ]]; then echo return @@ -464,17 +463,19 @@ _unload_modules() { } _check_and_call_test() { + local postfix local ret if declare -fF requires >/dev/null; then requires fi - RESULTS_DIR="$OUTPUT/nodev" + [[ -n $COND_DESC ]] && postfix=_${COND_DESC//[ =]/_} + RESULTS_DIR="$OUTPUT/nodev${postfix}" _call_test test ret=$? if (( RUN_ZONED_TESTS && CAN_BE_ZONED )); then - RESULTS_DIR="$OUTPUT/nodev_zoned" + RESULTS_DIR="$OUTPUT/nodev_zoned${postfix}" RUN_FOR_ZONED=1 _call_test test ret=$(( ret || $? )) @@ -484,6 +485,7 @@ _check_and_call_test() { } _check_and_call_test_device() { + local postfix local unset_skip_reason local ret @@ -491,6 +493,7 @@ _check_and_call_test_device() { requires fi + [[ -n $COND_DESC ]] && postfix=_${COND_DESC//[ =]/_} for TEST_DEV in "${TEST_DEVS[@]}"; do TEST_DEV_SYSFS="${TEST_DEV_SYSFS_DIRS["$TEST_DEV"]}" TEST_DEV_PART_SYSFS="${TEST_DEV_PART_SYSFS_DIRS["$TEST_DEV"]}" @@ -504,7 +507,7 @@ _check_and_call_test_device() { device_requires fi fi - RESULTS_DIR="$OUTPUT/$(basename "$TEST_DEV")" + RESULTS_DIR="$OUTPUT/$(basename "$TEST_DEV")""$postfix" if ! _call_test test_device; then ret=1 fi @@ -522,9 +525,11 @@ _run_test() { CHECK_DMESG=1 DMESG_FILTER="cat" RUN_FOR_ZONED=0 + COND_DESC="" FALLBACK_DEVICE=0 MODULES_TO_UNLOAD=() + local nr_conds cond_i local ret=0 # Ensure job control monitor mode is off in the sub-shell for test case @@ -535,8 +540,18 @@ _run_test() { . "tests/${TEST_NAME}" if declare -fF test >/dev/null; then - _check_and_call_test - ret=$? + if declare -fF set_conditions >/dev/null; then + nr_conds=$(set_conditions) + for ((cond_i = 0; cond_i < nr_conds; cond_i++)); do + set_conditions $cond_i + _check_and_call_test + ret=$(( ret || $? )) + unset SKIP_REASONS + done + else + _check_and_call_test + ret=$? + fi else if [[ ${#TEST_DEVS[@]} -eq 0 ]] && \ declare -fF fallback_device >/dev/null; then @@ -558,8 +573,18 @@ _run_test() { return 0 fi - _check_and_call_test_device - ret=$? + if declare -fF set_conditions >/dev/null; then + nr_conds=$(set_conditions) + for ((cond_i = 0; cond_i < nr_conds; cond_i++)); do + set_conditions $cond_i + _check_and_call_test_device + ret=$(( ret || $? )) + unset SKIP_REASONS + done + else + _check_and_call_test_device + ret=$? + fi if (( FALLBACK_DEVICE )); then cleanup_fallback_device diff --git a/common/shellcheck b/common/shellcheck index 8c324bd..ac0a51e 100644 --- a/common/shellcheck +++ b/common/shellcheck @@ -6,5 +6,5 @@ # Suppress unused global variable warnings. _silence_sc2034() { - echo "$CAN_BE_ZONED $CGROUP2_DIR $CHECK_DMESG $DESCRIPTION $DMESG_FILTER $FIO_PERF_FIELDS $FIO_PERF_PREFIX $QUICK $SKIP_REASONS ${TEST_RUN[*]} $TIMED" > /dev/null + echo "$CAN_BE_ZONED $CGROUP2_DIR $CHECK_DMESG $COND_DESC $DESCRIPTION $DMESG_FILTER $FIO_PERF_FIELDS $FIO_PERF_PREFIX $QUICK $SKIP_REASONS ${TEST_RUN[*]} $TIMED" > /dev/null } diff --git a/new b/new index 574d8b4..9cbba3c 100755 --- a/new +++ b/new @@ -180,6 +180,27 @@ DESCRIPTION="" # _require_test_dev_is_foo && _require_test_dev_supports_bar # } +# TODO: if the test case can run the same test for different conditions, define +# the helper function "set_conditions". When no argument is specified, return +# the number of condition variations. Blktests repeats the test case as many +# times as the returned number. When its argument is specified, refer to it as +# the condition variation index and set up the conditions for it. Also set the +# global variable COND_DESC which is printed at the test case run and used for +# the result directory name. Blktests calls set_conditions() before each run of +# the test case incrementing the argument index from 0. +# set_conditions() { +# local index=\$1 +# +# if [[ -z \$index ]]; then +# echo 2 # return number of condition variations +# return +# fi +# +# # Set test conditions based on the $index +# ... +# COND_DESC="Describe the conditions shortly" +# } + # TODO: define the test. The output of this function (stdout and stderr) will # be compared to tests/\${TEST_NAME}.out. If it does not match, the test is # considered a failure. If the test runs a command which has unnecessary