[v5,0/3] Make test selection easier by specifying description substrings instead of just numeric counters

Message

Philippe Blain via GitGitGadget Oct. 16, 2020, 10:50 p.m. UTC

This patch series enables me to change

 ./t9999-my-test.sh --run=1-3,5,17,19

into

 ./t9999-my-test.sh --run=setup,rename,symlink

and have it pick out tests with "setup", "rename", or "symlink" in their
description and run those. Saves me a lot of time, especially since numbers
for tests aren't readily accessible. The easiest way for me to get the
numbers corresponding to the tests I want to run, is to run all the tests
and look at the output to match up the descriptions with their numbers --
thus defeating the point of selecting just a subset of the tests to run in
the first place.

Changes since v4:

 * Changed to only split tests on commas

Elijah Newren (3):
  test-lib: allow selecting tests by substring/glob with --run
  t6006, t6012: adjust tests to use 'setup' instead of synonyms
  test-lib: reduce verbosity of skipped tests

 t/README                     | 48 ++++++++++++++++++++------------
 t/t0000-basic.sh             | 53 +++++++++++++++++++++---------------
 t/t6006-rev-list-format.sh   |  2 +-
 t/t6012-rev-list-simplify.sh |  2 +-
 t/test-lib.sh                | 19 +++++++------
 5 files changed, 75 insertions(+), 49 deletions(-)


base-commit: d98273ba77e1ab9ec755576bc86c716a97bf59d7
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-878%2Fnewren%2Ftest-selection-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-878/newren/test-selection-v5
Pull-Request: https://github.com/git/git/pull/878

Range-diff vs v4:

 1:  8f5d4ad318 ! 1:  389c2c171a test-lib: allow selecting tests by substring/glob with --run
     @@ t/README: For an individual test suite --run could be used to specify that
      +ends been included.  You may omit the first or the second number to
      +mean "from the first test" or "up to the very last test" respectively.
      +
     -+The argument to --run is split on commas and whitespace into separate
     -+strings, numbers, and ranges, and picks all tests that match any of
     -+the individual selection criteria.  If the substring of the
     -+description text that you want to match includes a comma or space, use
     -+the glob character '?' instead.  For example --run='unnecessary?update
     -+timing' would match on all tests that match either the glob
     -+*unnecessary?update* or the glob *timing*.
     ++The argument to --run is split on commas into separate strings,
     ++numbers, and ranges, and picks all tests that match any of the
     ++individual selection criteria.  If the substring of the description
     ++text that you want to match includes a comma, use the glob character
     ++'?' instead.  For example --run='rebase,merge?cherry-pick' would match
     ++on all tests that match either the glob *rebase* or the glob
     ++*merge?cherry-pick*.
       
       If --run starts with an unprefixed number or range the initial
       set of tests to run is empty. If the first item starts with '!'
     +@@ t/README: all the tests are added to the initial set.  After initial set is
     + determined every test number or range is added or excluded from
     + the set one by one, from left to right.
     + 
     +-Individual numbers or ranges could be separated either by a space
     +-or a comma.
     +-
     + For example, to run only tests up to a specific test (21), one
     + could do this:
     + 
     +@@ t/README: or this:
     + Common case is to run several setup tests (1, 2, 3) and then a
     + specific test (21) that relies on that setup:
     + 
     +-    $ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'
     ++    $ sh ./t9200-git-cvsexport-commit.sh --run='1,2,3,21'
     + 
     + or:
     + 
     +@@ t/README: or:
     + 
     + or:
     + 
     +-    $ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'
     ++    $ sh ./t9200-git-cvsexport-commit.sh --run='-3,21'
     + 
     + As noted above, the test set is built by going through the items
     + from left to right, so this:
     + 
     +-    $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'
     ++    $ sh ./t9200-git-cvsexport-commit.sh --run='1-4,!3'
     + 
     + will run tests 1, 2, and 4.  Items that come later have higher
     + precedence.  It means that this:
     + 
     +-    $ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'
     ++    $ sh ./t9200-git-cvsexport-commit.sh --run='!3,1-4'
     + 
     + would just run tests from 1 to 4, including 3.
     + 
      @@ t/README: test in the test suite except from 7 up to 11:
       
           $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'
     @@ t/README: test in the test suite except from 7 up to 11:
       "setup" test, so you cannot _arbitrarily_ disable one test and
      
       ## t/t0000-basic.sh ##
     +@@ t/t0000-basic.sh: test_expect_success 'GIT_SKIP_TESTS does not skip unmatched suite' "
     + 
     + test_expect_success '--run basic' "
     + 	run_sub_test_lib_test run-basic \
     +-		'--run basic' --run='1 3 5' <<-\\EOF &&
     ++		'--run basic' --run='1,3,5' <<-\\EOF &&
     + 	for i in 1 2 3 4 5 6
     + 	do
     + 		test_expect_success \"passing test #\$i\" 'true'
     +@@ t/t0000-basic.sh: test_expect_success '--run with a range' "
     + 
     + test_expect_success '--run with two ranges' "
     + 	run_sub_test_lib_test run-two-ranges \
     +-		'--run with two ranges' --run='1-2 5-6' <<-\\EOF &&
     ++		'--run with two ranges' --run='1-2,5-6' <<-\\EOF &&
     + 	for i in 1 2 3 4 5 6
     + 	do
     + 		test_expect_success \"passing test #\$i\" 'true'
     +@@ t/t0000-basic.sh: test_expect_success '--run with basic negation' "
     + 
     + test_expect_success '--run with two negations' "
     + 	run_sub_test_lib_test run-two-neg \
     +-		'--run with two negations' --run='"'!3 !6'"' <<-\\EOF &&
     ++		'--run with two negations' --run='"'!3,!6'"' <<-\\EOF &&
     + 	for i in 1 2 3 4 5 6
     + 	do
     + 		test_expect_success \"passing test #\$i\" 'true'
     +@@ t/t0000-basic.sh: test_expect_success '--run with two negations' "
     + 
     + test_expect_success '--run a range and negation' "
     + 	run_sub_test_lib_test run-range-and-neg \
     +-		'--run a range and negation' --run='"'-4 !2'"' <<-\\EOF &&
     ++		'--run a range and negation' --run='"'-4,!2'"' <<-\\EOF &&
     + 	for i in 1 2 3 4 5 6
     + 	do
     + 		test_expect_success \"passing test #\$i\" 'true'
     +@@ t/t0000-basic.sh: test_expect_success '--run range negation' "
     + test_expect_success '--run include, exclude and include' "
     + 	run_sub_test_lib_test run-inc-neg-inc \
     + 		'--run include, exclude and include' \
     +-		--run='"'1-5 !1-3 2'"' <<-\\EOF &&
     ++		--run='"'1-5,!1-3,2'"' <<-\\EOF &&
     + 	for i in 1 2 3 4 5 6
     + 	do
     + 		test_expect_success \"passing test #\$i\" 'true'
     +@@ t/t0000-basic.sh: test_expect_success '--run include, exclude and include, comma separated' "
     + test_expect_success '--run exclude and include' "
     + 	run_sub_test_lib_test run-neg-inc \
     + 		'--run exclude and include' \
     +-		--run='"'!3- 5'"' <<-\\EOF &&
     ++		--run='"'!3-,5'"' <<-\\EOF &&
     + 	for i in 1 2 3 4 5 6
     + 	do
     + 		test_expect_success \"passing test #\$i\" 'true'
      @@ t/t0000-basic.sh: test_expect_success '--run empty selectors' "
       	EOF
       "
     @@ t/test-lib.sh: match_pattern_list () {
       	title="$1"
       	shift
       	arg="$1"
     +@@ t/test-lib.sh: match_test_selector_list () {
     + 
     + 	# Both commas and whitespace are accepted as separators.
     + 	OLDIFS=$IFS
     +-	IFS=' 	,'
     ++	IFS=','
     + 	set -- $1
     + 	IFS=$OLDIFS
     + 
      @@ t/test-lib.sh: match_test_selector_list () {
       			*-*)
       				if expr "z${selector%%-*}" : "z[0-9]*[^0-9]" >/dev/null
 2:  bbae029cc8 = 2:  7c7a3d9a34 t6006, t6012: adjust tests to use 'setup' instead of synonyms
 3:  0e1a1125f7 = 3:  01494bc0ba test-lib: reduce verbosity of skipped tests