Message ID | da756b4bfb9d1ce0d1213d585e72acfbf667e2a2.1706921262.git.steadmon@google.com (mailing list archive) |
---|---|
State | Superseded |
Headers | show |
Series | test-tool: add unit test suite runner | expand |
Josh Steadmon <steadmon@google.com> writes: > -UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES)) > -UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS))) > +UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES)) Nice that we no longer need the special casing. > diff --git a/t/unit-tests/t-basic.c b/t/helper/test-example-tap.c > similarity index 95% > rename from t/unit-tests/t-basic.c > rename to t/helper/test-example-tap.c > index fda1ae59a6..21e4848e78 100644 > --- a/t/unit-tests/t-basic.c > +++ b/t/helper/test-example-tap.c > @@ -1,4 +1,5 @@ > -#include "test-lib.h" > +#include "t/unit-tests/test-lib.h" > +#include "test-tool.h" As the first thing both of these headers include is "git-compat-util.h", so the ordering should be safe either way, because everybody else in the directory seems to include "test-tool.h" before including headers that are specific to the subsystem it is testing, and "t/unit-tests/test-lib.h" in this case is the header that is specific to the unit-test subsystem being tested, it may raise fewer eyebrows if we swapped the order of the inclusion here. Other than that, looks good to me. Thanks.
On Fri, Feb 02, 2024 at 04:50:26PM -0800, Josh Steadmon wrote: > This has the additional benefit that test harnesses seeking to run all > unit tests can find them with a simple glob of "t/unit-tests/bin/t-*", > with no exceptions needed. This will be important in a later patch where > we add support for running the unit tests via a test-tool subcommand. Is this last paragraph still accurate? I think in this rebased version of the series, we'll continue to use $(UNIT_TESTS) derived from the source list rather than a glob in bin/. > --- a/t/Makefile > +++ b/t/Makefile > @@ -44,8 +44,7 @@ TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh)) > CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test))) > CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl > UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c) > -UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES)) > -UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS))) > +UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES)) This drops the intermediate UNIT_TEST_PROGRAMS, which makes sense. It was only used to keep the long lines a bit more readable. But it also drops the $(sort) call. Do we need to keep it? Certainly I'd think we want the contents of $(UNIT_TESTS) to be in a deterministic order. Does the $(wildcard) function already return things in sorted order? I can't find any mention in the documention. It seems to do so for me in a simple test, but aae5239be2 (t/Makefile: Use $(sort ...) explicitly where needed, 2011-09-04) argues otherwise. So I think we probably want to keep it (or possibly move it onto the UNIT_TEST_SOURCES line, which keeps it close to the wildcard call). -Peff
Jeff King <peff@peff.net> writes: > So I think we probably want to keep it (or possibly move it onto the > UNIT_TEST_SOURCES line, which keeps it close to the wildcard call). Sensible.
On 2024.02.07 12:55, Junio C Hamano wrote: > Josh Steadmon <steadmon@google.com> writes: > > > -UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES)) > > -UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS))) > > +UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES)) > > Nice that we no longer need the special casing. > > > diff --git a/t/unit-tests/t-basic.c b/t/helper/test-example-tap.c > > similarity index 95% > > rename from t/unit-tests/t-basic.c > > rename to t/helper/test-example-tap.c > > index fda1ae59a6..21e4848e78 100644 > > --- a/t/unit-tests/t-basic.c > > +++ b/t/helper/test-example-tap.c > > @@ -1,4 +1,5 @@ > > -#include "test-lib.h" > > +#include "t/unit-tests/test-lib.h" > > +#include "test-tool.h" > > As the first thing both of these headers include is > "git-compat-util.h", so the ordering should be safe either way, > because everybody else in the directory seems to include > "test-tool.h" before including headers that are specific to the > subsystem it is testing, and "t/unit-tests/test-lib.h" in this case > is the header that is specific to the unit-test subsystem being > tested, it may raise fewer eyebrows if we swapped the order of the > inclusion here. > > Other than that, looks good to me. > > Thanks. Fixed in V3.
On 2024.02.07 17:58, Jeff King wrote: > On Fri, Feb 02, 2024 at 04:50:26PM -0800, Josh Steadmon wrote: > > > This has the additional benefit that test harnesses seeking to run all > > unit tests can find them with a simple glob of "t/unit-tests/bin/t-*", > > with no exceptions needed. This will be important in a later patch where > > we add support for running the unit tests via a test-tool subcommand. > > Is this last paragraph still accurate? I think in this rebased version > of the series, we'll continue to use $(UNIT_TESTS) derived from the > source list rather than a glob in bin/. Right, thanks for the catch. Removed in V3. > > --- a/t/Makefile > > +++ b/t/Makefile > > @@ -44,8 +44,7 @@ TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh)) > > CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test))) > > CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl > > UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c) > > -UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES)) > > -UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS))) > > +UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES)) > > This drops the intermediate UNIT_TEST_PROGRAMS, which makes sense. It > was only used to keep the long lines a bit more readable. But it also > drops the $(sort) call. Do we need to keep it? > > Certainly I'd think we want the contents of $(UNIT_TESTS) to be in a > deterministic order. Does the $(wildcard) function already return things > in sorted order? I can't find any mention in the documention. It seems > to do so for me in a simple test, but aae5239be2 (t/Makefile: Use $(sort > ...) explicitly where needed, 2011-09-04) argues otherwise. I see this line in the docs [1]: "As with wildcard expansion in rules, the results of the wildcard function are sorted". GNU Make has restored the sorted behavior of $(wildcard) since 2018 [2]. I'll leave the sort off for now, but if folks feel like we need to support older versions of `make`, I'll add it back. [1] https://www.gnu.org/software/make/manual/html_node/Wildcard-Function.html [2] https://savannah.gnu.org/bugs/index.php?52076
Josh Steadmon <steadmon@google.com> writes: > I see this line in the docs [1]: "As with wildcard expansion in rules, > the results of the wildcard function are sorted". GNU Make has restored > the sorted behavior of $(wildcard) since 2018 [2]. I'll leave the sort > off for now, but if folks feel like we need to support older versions of > `make`, I'll add it back. > > [1] https://www.gnu.org/software/make/manual/html_node/Wildcard-Function.html > [2] https://savannah.gnu.org/bugs/index.php?52076 Thanks for digging. I thought I was certain that woldcard is sorted and stable and was quite perplexed when I could not find the mention in a version of doc I had handy ("""This is Edition 0.75, last updated 19 January 2020, of 'The GNU Make Manual', for GNU 'make' version 4.3.""").
On Mon, Feb 12, 2024 at 01:27:11PM -0800, Junio C Hamano wrote: > Josh Steadmon <steadmon@google.com> writes: > > > I see this line in the docs [1]: "As with wildcard expansion in rules, > > the results of the wildcard function are sorted". GNU Make has restored > > the sorted behavior of $(wildcard) since 2018 [2]. I'll leave the sort > > off for now, but if folks feel like we need to support older versions of > > `make`, I'll add it back. > > > > [1] https://www.gnu.org/software/make/manual/html_node/Wildcard-Function.html > > [2] https://savannah.gnu.org/bugs/index.php?52076 > > Thanks for digging. I thought I was certain that woldcard is sorted > and stable and was quite perplexed when I could not find the mention > in a version of doc I had handy ("""This is Edition 0.75, last > updated 19 January 2020, of 'The GNU Make Manual', for GNU 'make' > version 4.3."""). Likewise (mine is the latest version in Debian unstable). The change to sort comes from their[1] eedea52a, which was in GNU make 4.2.90. But the matching documentation change didn't happen until 5b993ae, which was 4.3.90 in late 2021. So that explains the mystery. Those dates imply to me that we should keep the $(sort), though. Six years is not so long in distro timescales, especially given that Debian unstable is on a 4-year-old version. (And if we did want to get rid of it, certainly we should do so consistently across the Makefile in a separate patch). -Peff [1] commit ids are from https://git.savannah.gnu.org/git/make.git
On Tuesday, February 13, 2024 2:41 AM, Peff wrote: >On Mon, Feb 12, 2024 at 01:27:11PM -0800, Junio C Hamano wrote: > >> Josh Steadmon <steadmon@google.com> writes: >> >> > I see this line in the docs [1]: "As with wildcard expansion in >> > rules, the results of the wildcard function are sorted". GNU Make >> > has restored the sorted behavior of $(wildcard) since 2018 [2]. I'll >> > leave the sort off for now, but if folks feel like we need to >> > support older versions of `make`, I'll add it back. >> > >> > [1] >> > https://www.gnu.org/software/make/manual/html_node/Wildcard-Function >> > .html [2] https://savannah.gnu.org/bugs/index.php?52076 >> >> Thanks for digging. I thought I was certain that woldcard is sorted >> and stable and was quite perplexed when I could not find the mention >> in a version of doc I had handy ("""This is Edition 0.75, last updated >> 19 January 2020, of 'The GNU Make Manual', for GNU 'make' >> version 4.3."""). > >Likewise (mine is the latest version in Debian unstable). The change to sort comes >from their[1] eedea52a, which was in GNU make 4.2.90. But the matching >documentation change didn't happen until 5b993ae, which was >4.3.90 in late 2021. So that explains the mystery. > >Those dates imply to me that we should keep the $(sort), though. Six years is not so >long in distro timescales, especially given that Debian unstable is on a 4-year-old >version. (And if we did want to get rid of it, certainly we should do so consistently >across the Makefile in a separate patch). I am stuck on 4.2.1 and cannot get to 4.3.90 any time soon. Can you want on this? It will take us out unless we can suppress the $(sort) Sincerely, Randall
Jeff King <peff@peff.net> writes: > Likewise (mine is the latest version in Debian unstable). The change to > sort comes from their[1] eedea52a, which was in GNU make 4.2.90. But the > matching documentation change didn't happen until 5b993ae, which was > 4.3.90 in late 2021. So that explains the mystery. > > Those dates imply to me that we should keep the $(sort), though. Six > years is not so long in distro timescales, especially given that Debian > unstable is on a 4-year-old version. (And if we did want to get rid of > it, certainly we should do so consistently across the Makefile in a > separate patch). Sounds sensible. The topic is expecting a reroll so I'll make sure I won't touch it until I see an update. Thanks.
On 2024.02.13 02:41, Jeff King wrote: > On Mon, Feb 12, 2024 at 01:27:11PM -0800, Junio C Hamano wrote: > > > Josh Steadmon <steadmon@google.com> writes: > > > > > I see this line in the docs [1]: "As with wildcard expansion in rules, > > > the results of the wildcard function are sorted". GNU Make has restored > > > the sorted behavior of $(wildcard) since 2018 [2]. I'll leave the sort > > > off for now, but if folks feel like we need to support older versions of > > > `make`, I'll add it back. > > > > > > [1] https://www.gnu.org/software/make/manual/html_node/Wildcard-Function.html > > > [2] https://savannah.gnu.org/bugs/index.php?52076 > > > > Thanks for digging. I thought I was certain that woldcard is sorted > > and stable and was quite perplexed when I could not find the mention > > in a version of doc I had handy ("""This is Edition 0.75, last > > updated 19 January 2020, of 'The GNU Make Manual', for GNU 'make' > > version 4.3."""). > > Likewise (mine is the latest version in Debian unstable). The change to > sort comes from their[1] eedea52a, which was in GNU make 4.2.90. But the > matching documentation change didn't happen until 5b993ae, which was > 4.3.90 in late 2021. So that explains the mystery. > > Those dates imply to me that we should keep the $(sort), though. Six > years is not so long in distro timescales, especially given that Debian > unstable is on a 4-year-old version. (And if we did want to get rid of > it, certainly we should do so consistently across the Makefile in a > separate patch). Makes sense, thanks for investigating. I've restored the $(sort) in V3.
On 2024.02.13 09:36, rsbecker@nexbridge.com wrote: > On Tuesday, February 13, 2024 2:41 AM, Peff wrote: > >On Mon, Feb 12, 2024 at 01:27:11PM -0800, Junio C Hamano wrote: > > > >> Josh Steadmon <steadmon@google.com> writes: > >> > >> > I see this line in the docs [1]: "As with wildcard expansion in > >> > rules, the results of the wildcard function are sorted". GNU Make > >> > has restored the sorted behavior of $(wildcard) since 2018 [2]. I'll > >> > leave the sort off for now, but if folks feel like we need to > >> > support older versions of `make`, I'll add it back. > >> > > >> > [1] > >> > https://www.gnu.org/software/make/manual/html_node/Wildcard-Function > >> > .html [2] https://savannah.gnu.org/bugs/index.php?52076 > >> > >> Thanks for digging. I thought I was certain that woldcard is sorted > >> and stable and was quite perplexed when I could not find the mention > >> in a version of doc I had handy ("""This is Edition 0.75, last updated > >> 19 January 2020, of 'The GNU Make Manual', for GNU 'make' > >> version 4.3."""). > > > >Likewise (mine is the latest version in Debian unstable). The change to sort comes > >from their[1] eedea52a, which was in GNU make 4.2.90. But the matching > >documentation change didn't happen until 5b993ae, which was > >4.3.90 in late 2021. So that explains the mystery. > > > >Those dates imply to me that we should keep the $(sort), though. Six years is not so > >long in distro timescales, especially given that Debian unstable is on a 4-year-old > >version. (And if we did want to get rid of it, certainly we should do so consistently > >across the Makefile in a separate patch). > > I am stuck on 4.2.1 and cannot get to 4.3.90 any time soon. Can you > want on this? It will take us out unless we can suppress the $(sort) Hi Randall, I'm not sure I follow here. The change in 4.2.90 is that wildcard expansion becomes sorted by default again. So adding the $(sort) back shouldn't cause any problems in 4.2.1. Or did I misunderstand your point?
On Thursday, February 22, 2024 6:58 PM, Josh Steadmon wrote: >On 2024.02.13 09:36, rsbecker@nexbridge.com wrote: >> On Tuesday, February 13, 2024 2:41 AM, Peff wrote: >> >On Mon, Feb 12, 2024 at 01:27:11PM -0800, Junio C Hamano wrote: >> > >> >> Josh Steadmon <steadmon@google.com> writes: >> >> >> >> > I see this line in the docs [1]: "As with wildcard expansion in >> >> > rules, the results of the wildcard function are sorted". GNU Make >> >> > has restored the sorted behavior of $(wildcard) since 2018 [2]. >> >> > I'll leave the sort off for now, but if folks feel like we need >> >> > to support older versions of `make`, I'll add it back. >> >> > >> >> > [1] >> >> > https://www.gnu.org/software/make/manual/html_node/Wildcard-Funct >> >> > ion .html [2] https://savannah.gnu.org/bugs/index.php?52076 >> >> >> >> Thanks for digging. I thought I was certain that woldcard is >> >> sorted and stable and was quite perplexed when I could not find the >> >> mention in a version of doc I had handy ("""This is Edition 0.75, >> >> last updated >> >> 19 January 2020, of 'The GNU Make Manual', for GNU 'make' >> >> version 4.3."""). >> > >> >Likewise (mine is the latest version in Debian unstable). The change >> >to sort comes from their[1] eedea52a, which was in GNU make 4.2.90. >> >But the matching documentation change didn't happen until 5b993ae, >> >which was >> >4.3.90 in late 2021. So that explains the mystery. >> > >> >Those dates imply to me that we should keep the $(sort), though. Six >> >years is not so long in distro timescales, especially given that >> >Debian unstable is on a 4-year-old version. (And if we did want to >> >get rid of it, certainly we should do so consistently across the Makefile in a >separate patch). >> >> I am stuck on 4.2.1 and cannot get to 4.3.90 any time soon. Can you >> want on this? It will take us out unless we can suppress the $(sort) > >Hi Randall, > >I'm not sure I follow here. The change in 4.2.90 is that wildcard expansion becomes >sorted by default again. So adding the $(sort) back shouldn't cause any problems in >4.2.1. Or did I misunderstand your point? I thought you were referring to a new ${sort} behaviour in 4.2.90. My bad.
diff --git a/Makefile b/Makefile index 23723367b8..ba55d817ee 100644 --- a/Makefile +++ b/Makefile @@ -802,6 +802,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o TEST_BUILTINS_OBJS += test-dump-untracked-cache.o TEST_BUILTINS_OBJS += test-env-helper.o TEST_BUILTINS_OBJS += test-example-decorate.o +TEST_BUILTINS_OBJS += test-example-tap.o TEST_BUILTINS_OBJS += test-find-pack.o TEST_BUILTINS_OBJS += test-fsmonitor-client.o TEST_BUILTINS_OBJS += test-genrandom.o @@ -1338,7 +1339,6 @@ THIRD_PARTY_SOURCES += compat/regex/% THIRD_PARTY_SOURCES += sha1collisiondetection/% THIRD_PARTY_SOURCES += sha1dc/% -UNIT_TEST_PROGRAMS += t-basic UNIT_TEST_PROGRAMS += t-mem-pool UNIT_TEST_PROGRAMS += t-strbuf UNIT_TEST_PROGRAMS += t-ctype @@ -3218,7 +3218,7 @@ perf: all .PRECIOUS: $(TEST_OBJS) -t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) +t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) $(UNIT_TEST_DIR)/test-lib.o t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) $(REFTABLE_TEST_LIB) $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS) diff --git a/t/Makefile b/t/Makefile index 281f4c3534..1283c90c10 100644 --- a/t/Makefile +++ b/t/Makefile @@ -44,8 +44,7 @@ TINTEROP = $(sort $(wildcard interop/i[0-9][0-9][0-9][0-9]-*.sh)) CHAINLINTTESTS = $(sort $(patsubst chainlint/%.test,%,$(wildcard chainlint/*.test))) CHAINLINT = '$(PERL_PATH_SQ)' chainlint.pl UNIT_TEST_SOURCES = $(wildcard unit-tests/t-*.c) -UNIT_TEST_PROGRAMS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES)) -UNIT_TESTS = $(sort $(filter-out unit-tests/bin/t-basic%,$(UNIT_TEST_PROGRAMS))) +UNIT_TESTS = $(patsubst unit-tests/%.c,unit-tests/bin/%$(X),$(UNIT_TEST_SOURCES)) # `test-chainlint` (which is a dependency of `test-lint`, `test` and `prove`) # checks all tests in all scripts via a single invocation, so tell individual diff --git a/t/unit-tests/t-basic.c b/t/helper/test-example-tap.c similarity index 95% rename from t/unit-tests/t-basic.c rename to t/helper/test-example-tap.c index fda1ae59a6..21e4848e78 100644 --- a/t/unit-tests/t-basic.c +++ b/t/helper/test-example-tap.c @@ -1,4 +1,5 @@ -#include "test-lib.h" +#include "t/unit-tests/test-lib.h" +#include "test-tool.h" /* * The purpose of this "unit test" is to verify a few invariants of the unit @@ -69,7 +70,7 @@ static void t_empty(void) ; /* empty */ } -int cmd_main(int argc, const char **argv) +int cmd__example_tap(int argc, const char **argv) { test_res = TEST(check_res = check_int(1, ==, 1), "passing test"); TEST(t_res(1), "passing test and assertion return 1"); diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index 33b9501c21..bb5c04c9c0 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -29,6 +29,7 @@ static struct test_cmd cmds[] = { { "dump-untracked-cache", cmd__dump_untracked_cache }, { "env-helper", cmd__env_helper }, { "example-decorate", cmd__example_decorate }, + { "example-tap", cmd__example_tap }, { "find-pack", cmd__find_pack }, { "fsmonitor-client", cmd__fsmonitor_client }, { "genrandom", cmd__genrandom }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index b72f07ded9..38001bd1c6 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -23,6 +23,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv); int cmd__dump_reftable(int argc, const char **argv); int cmd__env_helper(int argc, const char **argv); int cmd__example_decorate(int argc, const char **argv); +int cmd__example_tap(int argc, const char **argv); int cmd__find_pack(int argc, const char **argv); int cmd__fsmonitor_client(int argc, const char **argv); int cmd__genrandom(int argc, const char **argv); diff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh index 961b54b06c..83b1e3b7f5 100755 --- a/t/t0080-unit-test-output.sh +++ b/t/t0080-unit-test-output.sh @@ -8,50 +8,50 @@ test_expect_success 'TAP output from unit tests' ' cat >expect <<-EOF && ok 1 - passing test ok 2 - passing test and assertion return 1 - # check "1 == 2" failed at t/unit-tests/t-basic.c:76 + # check "1 == 2" failed at t/helper/test-example-tap.c:77 # left: 1 # right: 2 not ok 3 - failing test ok 4 - failing test and assertion return 0 not ok 5 - passing TEST_TODO() # TODO ok 6 - passing TEST_TODO() returns 1 - # todo check ${SQ}check(x)${SQ} succeeded at t/unit-tests/t-basic.c:25 + # todo check ${SQ}check(x)${SQ} succeeded at t/helper/test-example-tap.c:26 not ok 7 - failing TEST_TODO() ok 8 - failing TEST_TODO() returns 0 - # check "0" failed at t/unit-tests/t-basic.c:30 + # check "0" failed at t/helper/test-example-tap.c:31 # skipping test - missing prerequisite - # skipping check ${SQ}1${SQ} at t/unit-tests/t-basic.c:32 + # skipping check ${SQ}1${SQ} at t/helper/test-example-tap.c:33 ok 9 - test_skip() # SKIP ok 10 - skipped test returns 1 # skipping test - missing prerequisite ok 11 - test_skip() inside TEST_TODO() # SKIP ok 12 - test_skip() inside TEST_TODO() returns 1 - # check "0" failed at t/unit-tests/t-basic.c:48 + # check "0" failed at t/helper/test-example-tap.c:49 not ok 13 - TEST_TODO() after failing check ok 14 - TEST_TODO() after failing check returns 0 - # check "0" failed at t/unit-tests/t-basic.c:56 + # check "0" failed at t/helper/test-example-tap.c:57 not ok 15 - failing check after TEST_TODO() ok 16 - failing check after TEST_TODO() returns 0 - # check "!strcmp("\thello\\\\", "there\"\n")" failed at t/unit-tests/t-basic.c:61 + # check "!strcmp("\thello\\\\", "there\"\n")" failed at t/helper/test-example-tap.c:62 # left: "\011hello\\\\" # right: "there\"\012" - # check "!strcmp("NULL", NULL)" failed at t/unit-tests/t-basic.c:62 + # check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:63 # left: "NULL" # right: NULL - # check "${SQ}a${SQ} == ${SQ}\n${SQ}" failed at t/unit-tests/t-basic.c:63 + # check "${SQ}a${SQ} == ${SQ}\n${SQ}" failed at t/helper/test-example-tap.c:64 # left: ${SQ}a${SQ} # right: ${SQ}\012${SQ} - # check "${SQ}\\\\${SQ} == ${SQ}\\${SQ}${SQ}" failed at t/unit-tests/t-basic.c:64 + # check "${SQ}\\\\${SQ} == ${SQ}\\${SQ}${SQ}" failed at t/helper/test-example-tap.c:65 # left: ${SQ}\\\\${SQ} # right: ${SQ}\\${SQ}${SQ} not ok 17 - messages from failing string and char comparison - # BUG: test has no checks at t/unit-tests/t-basic.c:91 + # BUG: test has no checks at t/helper/test-example-tap.c:92 not ok 18 - test with no checks ok 19 - test with no checks returns 0 1..19 EOF - ! "$GIT_BUILD_DIR"/t/unit-tests/bin/t-basic >actual && + ! test-tool example-tap >actual && test_cmp expect actual '
While t/unit-tests/t-basic.c uses the unit-test framework added in e137fe3b29 (unit tests: add TAP unit test framework, 2023-11-09), it is not a true unit test in that it intentionally fails in order to exercise various codepaths in the unit-test framework. Thus, we intentionally exclude it when running unit tests through the various t/Makefile targets. Instead, it is executed by t0080-unit-test-output.sh, which verifies its output follows the TAP format expected for the various pass, skip, or fail cases. As such, it makes more sense for t-basic to be a helper item for t0080-unit-test-output.sh, so let's move it to t/helper/test-example-tap.c and adjust Makefiles as necessary. This has the additional benefit that test harnesses seeking to run all unit tests can find them with a simple glob of "t/unit-tests/bin/t-*", with no exceptions needed. This will be important in a later patch where we add support for running the unit tests via a test-tool subcommand. Signed-off-by: Josh Steadmon <steadmon@google.com> --- Makefile | 4 ++-- t/Makefile | 3 +-- .../t-basic.c => helper/test-example-tap.c} | 5 ++-- t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t0080-unit-test-output.sh | 24 +++++++++---------- 6 files changed, 20 insertions(+), 18 deletions(-) rename t/{unit-tests/t-basic.c => helper/test-example-tap.c} (95%)