| Message ID | cover.1549411880.git.jonathantanmy@google.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Resend of GIT_TEST_PROTOCOL_VERSION patches | expand |
On Tue, Feb 05, 2019 at 04:21:14PM -0800, Jonathan Tan wrote: > This is on the latest master (8feddda32c ("Fifth batch for 2.21", > 2019-02-05)) + js/protocol-advertise-multi. > > This is a resend of [1], which was built on the result of merging many > branches. Now that most of the branches have been merged to master, I > have rebased it on master. The only branch that I needed to merge was > js/protocol-advertise-multi. Thanks for working on this. With the exception of the final patch, this all seems pretty sane to me from a quick look. There is one thing that your test patches made me wonder. When we have to make an exception to a test (i.e., that doesn't work under v2), you do it by unsetting GIT_TEST_PROTOCOL_VERSION in the environment. That means we'll actually run the test, but not with the version that the caller specified. I wonder if it would be more obvious what's going on if we instead had a prereq like: test_expect_success !PROTO_V2 'ls-remote --symref' ' ... ' and just skipped those tests entirely (and in a way that appears in the TAP output). I think it would also future-proof us a bit for v2 becoming the default (i.e., when GIT_TEST_PROTOCOL_VERSION being blank does mean "use v2"). I dunno. It probably doesn't matter all that much, so it may not be worth going back and changing at this point. Just a thought. -Peff
On Wed, Feb 06 2019, Jeff King wrote: > On Tue, Feb 05, 2019 at 04:21:14PM -0800, Jonathan Tan wrote: > >> This is on the latest master (8feddda32c ("Fifth batch for 2.21", >> 2019-02-05)) + js/protocol-advertise-multi. >> >> This is a resend of [1], which was built on the result of merging many >> branches. Now that most of the branches have been merged to master, I >> have rebased it on master. The only branch that I needed to merge was >> js/protocol-advertise-multi. > > Thanks for working on this. With the exception of the final patch, this > all seems pretty sane to me from a quick look. > > There is one thing that your test patches made me wonder. When we have > to make an exception to a test (i.e., that doesn't work under v2), you > do it by unsetting GIT_TEST_PROTOCOL_VERSION in the environment. That > means we'll actually run the test, but not with the version that the > caller specified. > > I wonder if it would be more obvious what's going on if we instead had a > prereq like: > > test_expect_success !PROTO_V2 'ls-remote --symref' ' > ... > ' > > and just skipped those tests entirely (and in a way that appears in the > TAP output). > > I think it would also future-proof us a bit for v2 becoming the default > (i.e., when GIT_TEST_PROTOCOL_VERSION being blank does mean "use v2"). > > I dunno. It probably doesn't matter all that much, so it may not be > worth going back and changing at this point. Just a thought. So far we've had the convention that these GIT_TEST_* variables, e.g. the one for the commit graph, work the same way. Thus we guarantee that we get (in theory) 100% coverage even when running the tests in this special mode. I think it's better to keep it as-is.
On Wed, Feb 06, 2019 at 10:52:15PM +0100, Ævar Arnfjörð Bjarmason wrote: > > I wonder if it would be more obvious what's going on if we instead had a > > prereq like: > > > > test_expect_success !PROTO_V2 'ls-remote --symref' ' > > ... > > ' > > > > and just skipped those tests entirely (and in a way that appears in the > > TAP output). > > > > I think it would also future-proof us a bit for v2 becoming the default > > (i.e., when GIT_TEST_PROTOCOL_VERSION being blank does mean "use v2"). > > > > I dunno. It probably doesn't matter all that much, so it may not be > > worth going back and changing at this point. Just a thought. > > So far we've had the convention that these GIT_TEST_* variables, > e.g. the one for the commit graph, work the same way. Thus we guarantee > that we get (in theory) 100% coverage even when running the tests in > this special mode. I think it's better to keep it as-is. But what's the point of that? Don't you always have to run the test suite _twice_, once with the special variable and once without? Otherwise, you are not testing one case or the other. Or are you arguing that one might set many special variables in one go (to prefer running the suite only twice, instead of 2^N times). In which case we are better off running the test (as opposed to skipping it), as it might use one of the _other_ special variables besides GIT_TEST_PROTOCOL_VERSION. I can buy that line of reasoning. It still doesn't cover all cases that a true 2^N test would, but that clearly isn't going to be practical. -Peff
On Wed, Feb 06 2019, Jeff King wrote: > On Wed, Feb 06, 2019 at 10:52:15PM +0100, Ævar Arnfjörð Bjarmason wrote: > >> > I wonder if it would be more obvious what's going on if we instead had a >> > prereq like: >> > >> > test_expect_success !PROTO_V2 'ls-remote --symref' ' >> > ... >> > ' >> > >> > and just skipped those tests entirely (and in a way that appears in the >> > TAP output). >> > >> > I think it would also future-proof us a bit for v2 becoming the default >> > (i.e., when GIT_TEST_PROTOCOL_VERSION being blank does mean "use v2"). >> > >> > I dunno. It probably doesn't matter all that much, so it may not be >> > worth going back and changing at this point. Just a thought. >> >> So far we've had the convention that these GIT_TEST_* variables, >> e.g. the one for the commit graph, work the same way. Thus we guarantee >> that we get (in theory) 100% coverage even when running the tests in >> this special mode. I think it's better to keep it as-is. > > But what's the point of that? Don't you always have to run the test > suite _twice_, once with the special variable and once without? > Otherwise, you are not testing one case or the other. > > Or are you arguing that one might set many special variables in one go > (to prefer running the suite only twice, instead of 2^N times). In which > case we are better off running the test (as opposed to skipping it), as > it might use one of the _other_ special variables besides > GIT_TEST_PROTOCOL_VERSION. > > I can buy that line of reasoning. It still doesn't cover all cases that > a true 2^N test would, but that clearly isn't going to be practical. Maybe I'm misunderstanding what you're proposing, but as an example, let's say the test suite is just these two tests: test_expect_success 'some unrelated thing' '...' test_expect_success 'test protocol v2' 'GIT_TEST_PROTOCOL_VERSION=2 ...' And GIT_TEST_PROTOCOL_VERSION=0 is the default, let's say I want to test with GIT_TEST_PROTOCOL_VERSION=1 for whatever reason, I'd still like both tests to be run, not just 1/2 with GIT_TEST_PROTOCOL_VERSION=1 and 2/2 skipped because it's explicitly testing for the GIT_TEST_PROTOCOL_VERSION=2 case, whereas I asked for a GIT_TEST_PROTOCOL_VERSION=1. IOW the point of these tests is to piggy-back on the tests that *aren't* aware of the feature you're trying to test. So e.g. GIT_TEST_COMMIT_GRAPH=true should run our whole test suite with the commit graph, and *also* those tests that are explicitly aware of the commit graph, including those that for some reason would want to test for the case where it isn't enabled (to e.g. test that --contains works without the graph). Otherwise I can't say "I care more about GIT_TEST_COMMIT_GRAPH=true than not", and run just one test run with that, because I'll have blind spots in the commit graph tests themselves, and would then need to do another run with GIT_TEST_COMMIT_GRAPH= set to make sure I have 100% coverage.
On Wed, Feb 06, 2019 at 11:20:32PM +0100, Ævar Arnfjörð Bjarmason wrote: > >> So far we've had the convention that these GIT_TEST_* variables, > >> e.g. the one for the commit graph, work the same way. Thus we guarantee > >> that we get (in theory) 100% coverage even when running the tests in > >> this special mode. I think it's better to keep it as-is. > > > > But what's the point of that? Don't you always have to run the test > > suite _twice_, once with the special variable and once without? > > Otherwise, you are not testing one case or the other. > > > > Or are you arguing that one might set many special variables in one go > > (to prefer running the suite only twice, instead of 2^N times). In which > > case we are better off running the test (as opposed to skipping it), as > > it might use one of the _other_ special variables besides > > GIT_TEST_PROTOCOL_VERSION. > > > > I can buy that line of reasoning. It still doesn't cover all cases that > > a true 2^N test would, but that clearly isn't going to be practical. > > Maybe I'm misunderstanding what you're proposing, but as an example, > let's say the test suite is just these two tests: > > test_expect_success 'some unrelated thing' '...' > test_expect_success 'test protocol v2' 'GIT_TEST_PROTOCOL_VERSION=2 ...' > > And GIT_TEST_PROTOCOL_VERSION=0 is the default, let's say I want to test > with GIT_TEST_PROTOCOL_VERSION=1 for whatever reason, > > I'd still like both tests to be run, not just 1/2 with > GIT_TEST_PROTOCOL_VERSION=1 and 2/2 skipped because it's explicitly > testing for the GIT_TEST_PROTOCOL_VERSION=2 case, whereas I asked for a > GIT_TEST_PROTOCOL_VERSION=1. But that's my "why". The second test will run identically in both runs, regardless of your setting of GIT_TEST_PROTOCOL_VERSION. So there's value if you're only running the suite once in getting full coverage, but if you are literally going to run it with and without, then you're running the exact same code twice for no reason. And you have to run it both with and without, since otherwise all of the _other_ tests aren't seeing both options. > IOW the point of these tests is to piggy-back on the tests that *aren't* > aware of the feature you're trying to test. So > e.g. GIT_TEST_COMMIT_GRAPH=true should run our whole test suite with the > commit graph, and *also* those tests that are explicitly aware of the > commit graph, including those that for some reason would want to test > for the case where it isn't enabled (to e.g. test that --contains works > without the graph). > > Otherwise I can't say "I care more about GIT_TEST_COMMIT_GRAPH=true than > not", and run just one test run with that, because I'll have blind spots > in the commit graph tests themselves, and would then need to do another > run with GIT_TEST_COMMIT_GRAPH= set to make sure I have 100% coverage. So if we are still talking about the same 2-test setup above, which only has a GIT_TEST_PROTOCOL_VERSION override, then yeah, I get the point of being able to run a "stock" test, and then one with _both_ of the flags (GIT_TEST_PROTOCOL_VERSION and GIT_TEST_COMMIT_GRAPH) because you don't quite know how each test will interact with all of the "other" flags. So now I'm not 100% sure I understand what you're talking about, but I think maybe we are actually in agreement. ;) -Peff
On Thu, Feb 07 2019, Jeff King wrote: > On Wed, Feb 06, 2019 at 11:20:32PM +0100, Ævar Arnfjörð Bjarmason wrote: > >> >> So far we've had the convention that these GIT_TEST_* variables, >> >> e.g. the one for the commit graph, work the same way. Thus we guarantee >> >> that we get (in theory) 100% coverage even when running the tests in >> >> this special mode. I think it's better to keep it as-is. >> > >> > But what's the point of that? Don't you always have to run the test >> > suite _twice_, once with the special variable and once without? >> > Otherwise, you are not testing one case or the other. >> > >> > Or are you arguing that one might set many special variables in one go >> > (to prefer running the suite only twice, instead of 2^N times). In which >> > case we are better off running the test (as opposed to skipping it), as >> > it might use one of the _other_ special variables besides >> > GIT_TEST_PROTOCOL_VERSION. >> > >> > I can buy that line of reasoning. It still doesn't cover all cases that >> > a true 2^N test would, but that clearly isn't going to be practical. >> >> Maybe I'm misunderstanding what you're proposing, but as an example, >> let's say the test suite is just these two tests: >> >> test_expect_success 'some unrelated thing' '...' >> test_expect_success 'test protocol v2' 'GIT_TEST_PROTOCOL_VERSION=2 ...' >> >> And GIT_TEST_PROTOCOL_VERSION=0 is the default, let's say I want to test >> with GIT_TEST_PROTOCOL_VERSION=1 for whatever reason, >> >> I'd still like both tests to be run, not just 1/2 with >> GIT_TEST_PROTOCOL_VERSION=1 and 2/2 skipped because it's explicitly >> testing for the GIT_TEST_PROTOCOL_VERSION=2 case, whereas I asked for a >> GIT_TEST_PROTOCOL_VERSION=1. > > But that's my "why". The second test will run identically in both runs, > regardless of your setting of GIT_TEST_PROTOCOL_VERSION. So there's > value if you're only running the suite once in getting full coverage, > but if you are literally going to run it with and without, then you're > running the exact same code twice for no reason. And you have to run it > both with and without, since otherwise all of the _other_ tests aren't > seeing both options. Yeah, by always running the 2nd test regardless of what GIT_TEST_PROTOCOL_VERSION=* is set to we're wasting CPU if we know we're going to run both with GIT_TEST_PROTOCOL_VERSION=1 and GIT_TEST_PROTOCOL_VERSION=2. But we don't know that, and in terms of CPU & time the tests that rely on any given GIT_TEST_* flag are such a tiny part of the test suite, that I think it's fine to run them twice in such a scenario to guard against the case when we just run in one more or the other, and not both. >> IOW the point of these tests is to piggy-back on the tests that *aren't* >> aware of the feature you're trying to test. So >> e.g. GIT_TEST_COMMIT_GRAPH=true should run our whole test suite with the >> commit graph, and *also* those tests that are explicitly aware of the >> commit graph, including those that for some reason would want to test >> for the case where it isn't enabled (to e.g. test that --contains works >> without the graph). >> >> Otherwise I can't say "I care more about GIT_TEST_COMMIT_GRAPH=true than >> not", and run just one test run with that, because I'll have blind spots >> in the commit graph tests themselves, and would then need to do another >> run with GIT_TEST_COMMIT_GRAPH= set to make sure I have 100% coverage. > > So if we are still talking about the same 2-test setup above, which only > has a GIT_TEST_PROTOCOL_VERSION override, then yeah, I get the point of > being able to run a "stock" test, and then one with _both_ of the flags > (GIT_TEST_PROTOCOL_VERSION and GIT_TEST_COMMIT_GRAPH) because you don't > quite know how each test will interact with all of the "other" flags. Yeah that's another reason not to skip them, although we could imagine a prereq where we skip the v2 tests if GIT_TEST_PROTOCOL_VERSION=1 is set *and* no other GIT_TEST_*=* flag is set, but I think that would also be a bad idea. > So now I'm not 100% sure I understand what you're talking about, but I > think maybe we are actually in agreement. ;) I think there's two ways to view these GIT_TEST_FOO=BAR facilities: 1. Run all tests with "FOO" set to "BAR", skip those (via prereq) we know would break in that mode. 2. Ditto, but if a test says "I'm a test for FOO=!BAR" leave it alone. #2 is what we have now. I read your <20190206213458.GC12737@sigill.intra.peff.net> as suggesting that we should move to #1. You're correct that moving to #1 would force us to more exhaustively mark things so that if the default moves to "BAR" we just have to flip a switch.
This is on the latest master (8feddda32c ("Fifth batch for 2.21", 2019-02-05)) + js/protocol-advertise-multi. This is a resend of [1], which was built on the result of merging many branches. Now that most of the branches have been merged to master, I have rebased it on master. The only branch that I needed to merge was js/protocol-advertise-multi. js/protocol-advertise-multi has not been merged to next yet, but the general idea seems good to me, and I am making review of that patch one of my priorities. But if anyone is concerned that this set will be delayed by js/protocol-advertise-multi, I can extract the parts I need into a patch in this set (most likely just the part that unconditionally calls ssh with SendEnv in connect.c), although it is difficult to explain the necessity of that without reference to js/protocol-advertise-multi. Let me know what you think. Besides the rebase, this is unchanged from [1]. There is some discussion there between Ævar, Junio, and I, thinking that it is OK to merge this even if we didn't eliminate all errors. Right now, only 3 test cases fail with GIT_TEST_PROTOCOL_VERSION=2. Patch 1 implements the test variable, patches 2-7 eliminate false negatives (with explanations), and patch 8 fixes a genuine bug. [1] https://public-inbox.org/git/cover.1547677183.git.jonathantanmy@google.com/ Jonathan Tan (8): tests: define GIT_TEST_PROTOCOL_VERSION tests: always test fetch of unreachable with v0 t5503: fix overspecification of trace expectation t5512: compensate for v0 only sending HEAD symrefs t5700: only run with protocol version 1 tests: fix protocol version for overspecifications t5552: compensate for v2 filtering ref adv. remote-curl: in v2, fill credentials if needed protocol.c | 17 ++++++++-- remote-curl.c | 9 +++++- t/README | 3 ++ t/t5400-send-pack.sh | 2 +- t/t5500-fetch-pack.sh | 4 ++- t/t5503-tagfollow.sh | 2 +- t/t5512-ls-remote.sh | 18 ++++++++--- t/t5515-fetch-merge-logic.sh | 4 +++ t/t5516-fetch-push.sh | 22 ++++++++++--- t/t5539-fetch-http-shallow.sh | 5 ++- t/t5541-http-push-smart.sh | 14 +++++++-- t/t5551-http-fetch-smart.sh | 47 +++++++++++++++++++++------- t/t5552-skipping-fetch-negotiator.sh | 5 ++- t/t5700-protocol-v1.sh | 3 ++ t/t7406-submodule-update.sh | 5 ++- 15 files changed, 128 insertions(+), 32 deletions(-)