| Message ID | 20240915113115.GA2019070@coredump.intra.peff.net (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | 83799f1500768ade2452a07b08cf64a5fa46afde |
| Headers | show |
| Series | t9001: use a more distinct fake BugID | expand |
On Sun, Sep 15, 2024 at 07:31:15AM -0400, Jeff King wrote: > In the test "cc list is sanitized", we feed a commit with a variety of > trailers to send-email, and then check its output to see how it handled > them. For most of them, we are grepping for a specific mention of the > header, but there's a "BugID" header which we expect to be ignored. We > confirm this by grepping for "12345", the fake BugID, and making sure it > is not present. > > But we can be fooled by false positives! I just tracked down a flaky > test failure here that was caused by matching this unrelated line in the > output: > > <20240914090449.612345-1-author@example.com> > > which will change from run to run based on the time, pid, etc. > > Ideally we'd tighten the regex to make this more specifically, but since > the point is that it _shouldn't_ be mentioned, it's hard to say what the > right match would be (e.g., would there be a leading space?). > > Instead, let's just choose a match that is much less likely to appear. > The actual content of the header isn't important, since it's supposed to > be ignored. > > Signed-off-by: Jeff King <peff@peff.net> > --- > I guess this is fairly unlikely, as re-running the test with --stress > didn't reproduce after a few hundred attempts. Back of the envelope, I > guess any 5-digit sequence has a 1-in-10^5 chance of matching our > target. There are 2 in chances in a 6-digit pid. Some in the date, but > as there's no December 34th or hour 34, you're limited to a few specific > times like 01:23:45 (or at 11am), 12:34:5x, or perhaps 23:45 on the 1st, > 11th, or 21st of the month. > > So I think I just got really unlucky, but after spending several minutes > debugging, I wouldn't want to wish it on anybody else. ;) Heh, unlucky indeed. In any case, the change makes sense to me, thanks! Patrick
diff --git a/t/t9001-send-email.sh b/t/t9001-send-email.sh index df5336bb7e..e2430f7bfa 100755 --- a/t/t9001-send-email.sh +++ b/t/t9001-send-email.sh @@ -1324,7 +1324,7 @@ test_expect_success $PREREQ 'cc list is sanitized' ' Reviewed-by: Füñný Nâmé <odd_?=mail@example.com> Reported-by: bugger on Jira Reported-by: Douglas Reporter <doug@example.com> [from Jira profile] - BugID: 12345 + BugID: 12345should-not-appear Co-developed-by: "C. O. Developer" <codev@example.com> Signed-off-by: A. U. Thor <thor.au@example.com> EOF @@ -1337,7 +1337,7 @@ test_expect_success $PREREQ 'cc list is sanitized' ' " <odd_?=mail@example.com>" actual-show-all-headers && test_grep "^(body) Ignoring Reported-by .* bugger on Jira" actual-show-all-headers && test_grep "^(body) Adding cc: Douglas Reporter <doug@example.com>" actual-show-all-headers && - test_grep ! "12345" actual-show-all-headers && + test_grep ! "12345should-not-appear" actual-show-all-headers && test_grep "^(body) Adding cc: \"C. O. Developer\" <codev@example.com>" actual-show-all-headers && test_grep "^(body) Adding cc: \"A. U. Thor\" <thor.au@example.com>" actual-show-all-headers '
In the test "cc list is sanitized", we feed a commit with a variety of trailers to send-email, and then check its output to see how it handled them. For most of them, we are grepping for a specific mention of the header, but there's a "BugID" header which we expect to be ignored. We confirm this by grepping for "12345", the fake BugID, and making sure it is not present. But we can be fooled by false positives! I just tracked down a flaky test failure here that was caused by matching this unrelated line in the output: <20240914090449.612345-1-author@example.com> which will change from run to run based on the time, pid, etc. Ideally we'd tighten the regex to make this more specifically, but since the point is that it _shouldn't_ be mentioned, it's hard to say what the right match would be (e.g., would there be a leading space?). Instead, let's just choose a match that is much less likely to appear. The actual content of the header isn't important, since it's supposed to be ignored. Signed-off-by: Jeff King <peff@peff.net> --- I guess this is fairly unlikely, as re-running the test with --stress didn't reproduce after a few hundred attempts. Back of the envelope, I guess any 5-digit sequence has a 1-in-10^5 chance of matching our target. There are 2 in chances in a 6-digit pid. Some in the date, but as there's no December 34th or hour 34, you're limited to a few specific times like 01:23:45 (or at 11am), 12:34:5x, or perhaps 23:45 on the 1st, 11th, or 21st of the month. So I think I just got really unlucky, but after spending several minutes debugging, I wouldn't want to wish it on anybody else. ;) t/t9001-send-email.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)