[GSoC,v3] t: migrate t0110-urlmatch-normalization to the new framework

helper/test-urlmatch-normalization along with
t0110-urlmatch-normalization test the `url_normalize()` function from
'urlmatch.h'. Migrate them to the unit testing framework for better
performance. And also add different test_msg()s for better debugging.

In the migration, last two of the checks from `t_url_general_escape()`
were slightly changed compared to the shellscript. This involves changing

'\'' -> '
'\!' -> !

in the urls of those checks. This is because in C strings, we don't
need to escape "'" and "!". Other than these two, all the urls were
pasted verbatim from the shellscript.

Another change is the removal of MINGW prerequisite from one of the
test. It was there because[1] on Windows, the command line is a
Unicode string, it is not possible to pass arbitrary bytes to a
program. But in unit tests we don't have this limitation.

And since we can construct strings with arbitrary bytes in C, let's
also remove the test files which contain URLs with arbitrary bytes in
the 't/t0110' directory and instead embed those URLs in the unit test
code itself.

[1]: https://lore.kernel.org/git/53CAC8EF.6020707@gmail.com/

Mentored-by: Christian Couder <chriscool@tuxfamily.org>
Mentored-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
Signed-off-by: Ghanshyam Thakkar <shyamthakkar001@gmail.com>
---
This version addresses Junio's review and removes the restriction
of running the unit tests in the 't/' and 't/unit-tests/bin'
introduced in v2 by embedding the URLs in the code itself.

 Makefile                                |   2 +-
 t/helper/test-tool.c                    |   1 -
 t/helper/test-tool.h                    |   1 -
 t/helper/test-urlmatch-normalization.c  |  56 -----
 t/t0110-urlmatch-normalization.sh       | 182 ----------------
 t/t0110/README                          |   9 -
 t/t0110/url-1                           | Bin 20 -> 0 bytes
 t/t0110/url-10                          | Bin 23 -> 0 bytes
 t/t0110/url-11                          | Bin 25 -> 0 bytes
 t/t0110/url-2                           | Bin 20 -> 0 bytes
 t/t0110/url-3                           | Bin 23 -> 0 bytes
 t/t0110/url-4                           | Bin 23 -> 0 bytes
 t/t0110/url-5                           | Bin 23 -> 0 bytes
 t/t0110/url-6                           | Bin 23 -> 0 bytes
 t/t0110/url-7                           | Bin 23 -> 0 bytes
 t/t0110/url-8                           | Bin 23 -> 0 bytes
 t/t0110/url-9                           | Bin 23 -> 0 bytes
 t/unit-tests/t-urlmatch-normalization.c | 271 ++++++++++++++++++++++++
 18 files changed, 272 insertions(+), 250 deletions(-)
 delete mode 100644 t/helper/test-urlmatch-normalization.c
 delete mode 100755 t/t0110-urlmatch-normalization.sh
 delete mode 100644 t/t0110/README
 delete mode 100644 t/t0110/url-1
 delete mode 100644 t/t0110/url-10
 delete mode 100644 t/t0110/url-11
 delete mode 100644 t/t0110/url-2
 delete mode 100644 t/t0110/url-3
 delete mode 100644 t/t0110/url-4
 delete mode 100644 t/t0110/url-5
 delete mode 100644 t/t0110/url-6
 delete mode 100644 t/t0110/url-7
 delete mode 100644 t/t0110/url-8
 delete mode 100644 t/t0110/url-9
 create mode 100644 t/unit-tests/t-urlmatch-normalization.c

Range-diff against v2:
1:  cf47b17a33 ! 1:  a73b89c8e0 t: migrate t0110-urlmatch-normalization to the new framework
    @@ Commit message
         pasted verbatim from the shellscript.

         Another change is the removal of MINGW prerequisite from one of the
    -    test. It was there because[1] on Windows, the command line is a Unicode
    -    string, it is not possible to pass arbitrary bytes to a program. But
    -    in unit tests we don't have this limitation.
    +    test. It was there because[1] on Windows, the command line is a
    +    Unicode string, it is not possible to pass arbitrary bytes to a
    +    program. But in unit tests we don't have this limitation.

    -    With the addition of this unit test, we impose a new restriction of
    -    running the unit tests from either 't/' or 't/unit-tests/bin/'
    -    directories. This is to construct the path to files which contain some
    -    input urls under the 't/t-urlmatch-normalization' directory. This
    -    restriction is similar to one we have for end-to-end tests, where they
    -    can be ran from only 't/'. Addition of 't/unit-tests/bin/' is to allow
    -    for running individual tests which is not currently possible via any
    -    'make' targets and also 'unit-tests-test-tool' target is also ran from
    -    the 't/unit-tests/bin' directory.
    +    And since we can construct strings with arbitrary bytes in C, let's
    +    also remove the test files which contain URLs with arbitrary bytes in
    +    the 't/t0110' directory and instead embed those URLs in the unit test
    +    code itself.

         [1]: https://lore.kernel.org/git/53CAC8EF.6020707@gmail.com/

    @@ t/t0110-urlmatch-normalization.sh (deleted)
     -
     -test_done

    + ## t/t0110/README (deleted) ##
    +@@
    +-The url data files in this directory contain URLs with characters
    +-in the range 0x01-0x1f and 0x7f-0xff to test the proper normalization
    +-of unprintable characters.
    +-
    +-A select few characters in the 0x01-0x1f range are skipped to help
    +-avoid problems running the test itself.
    +-
    +-The urls are in test files in this directory rather than being
    +-embedded in the test script for portability.
    +
    + ## t/t0110/url-1 (deleted) ##
    + Binary files t/t0110/url-1 and /dev/null differ
    +
    + ## t/t0110/url-10 (deleted) ##
    + Binary files t/t0110/url-10 and /dev/null differ
    +
    + ## t/t0110/url-11 (deleted) ##
    + Binary files t/t0110/url-11 and /dev/null differ
    +
    + ## t/t0110/url-2 (deleted) ##
    + Binary files t/t0110/url-2 and /dev/null differ
    +
    + ## t/t0110/url-3 (deleted) ##
    + Binary files t/t0110/url-3 and /dev/null differ
    +
    + ## t/t0110/url-4 (deleted) ##
    + Binary files t/t0110/url-4 and /dev/null differ
    +
    + ## t/t0110/url-5 (deleted) ##
    + Binary files t/t0110/url-5 and /dev/null differ
    +
    + ## t/t0110/url-6 (deleted) ##
    + Binary files t/t0110/url-6 and /dev/null differ
    +
    + ## t/t0110/url-7 (deleted) ##
    + Binary files t/t0110/url-7 and /dev/null differ
    +
    + ## t/t0110/url-8 (deleted) ##
    + Binary files t/t0110/url-8 and /dev/null differ
    +
    + ## t/t0110/url-9 (deleted) ##
    + Binary files t/t0110/url-9 and /dev/null differ
    +
      ## t/unit-tests/t-urlmatch-normalization.c (new) ##
     @@
     +#include "test-lib.h"
     +#include "urlmatch.h"
    -+#include "strbuf.h"
     +
     +static void check_url_normalizable(const char *url, int normalizable)
     +{
    @@ t/unit-tests/t-urlmatch-normalization.c (new)
     +	free(url2_norm);
     +}
     +
    -+static void check_normalized_url_from_file(const char *file, const char *expect)
    -+{
    -+	struct strbuf content = STRBUF_INIT, path = STRBUF_INIT;
    -+	char *cwd_basename;
    -+
    -+	if (!check_int(strbuf_getcwd(&path), ==, 0))
    -+		return;
    -+
    -+	cwd_basename = basename(path.buf);
    -+	if (!check(!strcmp(cwd_basename, "t") || !strcmp(cwd_basename, "bin"))) {
    -+		test_msg("BUG: unit-tests should be run from either 't/' or 't/unit-tests/bin' directory");
    -+		return;
    -+	}
    -+
    -+	strbuf_strip_suffix(&path, "/unit-tests/bin");
    -+	strbuf_addf(&path, "/unit-tests/t-urlmatch-normalization/%s", file);
    -+
    -+	if (!check_int(strbuf_read_file(&content, path.buf, 0), >, 0)) {
    -+		test_msg("failed to read from file '%s': %s", file, strerror(errno));
    -+	} else {
    -+		char *url_norm;
    -+
    -+		strbuf_trim_trailing_newline(&content);
    -+		url_norm = url_normalize(content.buf, NULL);
    -+		if (!check_str(url_norm, expect))
    -+			test_msg("input file: %s", file);
    -+		free(url_norm);
    -+	}
    -+
    -+	strbuf_release(&content);
    -+	strbuf_release(&path);
    -+}
    -+
     +static void check_normalized_url_length(const char *url, size_t len)
     +{
     +	struct url_info info;
    @@ t/unit-tests/t-urlmatch-normalization.c (new)
     +
     +static void t_url_high_bit(void)
     +{
    -+	check_normalized_url_from_file("url-1",
    -+			    "x://q/%01%02%03%04%05%06%07%08%0E%0F%10%11%12");
    -+	check_normalized_url_from_file("url-2",
    -+			    "x://q/%13%14%15%16%17%18%19%1B%1C%1D%1E%1F%7F");
    -+	check_normalized_url_from_file("url-3",
    -+			    "x://q/%80%81%82%83%84%85%86%87%88%89%8A%8B%8C%8D%8E%8F");
    -+	check_normalized_url_from_file("url-4",
    -+			    "x://q/%90%91%92%93%94%95%96%97%98%99%9A%9B%9C%9D%9E%9F");
    -+	check_normalized_url_from_file("url-5",
    -+			    "x://q/%A0%A1%A2%A3%A4%A5%A6%A7%A8%A9%AA%AB%AC%AD%AE%AF");
    -+	check_normalized_url_from_file("url-6",
    -+			    "x://q/%B0%B1%B2%B3%B4%B5%B6%B7%B8%B9%BA%BB%BC%BD%BE%BF");
    -+	check_normalized_url_from_file("url-7",
    -+			    "x://q/%C0%C1%C2%C3%C4%C5%C6%C7%C8%C9%CA%CB%CC%CD%CE%CF");
    -+	check_normalized_url_from_file("url-8",
    -+			    "x://q/%D0%D1%D2%D3%D4%D5%D6%D7%D8%D9%DA%DB%DC%DD%DE%DF");
    -+	check_normalized_url_from_file("url-9",
    -+			    "x://q/%E0%E1%E2%E3%E4%E5%E6%E7%E8%E9%EA%EB%EC%ED%EE%EF");
    -+	check_normalized_url_from_file("url-10",
    -+			    "x://q/%F0%F1%F2%F3%F4%F5%F6%F7%F8%F9%FA%FB%FC%FD%FE%FF");
    ++	check_normalized_url(
    ++		"x://q/\x01\x02\x03\x04\x05\x06\x07\x08\x0e\x0f\x10\x11\x12",
    ++		"x://q/%01%02%03%04%05%06%07%08%0E%0F%10%11%12");
    ++	check_normalized_url(
    ++		"x://q/\x13\x14\x15\x16\x17\x18\x19\x1b\x1c\x1d\x1e\x1f\x7f",
    ++		"x://q/%13%14%15%16%17%18%19%1B%1C%1D%1E%1F%7F");
    ++	check_normalized_url(
    ++		"x://q/\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f",
    ++		"x://q/%80%81%82%83%84%85%86%87%88%89%8A%8B%8C%8D%8E%8F");
    ++	check_normalized_url(
    ++		"x://q/\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f",
    ++		"x://q/%90%91%92%93%94%95%96%97%98%99%9A%9B%9C%9D%9E%9F");
    ++	check_normalized_url(
    ++		"x://q/\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf",
    ++		"x://q/%A0%A1%A2%A3%A4%A5%A6%A7%A8%A9%AA%AB%AC%AD%AE%AF");
    ++	check_normalized_url(
    ++		"x://q/\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf",
    ++		"x://q/%B0%B1%B2%B3%B4%B5%B6%B7%B8%B9%BA%BB%BC%BD%BE%BF");
    ++	check_normalized_url(
    ++		"x://q/\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf",
    ++		"x://q/%C0%C1%C2%C3%C4%C5%C6%C7%C8%C9%CA%CB%CC%CD%CE%CF");
    ++	check_normalized_url(
    ++		"x://q/\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf",
    ++		"x://q/%D0%D1%D2%D3%D4%D5%D6%D7%D8%D9%DA%DB%DC%DD%DE%DF");
    ++	check_normalized_url(
    ++		"x://q/\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef",
    ++		"x://q/%E0%E1%E2%E3%E4%E5%E6%E7%E8%E9%EA%EB%EC%ED%EE%EF");
    ++	check_normalized_url(
    ++		"x://q/\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff",
    ++		"x://q/%F0%F1%F2%F3%F4%F5%F6%F7%F8%F9%FA%FB%FC%FD%FE%FF");
     +}
     +
     +static void t_url_utf8_escape(void)
     +{
    -+	check_normalized_url_from_file("url-11",
    -+			    "x://q/%C2%80%DF%BF%E0%A0%80%EF%BF%BD%F0%90%80%80%F0%AF%BF%BD");
    ++	check_normalized_url(
    ++		"x://q/\xc2\x80\xdf\xbf\xe0\xa0\x80\xef\xbf\xbd\xf0\x90\x80\x80\xf0\xaf\xbf\xbd",
    ++		"x://q/%C2%80%DF%BF%E0%A0%80%EF%BF%BD%F0%90%80%80%F0%AF%BF%BD");
     +}
     +
     +static void t_url_username_pass(void)
    @@ t/unit-tests/t-urlmatch-normalization.c (new)
     +	TEST(t_url_equivalents(), "url equivalents");
     +	return test_done();
     +}
    -
    - ## t/t0110/README => t/unit-tests/t-urlmatch-normalization/README ##
    -
    - ## t/t0110/url-1 => t/unit-tests/t-urlmatch-normalization/url-1 ##
    -
    - ## t/t0110/url-10 => t/unit-tests/t-urlmatch-normalization/url-10 ##
    -
    - ## t/t0110/url-11 => t/unit-tests/t-urlmatch-normalization/url-11 ##
    -
    - ## t/t0110/url-2 => t/unit-tests/t-urlmatch-normalization/url-2 ##
    -
    - ## t/t0110/url-3 => t/unit-tests/t-urlmatch-normalization/url-3 ##
    -
    - ## t/t0110/url-4 => t/unit-tests/t-urlmatch-normalization/url-4 ##
    -
    - ## t/t0110/url-5 => t/unit-tests/t-urlmatch-normalization/url-5 ##
    -
    - ## t/t0110/url-6 => t/unit-tests/t-urlmatch-normalization/url-6 ##
    -
    - ## t/t0110/url-7 => t/unit-tests/t-urlmatch-normalization/url-7 ##
    -
    - ## t/t0110/url-8 => t/unit-tests/t-urlmatch-normalization/url-8 ##
    -
    - ## t/t0110/url-9 => t/unit-tests/t-urlmatch-normalization/url-9 ##

Message ID	20240814142057.94671-1-shyamthakkar001@gmail.com (mailing list archive)
State	Superseded
Headers	show Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4535B1B1417 for <git@vger.kernel.org>; Wed, 14 Aug 2024 14:21:25 +0000 (UTC) From: Ghanshyam Thakkar <shyamthakkar001@gmail.com> To: git@vger.kernel.org Cc: Junio C Hamano <gitster@pobox.com>, Karthik Nayak <karthik.188@gmail.com>, Patrick Steinhardt <ps@pks.im>, Christian Couder <christian.couder@gmail.com>, Ghanshyam Thakkar <shyamthakkar001@gmail.com>, Christian Couder <chriscool@tuxfamily.org>, Kaartic Sivaraam <kaartic.sivaraam@gmail.com> Subject: [GSoC][PATCH v3] t: migrate t0110-urlmatch-normalization to the new framework Date: Wed, 14 Aug 2024 19:50:46 +0530 Message-ID: <20240814142057.94671-1-shyamthakkar001@gmail.com> In-Reply-To: <20240813172432.55487-1-shyamthakkar001@gmail.com> References: <20240813172432.55487-1-shyamthakkar001@gmail.com> Precedence: bulk MIME-Version: 1.0 Content-Transfer-Encoding: 8bit
Series	[GSoC,v3] t: migrate t0110-urlmatch-normalization to the new framework \| expand [GSoC,v3] t: migrate t0110-urlmatch-normalization to the new framework

[GSoC,v3] t: migrate t0110-urlmatch-normalization to the new framework

Commit Message

Comments

Patch