diff mbox series

[v2,2/7] test-tool genzeros: generate large amounts of data more efficiently

Message ID 6edcbae372ef63bd75ca6cc2d181f7506f35880f.1635454237.git.gitgitgadget@gmail.com (mailing list archive)
State Superseded
Headers show
Series Allow clean/smudge filters to handle huge files in the LLP64 data model | expand

Commit Message

Johannes Schindelin Oct. 28, 2021, 8:50 p.m. UTC 
From: Johannes Schindelin <johannes.schindelin@gmx.de>

In this developer's tests, producing one gigabyte worth of NULs in a
busy loop that writes out individual bytes, unbuffered, took ~27sec.
Writing chunked 256kB buffers instead only took ~0.6sec

This matters because we are about to introduce a pair of test cases that
want to be able to produce 5GB of NULs, and we cannot use `/dev/zero`
because of the HP NonStop platform's lack of support for that device.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 t/helper/test-genzeros.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

Comments

Junio C Hamano Oct. 28, 2021, 10:55 p.m. UTC | #1 
"Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com>
writes:

>  int cmd__genzeros(int argc, const char **argv)
>  {
> +	/* static, so that it is NUL-initialized */
> +	static char zeros[256 * 1024];

Of course, another benefit is that you do not waste 256kB of
stackspace for this array.  We could probably even mark it as
"const" to emphasize that it is initialized with and will stay
to be bunch of NULs.

>  	intmax_t count;
> +	ssize_t n;
>  
>  	if (argc > 2) {
>  		fprintf(stderr, "usage: %s [<count>]\n", argv[0]);
> @@ -12,9 +15,19 @@ int cmd__genzeros(int argc, const char **argv)
>  
>  	count = argc > 1 ? strtoimax(argv[1], NULL, 0) : -1;
>  
> -	while (count < 0 || count--) {
> -		if (putchar(0) == EOF)

> +	/* Writing out individual NUL bytes is slow... */
> +	while (count < 0)
> +		if (write(1, zeros, ARRAY_SIZE(zeros) < 0))
>  			return -1;

Be careful where you close your parentheses.  

I wonder if your compiler didn't warn that ARRAY_SIZE(x) will never
be negative?

> +	while (count > 0) {
> +		n = write(1, zeros, count < ARRAY_SIZE(zeros) ?
> +			  count : ARRAY_SIZE(zeros));

This side looks OK.

> +		if (n < 0)
> +			return -1;
> +
> +		count -= n;
>  	}
>  
>  	return 0;
diff mbox series

Patch

diff --git a/t/helper/test-genzeros.c b/t/helper/test-genzeros.c
index b1197e91a89..c061c429da9 100644
--- a/t/helper/test-genzeros.c
+++ b/t/helper/test-genzeros.c
@@ -3,7 +3,10 @@ 
 
 int cmd__genzeros(int argc, const char **argv)
 {
+	/* static, so that it is NUL-initialized */
+	static char zeros[256 * 1024];
 	intmax_t count;
+	ssize_t n;
 
 	if (argc > 2) {
 		fprintf(stderr, "usage: %s [<count>]\n", argv[0]);
@@ -12,9 +15,19 @@  int cmd__genzeros(int argc, const char **argv)
 
 	count = argc > 1 ? strtoimax(argv[1], NULL, 0) : -1;
 
-	while (count < 0 || count--) {
-		if (putchar(0) == EOF)
+	/* Writing out individual NUL bytes is slow... */
+	while (count < 0)
+		if (write(1, zeros, ARRAY_SIZE(zeros) < 0))
 			return -1;
+
+	while (count > 0) {
+		n = write(1, zeros, count < ARRAY_SIZE(zeros) ?
+			  count : ARRAY_SIZE(zeros));
+
+		if (n < 0)
+			return -1;
+
+		count -= n;
 	}
 
 	return 0;