[v5] convert: return early when not tracing

Commit Message

Haritha D July 30, 2024, 3:42 a.m. UTC

From: D Harithamma <[e-mail]>

When Git adds a file requiring encoding conversion and tracing of encoding
conversion is not requested via the GIT_TRACE_WORKING_TREE_ENCODING
environment variable, the `trace_encoding()` function still allocates &
prepares "human readable" copies of the file contents before and after
conversion to show in the trace. This results in a high memory footprint
and increased runtime without providing any user-visible benefit.

This fix introduces an early exit from the `trace_encoding()` function
when tracing is not requested, preventing unnecessary memory allocation
and processing.

Signed-off-by: Harithamma D <[e-mail]>
---
    Fix to avoid high memory footprint
    
    This fix avoids high memory footprint when adding files that require
    conversion
    
    Git has a trace_encoding routine that prints trace output when
    GIT_TRACE_WORKING_TREE_ENCODING=1 is set. This environment variable is
    used to debug the encoding contents. When a 40MB file is added, it
    requests close to 1.8GB of storage from xrealloc which can lead to out
    of memory errors. However, the check for GIT_TRACE_WORKING_TREE_ENCODING
    is done after the string is allocated. This resolves high memory
    footprints even when GIT_TRACE_WORKING_TREE_ENCODING is not active. This
    fix adds an early exit to avoid the unnecessary memory allocation.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1744%2FHarithaIBM%2FmemFootprintFix-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1744/HarithaIBM/memFootprintFix-v5
Pull-Request: https://github.com/git/git/pull/1744

Range-diff vs v4:

 1:  50758a4fb94 ! 1:  e2518b28f1c convert: avoid high memory footprint
     @@ Metadata
      Author: D Harithamma <[e-mail]>
      
       ## Commit message ##
     -    convert: avoid high memory footprint
     +    convert: return early when not tracing
      
          When Git adds a file requiring encoding conversion and tracing of encoding
          conversion is not requested via the GIT_TRACE_WORKING_TREE_ENCODING


 convert.c | 3 +++
 1 file changed, 3 insertions(+)


base-commit: 557ae147e6cdc9db121269b058c757ac5092f9c9

Comments

Junio C Hamano July 31, 2024, 2:42 a.m. UTC | #1

"Haritha  via GitGitGadget" <[e-mail]> writes:

> From: D Harithamma <[e-mail]>
>
> When Git adds a file requiring encoding conversion and tracing of encoding
> conversion is not requested via the GIT_TRACE_WORKING_TREE_ENCODING
> environment variable, the `trace_encoding()` function still allocates &
> prepares "human readable" copies of the file contents before and after
> conversion to show in the trace. This results in a high memory footprint
> and increased runtime without providing any user-visible benefit.
>
> This fix introduces an early exit from the `trace_encoding()` function
> when tracing is not requested, preventing unnecessary memory allocation
> and processing.
>
> Signed-off-by: Harithamma D <[e-mail]>
> ---

It seems that you forgot to adjust to

 https://lore.kernel.org/git/xmqqed7gyyyd.fsf@gitster.g/

where I asked you to be consistent in the authorship name and sign
off.

For now, as I like to allow "git shortlog --author=..." to group
contributions by a single author to a single bucket, I'll rewrite
both to the same name as used in d254e650 (build: support z/OS
(OS/390)., 2024-03-06), but will not merge it down to 'next' before
I hear what your response is.

Thanks.

Haritha D July 31, 2024, 9:32 a.m. UTC | #2

Hey Junio,

You're right. I forgot to respond to that question.
I used full name based on Jeff's suggestion. 
I will ensure it remains consistent.
Thank you reminding me.

Best Regards
Haritha

On 31/07/24, 8:12 AM, "Junio C Hamano" <[e-mail] <mailto:[e-mail]>> wrote:


"Haritha via GitGitGadget" <[e-mail] <mailto:[e-mail]>> writes:


> From: D Harithamma <[e-mail] <mailto:[e-mail]>>
>
> When Git adds a file requiring encoding conversion and tracing of encoding
> conversion is not requested via the GIT_TRACE_WORKING_TREE_ENCODING
> environment variable, the `trace_encoding()` function still allocates &
> prepares "human readable" copies of the file contents before and after
> conversion to show in the trace. This results in a high memory footprint
> and increased runtime without providing any user-visible benefit.
>
> This fix introduces an early exit from the `trace_encoding()` function
> when tracing is not requested, preventing unnecessary memory allocation
> and processing.
>
> Signed-off-by: Harithamma D <[e-mail] <mailto:[e-mail]>>
> ---


It seems that you forgot to adjust to


https://lore.kernel.org/git/xmqqed7gyyyd.fsf@gitster.g/ <https://lore.kernel.org/git/xmqqed7gyyyd.fsf@gitster.g/> 


where I asked you to be consistent in the authorship name and sign
off.


For now, as I like to allow "git shortlog --author=..." to group
contributions by a single author to a single bucket, I'll rewrite
both to the same name as used in d254e650 (build: support z/OS
(OS/390)., 2024-03-06), but will not merge it down to 'next' before
I hear what your response is.


Thanks.

Patch

diff --git a/convert.c b/convert.c
index d8737fe0f2d..c4ddc4de81b 100644
--- a/convert.c
+++ b/convert.c
@@ -324,6 +324,9 @@  static void trace_encoding(const char *context, const char *path,
 	struct strbuf trace = STRBUF_INIT;
 	int i;
 
+	if (!trace_want(&coe))
+		return;
+
 	strbuf_addf(&trace, "%s (%s, considered %s):\n", context, path, encoding);
 	for (i = 0; i < len && buf; ++i) {
 		strbuf_addf(