From patchwork Tue Apr 13 07:15:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 12199477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1746C433ED for ; Tue, 13 Apr 2021 07:16:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9C829613B7 for ; Tue, 13 Apr 2021 07:16:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237403AbhDMHQU (ORCPT ); Tue, 13 Apr 2021 03:16:20 -0400 Received: from cloud.peff.net ([104.130.231.41]:50754 "EHLO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229802AbhDMHQP (ORCPT ); Tue, 13 Apr 2021 03:16:15 -0400 Received: (qmail 28512 invoked by uid 109); 13 Apr 2021 07:15:55 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with ESMTP; Tue, 13 Apr 2021 07:15:55 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 22992 invoked by uid 111); 13 Apr 2021 07:15:56 -0000 Received: from coredump.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.2) by peff.net (qpsmtpd/0.94) with (TLS_AES_256_GCM_SHA384 encrypted) ESMTPS; Tue, 13 Apr 2021 03:15:56 -0400 Authentication-Results: peff.net; auth=none Date: Tue, 13 Apr 2021 03:15:54 -0400 From: Jeff King To: SZEDER =?utf-8?b?R8OhYm9y?= Cc: Rafael Silva , Jonathan Tan , Git Mailing List Subject: [PATCH 1/3] is_promisor_object(): free tree buffer after parsing Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org To get the list of all promisor objects, we not only include all objects in promisor packs, but also parse each of those objects to see which objects they reference. After parsing a tree object, the tree->buffer field will remain populated until we explicitly free it. So in a partial clone of blob:none, for example, we are essentially reading every tree in the repository (since they're all in the initial promisor pack), and keeping all of their uncompressed contents in memory at once. This patch frees the tree buffers after we've finished marking all of their reachable objects. We shouldn't need to do this for any other object type. While we are using some extra memory to store the structs, no other object type stores the whole contents in its parsed form (we do sometimes hold on to commit buffers, but less so these days due to commit graphs, plus most commands which care about promisor objects turn off the save_commit_buffer global). Even for a moderate-sized repository like git.git, this patch drops the peak heap (as measured by massif) for git-fsck from ~1.7GB to ~138MB. Fsck is a good candidate for measuring here because it doesn't interact with the promisor code except to call is_promisor_object(), so we can isolate just this problem. The added perf test shows only a tiny improvement on my machine for git.git, since 1.7GB isn't enough to cause any real memory pressure: Test HEAD^ HEAD -------------------------------------------------------------------------------- 5600.4: fsck 21.26(20.90+0.35) 20.84(20.79+0.04) -2.0% With linux.git the absolute change is a bit bigger, though still a small percentage: Test HEAD^ HEAD ----------------------------------------------------------------------------- 5600.4: fsck 262.26(259.13+3.12) 254.92(254.62+0.29) -2.8% I didn't have the patience to run it under massif with linux.git, but it's probably on the order of about 14GB improvement, since that's the sum of the sizes of all of the uncompressed trees (but still isn't enough to create memory pressure on this particular machine, which has 64GB of RAM). Smaller machines would probably see a bigger effect on runtime (and sadly our perf suite does not measure peak heap). Signed-off-by: Jeff King --- packfile.c | 1 + t/perf/p5600-partial-clone.sh | 4 ++++ 2 files changed, 5 insertions(+) diff --git a/packfile.c b/packfile.c index 8668345d93..b79cbc8cd4 100644 --- a/packfile.c +++ b/packfile.c @@ -2247,6 +2247,7 @@ static int add_promisor_object(const struct object_id *oid, return 0; while (tree_entry_gently(&desc, &entry)) oidset_insert(set, &entry.oid); + free_tree_buffer(tree); } else if (obj->type == OBJ_COMMIT) { struct commit *commit = (struct commit *) obj; struct commit_list *parents = commit->parents; diff --git a/t/perf/p5600-partial-clone.sh b/t/perf/p5600-partial-clone.sh index 3e04bd2ae1..754aaec3dc 100755 --- a/t/perf/p5600-partial-clone.sh +++ b/t/perf/p5600-partial-clone.sh @@ -23,4 +23,8 @@ test_perf 'checkout of result' ' git -C worktree checkout -f ' +test_perf 'fsck' ' + git -C bare.git fsck +' + test_done From patchwork Tue Apr 13 07:16:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 12199479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EC81C43461 for ; Tue, 13 Apr 2021 07:16:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E78516120E for ; Tue, 13 Apr 2021 07:16:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237967AbhDMHQ7 (ORCPT ); Tue, 13 Apr 2021 03:16:59 -0400 Received: from cloud.peff.net ([104.130.231.41]:50766 "EHLO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237707AbhDMHQ5 (ORCPT ); Tue, 13 Apr 2021 03:16:57 -0400 Received: (qmail 28530 invoked by uid 109); 13 Apr 2021 07:16:37 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with ESMTP; Tue, 13 Apr 2021 07:16:37 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 23019 invoked by uid 111); 13 Apr 2021 07:16:38 -0000 Received: from coredump.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.2) by peff.net (qpsmtpd/0.94) with (TLS_AES_256_GCM_SHA384 encrypted) ESMTPS; Tue, 13 Apr 2021 03:16:38 -0400 Authentication-Results: peff.net; auth=none Date: Tue, 13 Apr 2021 03:16:36 -0400 From: Jeff King To: SZEDER =?utf-8?b?R8OhYm9y?= Cc: Rafael Silva , Jonathan Tan , Git Mailing List Subject: [PATCH 2/3] lookup_unknown_object(): take a repository argument Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org All of the other lookup_foo() functions take a repository argument, but lookup_unknown_object() was never converted, and it uses the_repository internally. Let's fix that. We could leave a wrapper that uses the_repository, but there aren't that many calls, so we'll just convert them all. I looked briefly at each site to see if we had a repository struct (besides the_repository) we could pass, but none of them do (so this conversion to pass the_repository is a pure noop in each case, though it does take us one step closer to eventually getting rid of the_repository). Signed-off-by: Jeff King --- builtin/fsck.c | 2 +- builtin/pack-objects.c | 2 +- http-push.c | 2 +- object.c | 7 +++---- object.h | 2 +- refs.c | 2 +- t/helper/test-example-decorate.c | 6 +++--- upload-pack.c | 2 +- walker.c | 2 +- 9 files changed, 13 insertions(+), 14 deletions(-) diff --git a/builtin/fsck.c b/builtin/fsck.c index 70ff95837a..e6a80e5404 100644 --- a/builtin/fsck.c +++ b/builtin/fsck.c @@ -725,7 +725,7 @@ static int fsck_cache_tree(struct cache_tree *it) static void mark_object_for_connectivity(const struct object_id *oid) { - struct object *obj = lookup_unknown_object(oid); + struct object *obj = lookup_unknown_object(the_repository, oid); obj->flags |= HAS_OBJ; } diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 525c2d8552..c1186f50a3 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3386,7 +3386,7 @@ static void add_objects_in_unpacked_packs(void) for (i = 0; i < p->num_objects; i++) { nth_packed_object_id(&oid, p, i); - o = lookup_unknown_object(&oid); + o = lookup_unknown_object(the_repository, &oid); if (!(o->flags & OBJECT_ADDED)) mark_in_pack_object(o, p, &in_pack); o->flags |= OBJECT_ADDED; diff --git a/http-push.c b/http-push.c index b60d5fcc85..813123242e 100644 --- a/http-push.c +++ b/http-push.c @@ -1436,7 +1436,7 @@ static void one_remote_ref(const char *refname) * may be required for updating server info later. */ if (repo->can_update_info_refs && !has_object_file(&ref->old_oid)) { - obj = lookup_unknown_object(&ref->old_oid); + obj = lookup_unknown_object(the_repository, &ref->old_oid); fprintf(stderr, " fetch %s for %s\n", oid_to_hex(&ref->old_oid), refname); add_fetch_request(obj); diff --git a/object.c b/object.c index 78343781ae..14188453c5 100644 --- a/object.c +++ b/object.c @@ -177,12 +177,11 @@ void *object_as_type(struct object *obj, enum object_type type, int quiet) } } -struct object *lookup_unknown_object(const struct object_id *oid) +struct object *lookup_unknown_object(struct repository *r, const struct object_id *oid) { - struct object *obj = lookup_object(the_repository, oid); + struct object *obj = lookup_object(r, oid); if (!obj) - obj = create_object(the_repository, oid, - alloc_object_node(the_repository)); + obj = create_object(r, oid, alloc_object_node(r)); return obj; } diff --git a/object.h b/object.h index 59daadce21..87a6da47c8 100644 --- a/object.h +++ b/object.h @@ -145,7 +145,7 @@ struct object *parse_object_or_die(const struct object_id *oid, const char *name struct object *parse_object_buffer(struct repository *r, const struct object_id *oid, enum object_type type, unsigned long size, void *buffer, int *eaten_p); /** Returns the object, with potentially excess memory allocated. **/ -struct object *lookup_unknown_object(const struct object_id *oid); +struct object *lookup_unknown_object(struct repository *r, const struct object_id *oid); struct object_list *object_list_insert(struct object *item, struct object_list **list_p); diff --git a/refs.c b/refs.c index 261fd82beb..1616c7554a 100644 --- a/refs.c +++ b/refs.c @@ -337,7 +337,7 @@ static int filter_refs(const char *refname, const struct object_id *oid, enum peel_status peel_object(const struct object_id *name, struct object_id *oid) { - struct object *o = lookup_unknown_object(name); + struct object *o = lookup_unknown_object(the_repository, name); if (o->type == OBJ_NONE) { int type = oid_object_info(the_repository, name, NULL); diff --git a/t/helper/test-example-decorate.c b/t/helper/test-example-decorate.c index c8a1cde7d2..b9d1200eb9 100644 --- a/t/helper/test-example-decorate.c +++ b/t/helper/test-example-decorate.c @@ -26,8 +26,8 @@ int cmd__example_decorate(int argc, const char **argv) * Add 2 objects, one with a non-NULL decoration and one with a NULL * decoration. */ - one = lookup_unknown_object(&one_oid); - two = lookup_unknown_object(&two_oid); + one = lookup_unknown_object(the_repository, &one_oid); + two = lookup_unknown_object(the_repository, &two_oid); ret = add_decoration(&n, one, &decoration_a); if (ret) BUG("when adding a brand-new object, NULL should be returned"); @@ -56,7 +56,7 @@ int cmd__example_decorate(int argc, const char **argv) ret = lookup_decoration(&n, two); if (ret != &decoration_b) BUG("lookup should return added declaration"); - three = lookup_unknown_object(&three_oid); + three = lookup_unknown_object(the_repository, &three_oid); ret = lookup_decoration(&n, three); if (ret) BUG("lookup for unknown object should return NULL"); diff --git a/upload-pack.c b/upload-pack.c index e19583ae0f..5c1cd19612 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -1153,7 +1153,7 @@ static void receive_needs(struct upload_pack_data *data, static int mark_our_ref(const char *refname, const char *refname_full, const struct object_id *oid) { - struct object *o = lookup_unknown_object(oid); + struct object *o = lookup_unknown_object(the_repository, oid); if (ref_is_hidden(refname, refname_full)) { o->flags |= HIDDEN_REF; diff --git a/walker.c b/walker.c index 4984bf8b3d..c5e2921979 100644 --- a/walker.c +++ b/walker.c @@ -298,7 +298,7 @@ int walker_fetch(struct walker *walker, int targets, char **target, error("Could not interpret response from server '%s' as something to pull", target[i]); goto done; } - if (process(walker, lookup_unknown_object(&oids[i]))) + if (process(walker, lookup_unknown_object(the_repository, &oids[i]))) goto done; } From patchwork Tue Apr 13 07:17:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff King X-Patchwork-Id: 12199507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44816C433ED for ; Tue, 13 Apr 2021 07:17:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 27BE1613B2 for ; Tue, 13 Apr 2021 07:17:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239018AbhDMHSN (ORCPT ); Tue, 13 Apr 2021 03:18:13 -0400 Received: from cloud.peff.net ([104.130.231.41]:50780 "EHLO cloud.peff.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239201AbhDMHSJ (ORCPT ); Tue, 13 Apr 2021 03:18:09 -0400 Received: (qmail 28557 invoked by uid 109); 13 Apr 2021 07:17:49 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with ESMTP; Tue, 13 Apr 2021 07:17:49 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 23049 invoked by uid 111); 13 Apr 2021 07:17:50 -0000 Received: from coredump.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.2) by peff.net (qpsmtpd/0.94) with (TLS_AES_256_GCM_SHA384 encrypted) ESMTPS; Tue, 13 Apr 2021 03:17:50 -0400 Authentication-Results: peff.net; auth=none Date: Tue, 13 Apr 2021 03:17:48 -0400 From: Jeff King To: SZEDER =?utf-8?b?R8OhYm9y?= Cc: Rafael Silva , Jonathan Tan , Git Mailing List Subject: [PATCH 3/3] revision: avoid parsing with --exclude-promisor-objects Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When --exclude-promisor-objects is given, before traversing any objects we iterate over all of the objects in any promisor packs, marking them as UNINTERESTING and SEEN. We turn the oid we get from iterating the pack into an object with parse_object(), but this has two problems: - it's slow; we are zlib inflating (and reconstructing from deltas) every byte of every object in the packfile - it leaves the tree buffers attached to their structs, which means our heap usage will grow to store every uncompressed tree simultaneously. This can be gigabytes. We can obviously fix the second by freeing the tree buffers after we've parsed them. But we can observe that the function doesn't look at the object contents at all! The only reason we call parse_object() is that we need a "struct object" on which to set the flags. There are two options here: - we can look up just the object type via oid_object_info(), and then call the appropriate lookup_foo() function - we can call lookup_unknown_object(), which gives us an OBJ_NONE struct (which will get auto-converted later by object_as_type() via calls to lookup_commit(), etc). The first one is closer to the current code, but we do pay the price to look up the type for each object. The latter should be more efficient in CPU, though it wastes a little bit of memory (the "unknown" object structs are a union of all object types, so some of the structs are bigger than they need to be). It also runs the risk of triggering a latent bug in code that calls lookup_object() directly but isn't ready to handle OBJ_NONE (such code would already be buggy, but we use lookup_unknown_object() infrequently enough that it might be hiding). I went with the second option here. I don't think the risk is high (and we'd want to find and fix any such bugs anyway), and it should be more efficient overall. The new tests in p5600 show off the improvement (this is on git.git): Test HEAD^ HEAD ------------------------------------------------------------------------------- 5600.5: count commits 0.37(0.37+0.00) 0.38(0.38+0.00) +2.7% 5600.6: count non-promisor commits 11.74(11.37+0.37) 0.04(0.03+0.00) -99.7% The improvement is particularly big in this script because _every_ object in the newly-cloned partial repo is a promisor object. So after marking them all, there's nothing left to traverse. Signed-off-by: Jeff King --- revision.c | 2 +- t/perf/p5600-partial-clone.sh | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/revision.c b/revision.c index 553c0faa9b..7e73dafd96 100644 --- a/revision.c +++ b/revision.c @@ -3271,7 +3271,7 @@ static int mark_uninteresting(const struct object_id *oid, void *cb) { struct rev_info *revs = cb; - struct object *o = parse_object(revs->repo, oid); + struct object *o = lookup_unknown_object(revs->repo, oid); o->flags |= UNINTERESTING | SEEN; return 0; } diff --git a/t/perf/p5600-partial-clone.sh b/t/perf/p5600-partial-clone.sh index 754aaec3dc..ca785a3341 100755 --- a/t/perf/p5600-partial-clone.sh +++ b/t/perf/p5600-partial-clone.sh @@ -27,4 +27,12 @@ test_perf 'fsck' ' git -C bare.git fsck ' +test_perf 'count commits' ' + git -C bare.git rev-list --all --count +' + +test_perf 'count non-promisor commits' ' + git -C bare.git rev-list --all --count --exclude-promisor-objects +' + test_done