From patchwork Thu Aug 5 11:25:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12421037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60BE7C4338F for ; Thu, 5 Aug 2021 11:25:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 421CB6112D for ; Thu, 5 Aug 2021 11:25:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241016AbhHEL0D (ORCPT ); Thu, 5 Aug 2021 07:26:03 -0400 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:45369 "EHLO out3-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240808AbhHEL0C (ORCPT ); Thu, 5 Aug 2021 07:26:02 -0400 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 90B7F5C008E; Thu, 5 Aug 2021 07:25:48 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Thu, 05 Aug 2021 07:25:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=mzvGiBrhHrgbfMgcKD5j1UozQZB duOVUts4/SP/2L9k=; b=L3k1l0XvLX+Pa9r9wAWKX7tSq0NvaEL5/OswEzZnTr5 21GERK5koD1qP0in2OpMT5xXV69hm6k1yucxcRmuqiSlCesZjLSMpXvvyF7Zr/iA yG2avby/AVJhRpgSMoqmxzhATv3LaLoBv9CMRKNMXkPSEEkcw2+GtVg2YNpdUmZY 831/jQ8X4kOOLZXQCJ1Y+YV/h0yd8B6RZoh62nRn+4napQT77E35vgOLixm2oya4 Qda1zq8Vc/T4N14ta2ldDi4A18c6RlKv7s6T9DVrlwUW2vCWS/QzlnaNOAoJW0Rk DtidT2d8fGsy59FI5blvFR/6bxAl61y90tllA6eLvow== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=mzvGiB rhHrgbfMgcKD5j1UozQZBduOVUts4/SP/2L9k=; b=J1zYDqBAt+4ngQmkswI9+E 5ELsgK2fiWVtAQGxyQRd2mixu6wry+d3NVkMhSoltCcmO8y2B+m15+0Dg2Jt/I6K lclwtWd4SOWU1nvzHQM6wb4DUZuuzdXQzXgZWLp9wqkkkZ5bTmHMQHGMgcrfn3KE ttWvzqiwir/GD0e0CKC54c2wsPq9AD5O9lWUPXejUwq3s9v0kHHJ+vv5ku/NEmEL TSNeVggahlWIm+Jfq+XBd/tIrLYn/WfEUKc2YCjFMHh9vXFnY1e26S5TiJILM6vP UZDw8nH7p9AO+WIP7ZS2oSXWV3zXKzSE2AV1Dm+mAVFj1GbrxOVd/S5AgWePCsPQ == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrieelgdefjecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvuffkfhggtggujgesghdtreertddtjeenucfhrhhomheprfgrthhrihgt khcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvghrnh ephefgjeeuveejteduhefgffefffdvjeefjeeivdekfffgkeeugfehveetueefleeknecu vehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhssehpkh hsrdhimh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 5 Aug 2021 07:25:47 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 38af3e60 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 5 Aug 2021 11:25:46 +0000 (UTC) Date: Thu, 5 Aug 2021 13:25:45 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , Felipe Contreras , SZEDER =?iso-8859-1?q?G=E1bor?= , Chris Torek , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Taylor Blau Subject: [PATCH v4 6/6] revision: avoid hitting packfiles when commits are in commit-graph Message-ID: <900c5a9c60282e0fb57840305cbab013ffa398f1.1628162156.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When queueing references in git-rev-list(1), we try to either reuse an already parsed object or alternatively we load the object header from disk in order to determine its type. This is inefficient for commits though in cases where we have a commit graph available: instead of hitting the real object on disk to determine its type, we may instead search the object graph for the object ID. In case it's found, we can directly fill in the commit object, otherwise we can still hit the disk to determine the object's type. Expose a new function `parse_commit_in_graph_gently()`, which fills in an object of unknown type in case we find its object ID in the graph. This provides a big performance win in cases where there is a commit-graph available in the repository in case we load lots of references. The following has been executed in a real-world repository with about 2.2 million refs: Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev Time (mean ± σ): 4.508 s ± 0.039 s [User: 4.131 s, System: 0.377 s] Range (min … max): 4.455 s … 4.576 s 10 runs Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev Time (mean ± σ): 3.072 s ± 0.031 s [User: 2.707 s, System: 0.365 s] Range (min … max): 3.040 s … 3.144 s 10 runs Summary 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran 1.47 ± 0.02 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' Signed-off-by: Patrick Steinhardt --- commit-graph.c | 26 ++++++++++++++++++++++++++ commit-graph.h | 7 +++++++ revision.c | 17 +++++++++++++++-- 3 files changed, 48 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index 8c4c7262c8..cc7136243d 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -891,6 +891,32 @@ static int find_commit_pos_in_graph(struct commit *item, struct commit_graph *g, } } +int parse_commit_in_graph_gently(struct repository *repo, struct object *object) +{ + struct commit *commit; + uint32_t pos; + + if (object->parsed) { + if (object->type != OBJ_COMMIT) + return -1; + return 0; + } + + if (!repo->objects->commit_graph) + return -1; + + if (!search_commit_pos_in_graph(&object->oid, repo->objects->commit_graph, &pos)) + return -1; + + commit = object_as_type(object, OBJ_COMMIT, 1); + if (!commit) + return -1; + if (!fill_commit_in_graph(repo, commit, repo->objects->commit_graph, pos)) + return -1; + + return 0; +} + static int parse_commit_in_graph_one(struct repository *r, struct commit_graph *g, struct commit *item) diff --git a/commit-graph.h b/commit-graph.h index 96c24fb577..e2e93b0ee1 100644 --- a/commit-graph.h +++ b/commit-graph.h @@ -40,6 +40,13 @@ int open_commit_graph(const char *graph_file, int *fd, struct stat *st); */ int parse_commit_in_graph(struct repository *r, struct commit *item); +/* + * Given an object of unknown type, try to fill in the object in case it is a + * commit part of the commit-graph. Returns 0 if the object is a parsed commit + * or if it could be filled in via the commit graph, otherwise it returns -1. + */ +int parse_commit_in_graph_gently(struct repository *repo, struct object *object); + /* * It is possible that we loaded commit contents from the commit buffer, * but we also want to ensure the commit-graph content is correctly diff --git a/revision.c b/revision.c index 25f4784fdd..318b43026e 100644 --- a/revision.c +++ b/revision.c @@ -362,8 +362,21 @@ static struct object *get_reference(struct rev_info *revs, const char *name, struct object *object = lookup_unknown_object(revs->repo, oid); if (object->type == OBJ_NONE) { - int type = oid_object_info(revs->repo, oid, NULL); - if (type < 0 || !object_as_type(object, type, 1)) { + /* + * It's likely that the reference points to a commit, so we + * first try to look it up via the commit-graph. If successful, + * then we know it's a commit and don't have to unpack the + * object header. We still need to assert that the object + * exists, but given that we don't request any info about the + * object this is a lot faster than `oid_object_info()`. + */ + if (parse_commit_in_graph_gently(revs->repo, object) < 0) { + int type = oid_object_info(revs->repo, oid, NULL); + if (type < 0 || !object_as_type(object, type, 1)) { + object = NULL; + goto out; + } + } else if (!repo_has_object_file(revs->repo, oid)) { object = NULL; goto out; }