From patchwork Fri Jan 28 10:17:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12728181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F7F5C433EF for ; Fri, 28 Jan 2022 10:17:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346582AbiA1KRe (ORCPT ); Fri, 28 Jan 2022 05:17:34 -0500 Received: from wout4-smtp.messagingengine.com ([64.147.123.20]:54939 "EHLO wout4-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347467AbiA1KRH (ORCPT ); Fri, 28 Jan 2022 05:17:07 -0500 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 094A332020B8 for ; Fri, 28 Jan 2022 05:17:06 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Fri, 28 Jan 2022 05:17:07 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm3; bh=khyC3E3k6PB1acc0N8tSw3L8A8FjOL4+mYr16K 7y878=; b=q+vrHhLIj4SQOuFMWOn+vBmkoZIN3CtXeXyWIo5cI2dMmLSEP3D5mP 9tigWezp2cX+9zGSnZTF9BjZhdX67i8q6Q6XWy021wV9wyarP8S9vb3cZ9dSQIJ3 I5cbNjjmvTp+dK/HuD34Qsn57HOQ7ji8hWz2YQXM20GltuZnny91y9eV7ilFP8cp lgIf3mPO192Fvk9aY/ADH+StJY9RW2H4hTZhV/2ZG9Un42E+pEEmkFGDCnxxCdSj gP/o3V8GFmJWuhIi57mTg4Ybzxb3mnQcWZM9VW5tPhcvYz7N66UOPLtyymusFPY+ Cfq1gbj7MmeM4bxn9nGJLLcs+g5t7ipQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=khyC3E3k6PB1acc0N 8tSw3L8A8FjOL4+mYr16K7y878=; b=DvbQIjVXmtVLZK/+hq4d0yDLZWyZRy8IT 8N+kmEhYapMKWSGNBludRiPA9J8uz2y6Qj2OZoeKUbKWSK5qASOUxKl5g1pjFKPq rQ6EzY97+2q+ZfuQBn4MMaTW7xflktW/sbWoJE1vADyhBAYxBKAQbe/20PEC1rx5 boY+0it0Zj7oino/BdQM4Oqjwc+aFr0WPk5UwPpFq9gM+6WzC2TMHwPiiWjxQM/U 6Dqj99bwtZEluCGRefIaeOUTO8fQCwAyIO/VxK0nzBSzxsuIM/44yjKCykpxRl6n Lw2wqbo37QWfxc934pLQRoDtZrmTUANuzpppTiWVeS/k53g5zg1jQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrfeehgddugecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtderre dttdejnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhk shdrihhmqeenucggtffrrghtthgvrhhnpeehgfejueevjeetudehgffffeffvdejfeejie dvkeffgfekuefgheevteeufeelkeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 28 Jan 2022 05:17:06 -0500 (EST) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id c8475f5c (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Fri, 28 Jan 2022 10:17:04 +0000 (UTC) Date: Fri, 28 Jan 2022 11:17:03 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 1/2] fetch-pack: use commit-graph when computing cutoff Message-ID: <31cf8f87a149c0fc8013b869e0e30364f3c60e01.1643364888.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org During packfile negotiation we iterate over all refs announced by the remote side to check whether their IDs refer to commits already known to us. If a commit is known to us already, then its date is a potential cutoff point for commits we have in common with the remote side. There is potentially a lot of commits announced by the remote depending on how many refs there are in the remote repository, and for every one of them we need to search for it in our object database and, if found, parse the corresponding object to find out whether it is a candidate for the cutoff date. This can be sped up by trying to look up commits via the commit-graph first, which is a lot more efficient. One thing to keep in mind though is that the commit-graph corrects committer dates: * A commit with at least one parent has corrected committer date equal to the maximum of its commiter date and one more than the largest corrected committer date among its parents. As a result, it may be that the commit date we load via the commit graph is more recent than it would have been when loaded via the ODB, and as a result we may also choose a more recent cutoff point. But as the code documents, this is only a heuristic and it is okay if we determine a wrong cutoff date. The worst that can happen is that we report more commits as HAVEs to the server when using corrected dates. Loading commits via the commit-graph is typically much faster than loading commits via the object database. Benchmarks in a repository with about 2,1 million refs and an up-to-date commit-graph show a 20% speedup when mirror-fetching: Benchmark 1: git fetch --atomic +refs/*:refs/* (v2.35.0) Time (mean ± σ): 75.264 s ± 1.115 s [User: 68.199 s, System: 10.094 s] Range (min … max): 74.145 s … 76.862 s 5 runs Benchmark 2: git fetch --atomic +refs/*:refs/* (HEAD) Time (mean ± σ): 62.350 s ± 0.854 s [User: 55.412 s, System: 9.976 s] Range (min … max): 61.224 s … 63.216 s 5 runs Summary 'git fetch --atomic +refs/*:refs/* (HEAD)' ran 1.21 ± 0.02 times faster than 'git fetch --atomic +refs/*:refs/* (v2.35.0)' Signed-off-by: Patrick Steinhardt --- fetch-pack.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index dd6ec449f2..c5967e228e 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -696,26 +696,30 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator, trace2_region_enter("fetch-pack", "parse_remote_refs_and_find_cutoff", NULL); for (ref = *refs; ref; ref = ref->next) { - struct object *o; + struct commit *commit; - if (!has_object_file_with_flags(&ref->old_oid, + commit = lookup_commit_in_graph(the_repository, &ref->old_oid); + if (!commit) { + struct object *o; + + if (!has_object_file_with_flags(&ref->old_oid, OBJECT_INFO_QUICK | - OBJECT_INFO_SKIP_FETCH_OBJECT)) - continue; - o = parse_object(the_repository, &ref->old_oid); - if (!o) - continue; + OBJECT_INFO_SKIP_FETCH_OBJECT)) + continue; + o = parse_object(the_repository, &ref->old_oid); + if (!o || o->type != OBJ_COMMIT) + continue; + + commit = (struct commit *)o; + } /* * We already have it -- which may mean that we were * in sync with the other side at some time after * that (it is OK if we guess wrong here). */ - if (o->type == OBJ_COMMIT) { - struct commit *commit = (struct commit *)o; - if (!cutoff || cutoff < commit->date) - cutoff = commit->date; - } + if (!cutoff || cutoff < commit->date) + cutoff = commit->date; } trace2_region_leave("fetch-pack", "parse_remote_refs_and_find_cutoff", NULL); From patchwork Fri Jan 28 10:17:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12728182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89D70C433EF for ; Fri, 28 Jan 2022 10:17:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347176AbiA1KRh (ORCPT ); Fri, 28 Jan 2022 05:17:37 -0500 Received: from wout4-smtp.messagingengine.com ([64.147.123.20]:56905 "EHLO wout4-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345674AbiA1KRO (ORCPT ); Fri, 28 Jan 2022 05:17:14 -0500 Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id A5CA63202069 for ; Fri, 28 Jan 2022 05:17:13 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Fri, 28 Jan 2022 05:17:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm3; bh=+zodOWDJK8kYc9duP2VQbj6jPP/xgrAJ46esAP HZFj0=; b=Nx6nZ3M1TSRM+9m5jKLPTCXPDSiRQeFkRDJcQt/H74CL0g1BxtyZRx ij8zGqsMHMY75WOCT9Fc57dRYsKK7KCJ4R+5k2pEo9TWJlChqoQkT046BR5ChpN3 vpljDBAGxW4GJTD9aQ1OdYjbRImfxGP4UZeDsQQEM3/sbooVz5UDTXX+PS4uskzP +8LGtli4JIUqXjA8XmrpBjC7JTzZ0xErqR9X/AqsRo20oDmsGzxLnN8T87Mw+Obz xTyFEdVFFALz8MvSn8QwtZtZhflP6yMhXeTfQG6X7ruw7ayQp3KjYZqwZ+V3JtrY LxKg+Oz5jI0a/wk3SVeCaAtBuTXGcOuA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=+zodOWDJK8kYc9duP 2VQbj6jPP/xgrAJ46esAPHZFj0=; b=aproTLmV8wzzWqBRVw5xnHIoSSiDfTHZA zE19HgQ/5vKb7vm7ZnbbV3SkdeEiw9EOAD73dxHmmRr4owsREUB4vJoCnwCj7qX+ HnBW/4yBPxmdSqwb9GPqqoNvog3PN16h2LNAIfodwc8SDp8U8QnKFKB9QpM29bQn pLlRB4+WL1H/fQuyjHxancgq8dmhjXc7FX5W5jEuz9SdROPOHTTgO7I3GBsE8LRH T/4d6PZ0beCb4jLulLSI60khWVxqFVSZvPv1tVzMOnAdvYgkKpRG820JRBSa1Jns lRWPpDejgSvQG+WmoIfztsNU4BUF3mG52/5t3/g3h2eJ0QrkLCBHg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrfeehgddugecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtderre dttdejnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhk shdrihhmqeenucggtffrrghtthgvrhhnpeehgfejueevjeetudehgffffeffvdejfeejie dvkeffgfekuefgheevteeufeelkeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 28 Jan 2022 05:17:12 -0500 (EST) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 40975f70 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Fri, 28 Jan 2022 10:17:11 +0000 (UTC) Date: Fri, 28 Jan 2022 11:17:10 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 2/2] fetch: skip computing output width when not printing anything Message-ID: <5a3fd3232fd9e19e6f0054717a1f54c71bd8f272.1643364639.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When updating references via git-fetch(1), then by default we report to the user which references have been changed. This output is formatted in a nice table such that the different columns are aligned. Because the first column contains abbreviated object IDs we thus need to iterate over all refs which have changed and compute the minimum length for their respective abbreviated hashes. While this effort makes sense in most cases, it is wasteful when the user passes the `--quiet` flag: we don't print the summary, but still compute the length. Skip computing the summary width when the user asked for us to be quiet. This gives us a small speedup of nearly 10% when doing a dry-run mirror-fetch in a repository with thousands of references being updated: Benchmark 1: git fetch --prune --dry-run +refs/*:refs/* (HEAD~) Time (mean ± σ): 34.048 s ± 0.233 s [User: 30.739 s, System: 4.640 s] Range (min … max): 33.785 s … 34.296 s 5 runs Benchmark 2: git fetch --prune --dry-run +refs/*:refs/* (HEAD) Time (mean ± σ): 30.768 s ± 0.287 s [User: 27.534 s, System: 4.565 s] Range (min … max): 30.432 s … 31.181 s 5 runs Summary 'git fetch --prune --dry-run +refs/*:refs/* (HEAD)' ran 1.11 ± 0.01 times faster than 'git fetch --prune --dry-run +refs/*:refs/* (HEAD~)' Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 5f06b21f8e..ebbde5d56d 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1093,12 +1093,15 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, struct ref *rm; char *url; int want_status; - int summary_width = transport_summary_width(ref_map); + int summary_width = 0; rc = open_fetch_head(&fetch_head); if (rc) return -1; + if (verbosity >= 0) + summary_width = transport_summary_width(ref_map); + if (raw_url) url = transport_anonymize_url(raw_url); else @@ -1344,7 +1347,6 @@ static int prune_refs(struct refspec *rs, struct ref *ref_map, int url_len, i, result = 0; struct ref *ref, *stale_refs = get_stale_heads(rs, ref_map); char *url; - int summary_width = transport_summary_width(stale_refs); const char *dangling_msg = dry_run ? _(" (%s will become dangling)") : _(" (%s has become dangling)"); @@ -1373,6 +1375,8 @@ static int prune_refs(struct refspec *rs, struct ref *ref_map, } if (verbosity >= 0) { + int summary_width = transport_summary_width(stale_refs); + for (ref = stale_refs; ref; ref = ref->next) { struct strbuf sb = STRBUF_INIT; if (!shown_url) {