From patchwork Thu Feb 10 12:28:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12741869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDFACC433F5 for ; Thu, 10 Feb 2022 12:28:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241475AbiBJM2M (ORCPT ); Thu, 10 Feb 2022 07:28:12 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:45908 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240930AbiBJM2L (ORCPT ); Thu, 10 Feb 2022 07:28:11 -0500 Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69E3AE36 for ; Thu, 10 Feb 2022 04:28:12 -0800 (PST) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id C6A3A5C01B9; Thu, 10 Feb 2022 07:28:11 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Thu, 10 Feb 2022 07:28:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm3; bh=8XhV+N4YcpiPPi42QnP52B/EntfnSiaJA2NtGu J6kvY=; b=oseMiXJPFusm4fF+sOeweLpBWPg2CouuJ2QR9UQE2VZSZ385BtWZpx CrVNDyHVcAb19qMGzbIATse5ix150cEixf8eDQaVxck6p2/GTSpHfEMUBXT2IZuY 4a+3pD6BJaimnJZQvnmBWd3NEMUteS6s5dujxpsR/zl2rhxpHXQ3T39HJneqVCRQ zbXi1WMvOXee9rcNSwYvo8p4WYRQPPZxr89xDFQP3yL5EfBMEUHPA8HChsgjirZe NMZ8BklYheULldIaYXzYxiQV3nlSo9iKWLQFpKVUTk6uZtZfaKHLU7RoNrD7eenu W4mYRTh8TV4YCLXpXdx4807MeL0yrg8Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=8XhV+N4YcpiPPi42Q nP52B/EntfnSiaJA2NtGuJ6kvY=; b=KLe+jr3N99KWsNvWwc2BCwAbXPJztVOs+ v7mVB7av2tocK5Rnu27oC/dYAuMTtxOsJOzlVI4YyJCwUaszecet4GBJVI3bdwUT 68GHNNpfKRz5zuZxnWSGrqzuTcHktgnYSo7Pf6B9j1Ywd2qwWx8jJVo247zqvOfD JJ9vBarz3zrM8wdDejhNcgV5vqqJHlqZfMjgSF8J1MArIonGB1RJdoFuslJKblds dpWXoIP2UmbjMbIG1kGZP/nWS2q6GaAsSj4/6RxPpROg2nLuo9Idi5+KOhcvY2Hi goHfNcsiMARJbbfOoWkxEn9j6sX1gwZlFOUX0psXapm/cCt9o2bYw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddriedugdeflecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvuffkfhggtggujgesghdtreertddtjeenucfhrhhomheprfgrthhrihgt khcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvghrnh ephefgjeeuveejteduhefgffefffdvjeefjeeivdekfffgkeeugfehveetueefleeknecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhssehpkh hsrdhimh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Feb 2022 07:28:10 -0500 (EST) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id d9219af4 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 10 Feb 2022 12:28:10 +0000 (UTC) Date: Thu, 10 Feb 2022 13:28:09 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Derrick Stolee , Christian Couder Subject: [PATCH v3 1/2] fetch-pack: use commit-graph when computing cutoff Message-ID: <077d06764cafde470ea0589072d387d7737bb91e.1644495978.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org During packfile negotiation we iterate over all refs announced by the remote side to check whether their IDs refer to commits already known to us. If a commit is known to us already, then its date is a potential cutoff point for commits we have in common with the remote side. There is potentially a lot of commits announced by the remote depending on how many refs there are in the remote repository, and for every one of them we need to search for it in our object database and, if found, parse the corresponding object to find out whether it is a candidate for the cutoff date. This can be sped up by trying to look up commits via the commit-graph first, which is a lot more efficient. Benchmarks in a repository with about 2,1 million refs and an up-to-date commit-graph show an almost 20% speedup when mirror-fetching: Benchmark 1: git fetch +refs/*:refs/* (v2.35.0) Time (mean ± σ): 115.587 s ± 2.009 s [User: 109.874 s, System: 11.305 s] Range (min … max): 113.584 s … 118.820 s 5 runs Benchmark 2: git fetch +refs/*:refs/* (HEAD) Time (mean ± σ): 96.859 s ± 0.624 s [User: 91.948 s, System: 10.980 s] Range (min … max): 96.180 s … 97.875 s 5 runs Summary 'git fetch +refs/*:refs/* (HEAD)' ran 1.19 ± 0.02 times faster than 'git fetch +refs/*:refs/* (v2.35.0)' Signed-off-by: Patrick Steinhardt --- fetch-pack.c | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index dd6ec449f2..c5967e228e 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -696,26 +696,30 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator, trace2_region_enter("fetch-pack", "parse_remote_refs_and_find_cutoff", NULL); for (ref = *refs; ref; ref = ref->next) { - struct object *o; + struct commit *commit; - if (!has_object_file_with_flags(&ref->old_oid, + commit = lookup_commit_in_graph(the_repository, &ref->old_oid); + if (!commit) { + struct object *o; + + if (!has_object_file_with_flags(&ref->old_oid, OBJECT_INFO_QUICK | - OBJECT_INFO_SKIP_FETCH_OBJECT)) - continue; - o = parse_object(the_repository, &ref->old_oid); - if (!o) - continue; + OBJECT_INFO_SKIP_FETCH_OBJECT)) + continue; + o = parse_object(the_repository, &ref->old_oid); + if (!o || o->type != OBJ_COMMIT) + continue; + + commit = (struct commit *)o; + } /* * We already have it -- which may mean that we were * in sync with the other side at some time after * that (it is OK if we guess wrong here). */ - if (o->type == OBJ_COMMIT) { - struct commit *commit = (struct commit *)o; - if (!cutoff || cutoff < commit->date) - cutoff = commit->date; - } + if (!cutoff || cutoff < commit->date) + cutoff = commit->date; } trace2_region_leave("fetch-pack", "parse_remote_refs_and_find_cutoff", NULL); From patchwork Thu Feb 10 12:28:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12741870 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 485ECC433F5 for ; Thu, 10 Feb 2022 12:28:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241537AbiBJM2U (ORCPT ); Thu, 10 Feb 2022 07:28:20 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:46620 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241541AbiBJM2T (ORCPT ); Thu, 10 Feb 2022 07:28:19 -0500 Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4469E34 for ; Thu, 10 Feb 2022 04:28:19 -0800 (PST) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 14F805C009E; Thu, 10 Feb 2022 07:28:19 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Thu, 10 Feb 2022 07:28:19 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm3; bh=oxykHxvSU6aItsfh9KyImFXEsg639j+oG6cbc6 GCuys=; b=Si0PKL2crcNq3PfyQpDA1DJccLZ+DSCCjQa57DYyuyBc/2s7Vf22RR NjJcAU3ihAQdoi6wYnz2Imv56fefg3AkGfmmxbgm77MrPEDvlvcDANxi12zrtWDE seCc0eWRO9U5J6XIQpRPwas7o/rIVAUPInCEkb+kbaudRdjegAiXPH/5BkHgp38o iW3rGGt9CnQGFNeLeF6MAB0FQutEWCmvZ8E/96FjxCg1I8G7sYLaKlNJ/bpUAUnc 85cEe9rnFu+mbXO3/lSGUupWZ+c9Nj0Gv4aZpRAintwkEuuwNv4vAVHbxxjhX2Xf F0kkZazkjM4y5T7WDH7atBgsOirLPCsw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=oxykHxvSU6aItsfh9 KyImFXEsg639j+oG6cbc6GCuys=; b=kkHNVSKtVbUhMBVRlOdnRR0P56RrZ88CU UjRrF1F8nIHXw4jCGeKxerofju5B4H3wMcyly7RYeQ39iBDcv5MVSYzuX3jYse2e xNGR0YIxYw+zKl/kxuWmEA8yCCvBSVpuJhE/DlC4vyF5Iezm6u0+FvYIBOiZJsne wD+ydwgSdFpj2s/9RWfKwSIow82gODSBVmyQ++ovgp+5HPxA0PJ0eAhxOcW0l7O8 tYeQa7ZvF9JGL6+TvUqsEk+/pAjT4OrwN+lRJ41y5OcupNUAzgEuARsbywXax53W LBHX8w8aA8ctYvOrTF9AiTA6683KvvskVDVENMTYYo0hfwjEcGfow== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddriedugdeflecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvuffkfhggtggujgesghdtreertddtjeenucfhrhhomheprfgrthhrihgt khcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvghrnh ephefgjeeuveejteduhefgffefffdvjeefjeeivdekfffgkeeugfehveetueefleeknecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhssehpkh hsrdhimh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Feb 2022 07:28:17 -0500 (EST) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 2ca441bc (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 10 Feb 2022 12:28:17 +0000 (UTC) Date: Thu, 10 Feb 2022 13:28:16 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Derrick Stolee , Christian Couder Subject: [PATCH v3 2/2] fetch: skip computing output width when not printing anything Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When updating references via git-fetch(1), then by default we report to the user which references have been changed. This output is formatted in a nice table such that the different columns are aligned. Because the first column contains abbreviated object IDs we thus need to iterate over all refs which have changed and compute the minimum length for their respective abbreviated hashes. While this effort makes sense in most cases, it is wasteful when the user passes the `--quiet` flag: we don't print the summary, but still compute the length. Skip computing the summary width when the user asked for us to be quiet. This gives us a speedup of nearly 10% when doing a mirror-fetch in a repository with thousands of references being updated: Benchmark 1: git fetch --quiet +refs/*:refs/* (HEAD~) Time (mean ± σ): 96.078 s ± 0.508 s [User: 91.378 s, System: 10.870 s] Range (min … max): 95.449 s … 96.760 s 5 runs Benchmark 2: git fetch --quiet +refs/*:refs/* (HEAD) Time (mean ± σ): 88.214 s ± 0.192 s [User: 83.274 s, System: 10.978 s] Range (min … max): 87.998 s … 88.446 s 5 runs Summary 'git fetch --quiet +refs/*:refs/* (HEAD)' ran 1.09 ± 0.01 times faster than 'git fetch --quiet +refs/*:refs/* (HEAD~)' Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 5b3b18a72f..7ef305c66d 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1094,12 +1094,15 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, struct ref *rm; char *url; int want_status; - int summary_width = transport_summary_width(ref_map); + int summary_width = 0; rc = open_fetch_head(&fetch_head); if (rc) return -1; + if (verbosity >= 0) + summary_width = transport_summary_width(ref_map); + if (raw_url) url = transport_anonymize_url(raw_url); else @@ -1345,7 +1348,6 @@ static int prune_refs(struct refspec *rs, struct ref *ref_map, int url_len, i, result = 0; struct ref *ref, *stale_refs = get_stale_heads(rs, ref_map); char *url; - int summary_width = transport_summary_width(stale_refs); const char *dangling_msg = dry_run ? _(" (%s will become dangling)") : _(" (%s has become dangling)"); @@ -1374,6 +1376,8 @@ static int prune_refs(struct refspec *rs, struct ref *ref_map, } if (verbosity >= 0) { + int summary_width = transport_summary_width(stale_refs); + for (ref = stale_refs; ref; ref = ref->next) { struct strbuf sb = STRBUF_INIT; if (!shown_url) {