From patchwork Thu Feb 10 12:28:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12741868 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67D9AC433F5 for ; Thu, 10 Feb 2022 12:28:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237740AbiBJM2F (ORCPT ); Thu, 10 Feb 2022 07:28:05 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:45544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232819AbiBJM2F (ORCPT ); Thu, 10 Feb 2022 07:28:05 -0500 X-Greylist: delayed 262 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Thu, 10 Feb 2022 04:28:06 PST Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 382BFE0D for ; Thu, 10 Feb 2022 04:28:06 -0800 (PST) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 844215C009E; Thu, 10 Feb 2022 07:28:05 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Thu, 10 Feb 2022 07:28:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm3; bh=LF2Qr0+Ra0K2X2VH8dOKX5NWUhm9M2V6kCQeLQ SGALA=; b=GCpGlFDVe63MPxYO2P36Nc8l5Jp/0b5gI7aZ0b46c0lNyXzY//laCc dkIciT9t+CgForoGODib+Ovfwcpk97cCX+oj45ZDUBsY6M0IrUaRr9u9sjGcjNZx s0IWqqL+Mr22+kRzeZyerZqCr4KabanLFkhblQ61X3eDjv1tHp6qxlAM0JaVXWwK 5g9HoIu1vn7I+EaYRlpocazvmFmdlfdDYZ1l7n8XjpAfp3Z72C+JtdAK3m/+i39o wm8SRv+lKnbh00dmSkGYqtow28vPj0ETnmVJyvTfzaJ5QYTOEw80JSLkoGI43wiL umJcYQ7XPYIrF6f+pZchpfewWNmBHkFg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=LF2Qr0+Ra0K2X2VH8 dOKX5NWUhm9M2V6kCQeLQSGALA=; b=i3divHQXj+/rmSryi5H2Yt0UBgno2uDOu Uv7wybBcR6d3eSKRD2P2ihEG2UEZ/aYbR9TyeUZcRy/mCXQdpgJo+dbGRawZcooc +VDD8DppAeeqXmCuXSDCS5tvm97mgvT4D6WP2x5ZxZKU/zlKhj7/m364s6kCTS4H b1wfGSoP0UUXJXh2bo4y94b0cjmaMlqgWOL1ggMjETKsI9OumvEAletTw3ulxeUx OwSV1vpRQmTuHJtE1L5frDgYFzY8YFbMO5JP+xv5em6i1AFtp7FBhlsYABJHRBAk h/q0+EwsdFuEsH/C7tbt/QhyIruQv9mu2SB+e6MDevAaHiUToNYlQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddriedugdeflecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvuffkfhggtggujgesghdtreertddtjeenucfhrhhomheprfgrthhrihgt khcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvghrnh ephefgjeeuveejteduhefgffefffdvjeefjeeivdekfffgkeeugfehveetueefleeknecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhssehpkh hsrdhimh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Feb 2022 07:28:04 -0500 (EST) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 0e5abaf1 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 10 Feb 2022 12:28:03 +0000 (UTC) Date: Thu, 10 Feb 2022 13:28:01 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , Derrick Stolee , Christian Couder Subject: [PATCH v3 0/2] fetch: speed up mirror-fetches with many refs Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Hi, this is the third version of my patch series which aimn to speed up mirror-fetches in repos with huge amounts of refs. Again, the only change compared to v2 is a change in commit messages: Chris has rightly pointed out that the benchmarks were a bit confusing, so I've updated them to hopefully be less so. Thanks for your feedback! Patrick Patrick Steinhardt (2): fetch-pack: use commit-graph when computing cutoff fetch: skip computing output width when not printing anything builtin/fetch.c | 8 ++++++-- fetch-pack.c | 28 ++++++++++++++++------------ 2 files changed, 22 insertions(+), 14 deletions(-) Range-diff against v2: 1: 6fac914f0f ! 1: 077d06764c fetch-pack: use commit-graph when computing cutoff @@ Commit message the commit-graph first, which is a lot more efficient. Benchmarks in a repository with about 2,1 million refs and an up-to-date - commit-graph show a 20% speedup when mirror-fetching: + commit-graph show an almost 20% speedup when mirror-fetching: - Benchmark 1: git fetch --atomic +refs/*:refs/* (v2.35.0) - Time (mean ± σ): 75.264 s ± 1.115 s [User: 68.199 s, System: 10.094 s] - Range (min … max): 74.145 s … 76.862 s 5 runs + Benchmark 1: git fetch +refs/*:refs/* (v2.35.0) + Time (mean ± σ): 115.587 s ± 2.009 s [User: 109.874 s, System: 11.305 s] + Range (min … max): 113.584 s … 118.820 s 5 runs - Benchmark 2: git fetch --atomic +refs/*:refs/* (HEAD) - Time (mean ± σ): 62.350 s ± 0.854 s [User: 55.412 s, System: 9.976 s] - Range (min … max): 61.224 s … 63.216 s 5 runs + Benchmark 2: git fetch +refs/*:refs/* (HEAD) + Time (mean ± σ): 96.859 s ± 0.624 s [User: 91.948 s, System: 10.980 s] + Range (min … max): 96.180 s … 97.875 s 5 runs Summary - 'git fetch --atomic +refs/*:refs/* (HEAD)' ran - 1.21 ± 0.02 times faster than 'git fetch --atomic +refs/*:refs/* (v2.35.0)' + 'git fetch +refs/*:refs/* (HEAD)' ran + 1.19 ± 0.02 times faster than 'git fetch +refs/*:refs/* (v2.35.0)' Signed-off-by: Patrick Steinhardt 2: 4b9bbcf795 ! 2: ef1fd07be5 fetch: skip computing output width when not printing anything @@ Commit message don't print the summary, but still compute the length. Skip computing the summary width when the user asked for us to be quiet. - This gives us a small speedup of nearly 10% when doing a dry-run - mirror-fetch in a repository with thousands of references being updated: + This gives us a speedup of nearly 10% when doing a mirror-fetch in a + repository with thousands of references being updated: - Benchmark 1: git fetch --prune --dry-run +refs/*:refs/* (HEAD~) - Time (mean ± σ): 34.048 s ± 0.233 s [User: 30.739 s, System: 4.640 s] - Range (min … max): 33.785 s … 34.296 s 5 runs + Benchmark 1: git fetch --quiet +refs/*:refs/* (HEAD~) + Time (mean ± σ): 96.078 s ± 0.508 s [User: 91.378 s, System: 10.870 s] + Range (min … max): 95.449 s … 96.760 s 5 runs - Benchmark 2: git fetch --prune --dry-run +refs/*:refs/* (HEAD) - Time (mean ± σ): 30.768 s ± 0.287 s [User: 27.534 s, System: 4.565 s] - Range (min … max): 30.432 s … 31.181 s 5 runs + Benchmark 2: git fetch --quiet +refs/*:refs/* (HEAD) + Time (mean ± σ): 88.214 s ± 0.192 s [User: 83.274 s, System: 10.978 s] + Range (min … max): 87.998 s … 88.446 s 5 runs Summary - 'git fetch --prune --dry-run +refs/*:refs/* (HEAD)' ran - 1.11 ± 0.01 times faster than 'git fetch --prune --dry-run +refs/*:refs/* (HEAD~)' + 'git fetch --quiet +refs/*:refs/* (HEAD)' ran + 1.09 ± 0.01 times faster than 'git fetch --quiet +refs/*:refs/* (HEAD~)' Signed-off-by: Patrick Steinhardt