From patchwork Wed Sep 1 13:09:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12469345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 811FAC4320E for ; Wed, 1 Sep 2021 13:09:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6095A6108E for ; Wed, 1 Sep 2021 13:09:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344297AbhIANKn (ORCPT ); Wed, 1 Sep 2021 09:10:43 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:58581 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244182AbhIANKl (ORCPT ); Wed, 1 Sep 2021 09:10:41 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id B43AB5C01ED; Wed, 1 Sep 2021 09:09:44 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Wed, 01 Sep 2021 09:09:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=S+iORiJSq+O9mauh7nhC1KPkPx4 ePloJrZEVJJQhN9w=; b=bTriAwbG70Hk3cQJAe2nIh0OShDtBli2k7ngfOKlHPV UYpVmEJeDpIpuUrWO6qyjoSPMVQo4ZnyESiaBfgLnB1K3H3NLCkNsZTkHkoNUrns xvjT2h82k8K7EZrgH1QILJVaeJpFstGYMg8VMvbXkQL68Exuvg0smHwbnZmBMb5y ivrXFNtnc1mCJdoDSE9bTfNBCp3YIk05PHo5AKDdbj5q4kjvf2lZwIhDCm6OWIUz OYD3p1MYYFncIaTRZ8QthdSYtAjuWH5Pq5BGn3ICt86C8120Tbso2CCt49j1O+eS OmKGcnZ10a+reoyzjIwXe/mTDkdTxaov2LJV0bjuDyQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=S+iORi JSq+O9mauh7nhC1KPkPx4ePloJrZEVJJQhN9w=; b=HeJ0OozaI8bJcauxiESm4N hhYigzNKwhzvbA/5EyzzWtlupbgBaIH2j4nAprUZpvtPyZ4RNxa0rvU+VB5ypNSf qGBMUaXuMs4wkJfACDG4ylDV9N0J3z2vkdA4vCgFEJhUlt4Wibc45v6ptEYrbpQt +O8DaPmHuskyCV7ZdDMhU6yS9K500Oy51b0rujYVosGET1jLZ1qNw16Datp+Duee xwhBuaZ4vv4c5ajUiVd6rdKNquDV4dSIg5nPQo+BsNeA3f2J2mNKpR6kaSLWaNLA 0XKi8uNTxtErU3JZDPL9YUvl4EapdO2W96TCLR64MJhObyc70oIjRHBuap8lBydg == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddvfedgiedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttdejnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgfejueevjeetudehgffffeffvdejfeejiedvkeffgfekuefgheevteeufeelkeen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Sep 2021 09:09:43 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id baa47f15 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Sep 2021 13:09:42 +0000 (UTC) Date: Wed, 1 Sep 2021 15:09:41 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v3 1/7] fetch: speed up lookup of want refs via commit-graph Message-ID: <8214f04971c4ea628f23d2e9ea143d87570c29d2.1630501732.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When updating our local refs based on the refs fetched from the remote, we need to iterate through all requested refs and load their respective commits such that we can determine whether they need to be appended to FETCH_HEAD or not. In cases where we're fetching from a remote with exceedingly many refs, resolving these refs can be quite expensive given that we repeatedly need to unpack object headers for each of the referenced objects. Speed this up by opportunistically trying to resolve object IDs via the commit graph. We only do so for any refs which are not in "refs/tags": more likely than not, these are going to be a commit anyway, and this lets us avoid having to unpack object headers completely in case the object is a commit that is part of the commit-graph. This significantly speeds up mirror-fetches in a real-world repository with 2.3M refs: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 56.482 s ± 0.384 s [User: 53.340 s, System: 5.365 s] Range (min … max): 56.050 s … 57.045 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 33.727 s ± 0.170 s [User: 30.252 s, System: 5.194 s] Range (min … max): 33.452 s … 33.871 s 5 runs Summary 'HEAD: git-fetch' ran 1.67 ± 0.01 times faster than 'HEAD~: git-fetch' Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 25740c13df..bd7c0da232 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1074,7 +1074,6 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, int connectivity_checked, struct ref *ref_map) { struct fetch_head fetch_head; - struct commit *commit; int url_len, i, rc = 0; struct strbuf note = STRBUF_INIT, err = STRBUF_INIT; struct ref_transaction *transaction = NULL; @@ -1122,6 +1121,7 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, want_status <= FETCH_HEAD_IGNORE; want_status++) { for (rm = ref_map; rm; rm = rm->next) { + struct commit *commit = NULL; struct ref *ref = NULL; if (rm->status == REF_STATUS_REJECT_SHALLOW) { @@ -1131,11 +1131,23 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, continue; } - commit = lookup_commit_reference_gently(the_repository, - &rm->old_oid, - 1); - if (!commit) - rm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE; + /* + * References in "refs/tags/" are often going to point + * to annotated tags, which are not part of the + * commit-graph. We thus only try to look up refs in + * the graph which are not in that namespace to not + * regress performance in repositories with many + * annotated tags. + */ + if (!starts_with(rm->name, "refs/tags/")) + commit = lookup_commit_in_graph(the_repository, &rm->old_oid); + if (!commit) { + commit = lookup_commit_reference_gently(the_repository, + &rm->old_oid, + 1); + if (!commit) + rm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE; + } if (rm->fetch_head_status != want_status) continue; From patchwork Wed Sep 1 13:09:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12469347 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1677BC432BE for ; Wed, 1 Sep 2021 13:09:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F335D60FC0 for ; Wed, 1 Sep 2021 13:09:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344273AbhIANKu (ORCPT ); Wed, 1 Sep 2021 09:10:50 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:51381 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244182AbhIANKq (ORCPT ); Wed, 1 Sep 2021 09:10:46 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 7DED45C01F9; Wed, 1 Sep 2021 09:09:49 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 01 Sep 2021 09:09:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=Z3o9N9TRMo6/gqEm2o0lB/9U1F8 Sig62K77mujfeFnc=; b=ZI/wTwmU5PZZkQc+39cNJxL3clT1PFXSCTwAFcyohfo AHHVygSjC1woAH9r9yFmTq2e30abPHB1YKL3P5wn5fv5KHjwkzAo0uBVsJu6hkrE 0DJ/2oTJYpyoVJY5Dbg6DeWMmiCrzL+w/3YLQxCA18TfdieXU9VGqzmBIFmt1enf F8BsH3rxFFwSmYx0uLny6+o/spwGTH8uKLSxBue5n0Edzyauq2uiser+z0aqShLd 4+qyMNB2n8OBzElvVz6US5lK3OTAlBTjs689RALHNCP5v2iDaRbbzEp0V6YcEqCR bgL/LDyNp+SDs7g1A7Cs0QnMfGSzt2i36x7ILg7FeBA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=Z3o9N9 TRMo6/gqEm2o0lB/9U1F8Sig62K77mujfeFnc=; b=wR+C2naQuI17PoZG35Km7v Za44EdOdUYz3EwcY3ZZdB3I5PUauIB+n0wShFXCAGL7tMjyTDvP7AoLOUOT/Zm39 6nfEy0Nq8Isdg20mC27PyG+r8YXm6OHFP52MsN+eccJQksgFrZTSq9OzKwVzSt9Z 7EkYSOWwBOZncyYmnLPTxQ2LCUQ3uuHPatS50kdx4EfEILQdy1sNhf38erSvGxIy uNtod7qsD3QyE30+R0odsIc8sRao9Iv9wzQJjoY2GhAsGS6U9Tt159WxSmpYpvF6 Kyl7ugNfNwcsRHyfFgJ0mMcQ39t8Phz+0jStqSbEp3eewY/z1/X+qAp0/mze3zeg == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddvfedgieduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttdejnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgfejueevjeetudehgffffeffvdejfeejiedvkeffgfekuefgheevteeufeelkeen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Sep 2021 09:09:47 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 50c62e97 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Sep 2021 13:09:47 +0000 (UTC) Date: Wed, 1 Sep 2021 15:09:45 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v3 2/7] fetch: avoid unpacking headers in object existence check Message-ID: <991a27cb8286195e8ad928807b96ed17ebe1e5a5.1630501732.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When updating local refs after the fetch has transferred all objects, we do an object existence test as a safety guard to avoid updating a ref to an object which we don't have. We do so via `oid_object_info()`: if it returns an error, then we know the object does not exist. One side effect of `oid_object_info()` is that it parses the object's type, and to do so it must unpack the object header. This is completely pointless: we don't care for the type, but only want to assert that the object exists. Refactor the code to use `repo_has_object_file()`, which both makes the code's intent clearer and is also faster because it does not unpack object headers. In a real-world repo with 2.3M refs, this results in a small speedup when doing a mirror-fetch: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 33.686 s ± 0.176 s [User: 30.119 s, System: 5.262 s] Range (min … max): 33.512 s … 33.944 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 31.247 s ± 0.195 s [User: 28.135 s, System: 5.066 s] Range (min … max): 30.948 s … 31.472 s 5 runs Summary 'HEAD: git-fetch' ran 1.08 ± 0.01 times faster than 'HEAD~: git-fetch' Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index bd7c0da232..0b18c47732 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -846,13 +846,11 @@ static int update_local_ref(struct ref *ref, int summary_width) { struct commit *current = NULL, *updated; - enum object_type type; struct branch *current_branch = branch_get(NULL); const char *pretty_ref = prettify_refname(ref->name); int fast_forward = 0; - type = oid_object_info(the_repository, &ref->new_oid, NULL); - if (type < 0) + if (!repo_has_object_file(the_repository, &ref->new_oid)) die(_("object %s not found"), oid_to_hex(&ref->new_oid)); if (oideq(&ref->old_oid, &ref->new_oid)) { From patchwork Wed Sep 1 13:09:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12469349 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 466CEC4320E for ; Wed, 1 Sep 2021 13:09:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 304B861008 for ; Wed, 1 Sep 2021 13:09:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344352AbhIANKv (ORCPT ); Wed, 1 Sep 2021 09:10:51 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:54997 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344135AbhIANKt (ORCPT ); Wed, 1 Sep 2021 09:10:49 -0400 Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id B6AAB5C01DB; Wed, 1 Sep 2021 09:09:52 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 01 Sep 2021 09:09:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=sMga0CT9q7GLN5d1pTUK0ZuNacL OrPM0ishLT683fWM=; b=pbzKIhK0sXmI8P6FH5DNMvzcNLwDe2SZcltzhwf0Y6F iHkV4jdhTLbs7su1JywazBb1Z0IwA1Rom9b4ISUyg0PTaDjak9MbkoFj5dctvcTD /tgY9V+LAx+wkdyxBjdp9pH99qJj5Md6+aJJ86GLD9b2iXm0Dsd09mBevX9qAWuZ 9p5ixQcnq2QKIHB43JUmZZk2py6Rybw+qforiZ2uoDp8KXY3OND+KBrVY6bbpTXA 2te36woVopFEDOrzqgmkkvgw1p/NCG+DMj4cZEOiQkz7+7Pz2KxQNEnLPpfc4lwq 1i2pFaxY4WruJPWR3pE0D1ZF6nsqqQc5x0lQCmcXY/g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=sMga0C T9q7GLN5d1pTUK0ZuNacLOrPM0ishLT683fWM=; b=MGyRvgi1znzN/A6N8UKQPj YPrG7NICLIY4Xq2nsyRPhrAOrtANM/dBmICSNdyF3DH7hxNgrayZ3cGqJ8gSojd3 dQ53pHaTzhj9elHix3XcErod15KgN1hbU3HNy0D+IEcTEg+82zC1hVk19XHuHHx+ gXG/XuGV2vPUHJ44p1uqKQNsvSjCDIaD63mly4Cr2AuVNzBNQsi9iEZ2aewkfG/X Dxes2xkNkxngyVDenmmTi4jt440LNLu2XhnibwLw5ucLsI1lOLZKkr09O5s4guO8 dnkwb7ywUPNfL2j52qBTvIiv/WDkxtN/yJicdZ7aoRdpLTdmQ10B25gQpIuZx0bw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddvfedgieduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttdejnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgfejueevjeetudehgffffeffvdejfeejiedvkeffgfekuefgheevteeufeelkeen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Sep 2021 09:09:51 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id b8d3e360 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Sep 2021 13:09:51 +0000 (UTC) Date: Wed, 1 Sep 2021 15:09:50 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v3 3/7] connected: refactor iterator to return next object ID directly Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The object ID iterator used by the connectivity checks returns the next object ID via an out-parameter and then uses a return code to indicate whether an item was found. This is a bit roundabout: instead of a separate error code, we can just return the next object ID directly and use `NULL` pointers as indicator that the iterator got no items left. Furthermore, this avoids a copy of the object ID. Refactor the iterator and all its implementations to return object IDs directly. This brings a tiny performance improvement when doing a mirror-fetch of a repository with about 2.3M refs: Benchmark #1: 328dc58b49919c43897240f2eabfa30be2ce32a4~: git-fetch Time (mean ± σ): 30.110 s ± 0.148 s [User: 27.161 s, System: 5.075 s] Range (min … max): 29.934 s … 30.406 s 10 runs Benchmark #2: 328dc58b49919c43897240f2eabfa30be2ce32a4: git-fetch Time (mean ± σ): 29.899 s ± 0.109 s [User: 26.916 s, System: 5.104 s] Range (min … max): 29.696 s … 29.996 s 10 runs Summary '328dc58b49919c43897240f2eabfa30be2ce32a4: git-fetch' ran 1.01 ± 0.01 times faster than '328dc58b49919c43897240f2eabfa30be2ce32a4~: git-fetch' While this 1% speedup could be labelled as statistically insignificant, the speedup is consistent on my machine. Furthermore, this is an end to end test, so it is expected that the improvement in the connectivity check itself is more significant. Signed-off-by: Patrick Steinhardt --- builtin/clone.c | 8 +++----- builtin/fetch.c | 7 +++---- builtin/receive-pack.c | 17 +++++++---------- connected.c | 15 ++++++++------- connected.h | 2 +- fetch-pack.c | 7 +++---- 6 files changed, 25 insertions(+), 31 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 66fe66679c..4a1056fcc2 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -657,7 +657,7 @@ static void write_followtags(const struct ref *refs, const char *msg) } } -static int iterate_ref_map(void *cb_data, struct object_id *oid) +static const struct object_id *iterate_ref_map(void *cb_data) { struct ref **rm = cb_data; struct ref *ref = *rm; @@ -668,13 +668,11 @@ static int iterate_ref_map(void *cb_data, struct object_id *oid) */ while (ref && !ref->peer_ref) ref = ref->next; - /* Returning -1 notes "end of list" to the caller. */ if (!ref) - return -1; + return NULL; - oidcpy(oid, &ref->old_oid); *rm = ref->next; - return 0; + return &ref->old_oid; } static void update_remote_refs(const struct ref *refs, diff --git a/builtin/fetch.c b/builtin/fetch.c index 0b18c47732..f67fe543ad 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -962,7 +962,7 @@ static int update_local_ref(struct ref *ref, } } -static int iterate_ref_map(void *cb_data, struct object_id *oid) +static const struct object_id *iterate_ref_map(void *cb_data) { struct ref **rm = cb_data; struct ref *ref = *rm; @@ -970,10 +970,9 @@ static int iterate_ref_map(void *cb_data, struct object_id *oid) while (ref && ref->status == REF_STATUS_REJECT_SHALLOW) ref = ref->next; if (!ref) - return -1; /* end of the list */ + return NULL; *rm = ref->next; - oidcpy(oid, &ref->old_oid); - return 0; + return &ref->old_oid; } struct fetch_head { diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index 2d1f97e1ca..041e915454 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -1306,7 +1306,7 @@ static void refuse_unconfigured_deny_delete_current(void) rp_error("%s", _(refuse_unconfigured_deny_delete_current_msg)); } -static int command_singleton_iterator(void *cb_data, struct object_id *oid); +static const struct object_id *command_singleton_iterator(void *cb_data); static int update_shallow_ref(struct command *cmd, struct shallow_info *si) { struct shallow_lock shallow_lock = SHALLOW_LOCK_INIT; @@ -1731,16 +1731,15 @@ static void check_aliased_updates(struct command *commands) string_list_clear(&ref_list, 0); } -static int command_singleton_iterator(void *cb_data, struct object_id *oid) +static const struct object_id *command_singleton_iterator(void *cb_data) { struct command **cmd_list = cb_data; struct command *cmd = *cmd_list; if (!cmd || is_null_oid(&cmd->new_oid)) - return -1; /* end of list */ + return NULL; *cmd_list = NULL; /* this returns only one */ - oidcpy(oid, &cmd->new_oid); - return 0; + return &cmd->new_oid; } static void set_connectivity_errors(struct command *commands, @@ -1770,7 +1769,7 @@ struct iterate_data { struct shallow_info *si; }; -static int iterate_receive_command_list(void *cb_data, struct object_id *oid) +static const struct object_id *iterate_receive_command_list(void *cb_data) { struct iterate_data *data = cb_data; struct command **cmd_list = &data->cmds; @@ -1781,13 +1780,11 @@ static int iterate_receive_command_list(void *cb_data, struct object_id *oid) /* to be checked in update_shallow_ref() */ continue; if (!is_null_oid(&cmd->new_oid) && !cmd->skip_update) { - oidcpy(oid, &cmd->new_oid); *cmd_list = cmd->next; - return 0; + return &cmd->new_oid; } } - *cmd_list = NULL; - return -1; /* end of list */ + return NULL; } static void reject_updates_to_hidden(struct command *commands) diff --git a/connected.c b/connected.c index b5f9523a5f..cf68e37a97 100644 --- a/connected.c +++ b/connected.c @@ -24,7 +24,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, struct child_process rev_list = CHILD_PROCESS_INIT; FILE *rev_list_in; struct check_connected_options defaults = CHECK_CONNECTED_INIT; - struct object_id oid; + const struct object_id *oid; int err = 0; struct packed_git *new_pack = NULL; struct transport *transport; @@ -34,7 +34,8 @@ int check_connected(oid_iterate_fn fn, void *cb_data, opt = &defaults; transport = opt->transport; - if (fn(cb_data, &oid)) { + oid = fn(cb_data); + if (!oid) { if (opt->err_fd) close(opt->err_fd); return err; @@ -73,7 +74,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, for (p = get_all_packs(the_repository); p; p = p->next) { if (!p->pack_promisor) continue; - if (find_pack_entry_one(oid.hash, p)) + if (find_pack_entry_one(oid->hash, p)) goto promisor_pack_found; } /* @@ -83,7 +84,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, goto no_promisor_pack_found; promisor_pack_found: ; - } while (!fn(cb_data, &oid)); + } while ((oid = fn(cb_data)) != NULL); return 0; } @@ -133,12 +134,12 @@ int check_connected(oid_iterate_fn fn, void *cb_data, * are sure the ref is good and not sending it to * rev-list for verification. */ - if (new_pack && find_pack_entry_one(oid.hash, new_pack)) + if (new_pack && find_pack_entry_one(oid->hash, new_pack)) continue; - if (fprintf(rev_list_in, "%s\n", oid_to_hex(&oid)) < 0) + if (fprintf(rev_list_in, "%s\n", oid_to_hex(oid)) < 0) break; - } while (!fn(cb_data, &oid)); + } while ((oid = fn(cb_data)) != NULL); if (ferror(rev_list_in) || fflush(rev_list_in)) { if (errno != EPIPE && errno != EINVAL) diff --git a/connected.h b/connected.h index 8d5a6b3ad6..6e59c92aa3 100644 --- a/connected.h +++ b/connected.h @@ -9,7 +9,7 @@ struct transport; * When called after returning the name for the last object, return -1 * to signal EOF, otherwise return 0. */ -typedef int (*oid_iterate_fn)(void *, struct object_id *oid); +typedef const struct object_id *(*oid_iterate_fn)(void *); /* * Named-arguments struct for check_connected. All arguments are diff --git a/fetch-pack.c b/fetch-pack.c index b0c7be717c..7b0e69884d 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1906,16 +1906,15 @@ static void update_shallow(struct fetch_pack_args *args, oid_array_clear(&ref); } -static int iterate_ref_map(void *cb_data, struct object_id *oid) +static const struct object_id *iterate_ref_map(void *cb_data) { struct ref **rm = cb_data; struct ref *ref = *rm; if (!ref) - return -1; /* end of the list */ + return NULL; *rm = ref->next; - oidcpy(oid, &ref->old_oid); - return 0; + return &ref->old_oid; } struct ref *fetch_pack(struct fetch_pack_args *args, From patchwork Wed Sep 1 13:09:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12469351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00588C432BE for ; Wed, 1 Sep 2021 13:09:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D055F60FC0 for ; Wed, 1 Sep 2021 13:09:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344354AbhIANKy (ORCPT ); Wed, 1 Sep 2021 09:10:54 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:44391 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244182AbhIANKx (ORCPT ); Wed, 1 Sep 2021 09:10:53 -0400 Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id E6F555C01FE; Wed, 1 Sep 2021 09:09:56 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 01 Sep 2021 09:09:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=Rxc/f3vvs3P5vlcJsmH6D9aAmB3 28+ifM6da3gvaezo=; b=sCmzTKRsUARqgLvSzA5HnLI+B5Usx6wetWCQZ3KXWxy nElqiBePHu9P6Q1Dp4uy1JfNrLBjD/udLjLbGtsEgw2ved62+H1sRiuKom8OXPET 1DwKB/0HIlmcZntGvUzx1PpQmyvtHdHwYxY9u1eiS44lvZ33cyc31oWu+aA7SXjG I5EaFils9whwy487WSbVWdqICG7A0nj6WkhhNFvTTpwp2HLO5WMfjR4i5o82gjKK X9KK4C3PzORP0CiXfxJuEuM00X2Ump8HSBdzcIFqzY2LIANtztn6q9vkaq44VwTK pVfbXMmEf3uBatX9uE0UIZonM1rYkXFgqL1KYlUfvDQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=Rxc/f3 vvs3P5vlcJsmH6D9aAmB328+ifM6da3gvaezo=; b=fQtJQbQ6MT9jRNk/SiOliq An2aE9zJUIx12ODvh2BY0LgogY+9ShbbIu6Ysa1jvg0UpZ7LnIXBQRV91fj9Mb+k hsNxAkrwGks4Nl6pZG9s7htyT3Mta6JG2zuc46LzlUokkxRpt25GZfX8DTaLRbAX 6pCuvXdziYLdzS+oa+eDvOwGBWMhjhuSViqRcnjFw3HZTHV+EI1jizYO2g+3rt/I j58RdjEDlOLrYnT4Xgg7oO0lQzZ5mx6jZ80FiTwwXKENpKEPdw/r7B/OVl8yEzcK BQ7GnvFwbdkxQ+LwlGo8ykw2Ol1rIWaRDftqsHQHLX+3Ze6GviEg61jKmlwPkGXA == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddvfedgieduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttdejnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgfejueevjeetudehgffffeffvdejfeejiedvkeffgfekuefgheevteeufeelkeen ucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Sep 2021 09:09:55 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 02a0b292 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Sep 2021 13:09:55 +0000 (UTC) Date: Wed, 1 Sep 2021 15:09:54 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v3 4/7] fetch-pack: optimize loading of refs via commit graph Message-ID: <99d3316d48162a318b1ac0473eb9c837c60e3833.1630501732.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In order to negotiate a packfile, we need to dereference refs to see which commits we have in common with the remote. To do so, we first look up the object's type -- if it's a tag, we peel until we hit a non-tag object. If we hit a commit eventually, then we return that commit. In case the object ID points to a commit directly, we can avoid the initial lookup of the object type by opportunistically looking up the commit via the commit-graph, if available, which gives us a slight speed bump of about 2% in a huge repository with about 2.3M refs: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 31.634 s ± 0.258 s [User: 28.400 s, System: 5.090 s] Range (min … max): 31.280 s … 31.896 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 31.129 s ± 0.543 s [User: 27.976 s, System: 5.056 s] Range (min … max): 30.172 s … 31.479 s 5 runs Summary 'HEAD: git-fetch' ran 1.02 ± 0.02 times faster than 'HEAD~: git-fetch' In case this fails, we fall back to the old code which peels the objects to a commit. Signed-off-by: Patrick Steinhardt --- fetch-pack.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/fetch-pack.c b/fetch-pack.c index 7b0e69884d..da92a5e474 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -119,6 +119,11 @@ static struct commit *deref_without_lazy_fetch(const struct object_id *oid, { enum object_type type; struct object_info info = { .typep = &type }; + struct commit *commit; + + commit = lookup_commit_in_graph(the_repository, oid); + if (commit) + return commit; while (1) { if (oid_object_info_extended(the_repository, oid, &info, From patchwork Wed Sep 1 13:09:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12469353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90F0EC432BE for ; Wed, 1 Sep 2021 13:10:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 72CD360FED for ; Wed, 1 Sep 2021 13:10:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344418AbhIANLI (ORCPT ); Wed, 1 Sep 2021 09:11:08 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:54753 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344384AbhIANLA (ORCPT ); Wed, 1 Sep 2021 09:11:00 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 0661A5C0161; Wed, 1 Sep 2021 09:10:01 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 01 Sep 2021 09:10:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=7vhOzDwM4dU7PF+O6vo3x37sYB2 N4N1+ABlbxmsYUOc=; b=Aeao0gWJgnp/3XJcOP63x7rOZWHp5rV05BxzT+92Dsk JJ8l5ac8CQLuD/Sd9+yJfUpqvzA+qJP8OZ2H1kd5tFv0hMDigiY4nggSKaHK5k+R HxlFUOzCewYAqOq2sVVG/fQwYa8zOmxxg7Sv990ARu5kvKypFrrX17RYBrD23Mav Q8IbVK6cTBXvhG4WJHERU+ACIxbEVzmCcaJoWmGFxbVi1++ZvoOqqH2LZecgagCf ROV0AmQn/zw1poYQOyj+bQ3Emwe+EKhJ7dURUDADV9SbqjmCsHjSJ0soxDVkhUKi r2RZOkHjeeSQVnf6fpQGcCWZh3YwreMQSGxdQrd7J0g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=7vhOzD wM4dU7PF+O6vo3x37sYB2N4N1+ABlbxmsYUOc=; b=CC5Uh9lsJmgIO/4XzeUsKO cx5qbBstOAppdN/HaeGeYtk3f7xRjT+IeLeVu3xoBofmUznNvM0bIHcxKEarlN1u CrsinUn6I4WHttRxmkY8gSwZAM5L9VEl2Fb0XcoOMMXGxpGPLh6Zc4dlqHjlryZp Am0I2KuvU9hM2emkUAjZoOWrwq44NKObf0lOurXWrCfzKTcJPHRV1eMG7HKldgbM NWECLZVF/6gKCvJs+U/fNFBME3bGtLkXWp6tTISQhLxlMov5udc9xFQPOYz5VxUA c5NMuq4eP3SpXsuCy2nGkJj4GlD+tOLgl1S5mR0DesAQ33SPMoi6g9CufrWDFcCQ == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddvfedgieduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgefhtdefueffheekgfffudelffejtdfhvdejkedthfehvdelgfetgfdvtedthfen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Sep 2021 09:10:00 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 0530a8a2 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Sep 2021 13:09:59 +0000 (UTC) Date: Wed, 1 Sep 2021 15:09:58 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v3 5/7] fetch: refactor fetch refs to be more extendable Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Refactor `fetch_refs()` code to make it more extendable by explicitly handling error cases. The refactored code should behave the same. Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index f67fe543ad..bafeb3d44b 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1293,18 +1293,28 @@ static int check_exist_and_connected(struct ref *ref_map) static int fetch_refs(struct transport *transport, struct ref *ref_map) { - int ret = check_exist_and_connected(ref_map); + int ret; + + /* + * We don't need to perform a fetch in case we can already satisfy all + * refs. + */ + ret = check_exist_and_connected(ref_map); if (ret) { trace2_region_enter("fetch", "fetch_refs", the_repository); ret = transport_fetch_refs(transport, ref_map); trace2_region_leave("fetch", "fetch_refs", the_repository); + if (ret) + goto out; } - if (!ret) - /* - * Keep the new pack's ".keep" file around to allow the caller - * time to update refs to reference the new objects. - */ - return 0; + + /* + * Keep the new pack's ".keep" file around to allow the caller + * time to update refs to reference the new objects. + */ + return ret; + +out: transport_unlock_pack(transport); return ret; } From patchwork Wed Sep 1 13:10:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12469355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62232C4320E for ; Wed, 1 Sep 2021 13:10:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3CCD960FC0 for ; Wed, 1 Sep 2021 13:10:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244182AbhIANLK (ORCPT ); Wed, 1 Sep 2021 09:11:10 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:34169 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344395AbhIANLC (ORCPT ); Wed, 1 Sep 2021 09:11:02 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 35CF65C01DB; Wed, 1 Sep 2021 09:10:05 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 01 Sep 2021 09:10:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=5jvutvsZjig5pcuWKaPZlw5fQWV hdiBX/ijikES1MCU=; b=iJt+9mo7Pb25GD/qeamoS79hVdkoe037lit2zVCnkaZ 1kijqcen1roPEECKHBjq+mVIgMSSy4CnM6VbqGnrIr1H9MntUibBJkbomSuXAeDs uBnO731lnCGWv7NL31rVFX+fIU3wLo5l4Y1jkXd0DCPI5IAiDuXCTb/DygtA+5jU bZOzZxiWCqsD3q32ssa2aXEMWEvlawA1q3MUizvUxLl/++osSMf7uOC0/ncS94Jx 941X6kfgKcDNspSohyxUwTeWGH9BWdbDOz6ZumCgEvEUyoKZ4xPwCzl8AiN/EkEb MoR449+sbrlO3pCzev0FvRsogy1CNW0mNLVLf5YC4Fw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=5jvutv sZjig5pcuWKaPZlw5fQWVhdiBX/ijikES1MCU=; b=mGhqjbdXqJEI2nGleLJLMm s/50G+C/jAhieFYkG08o7Ch341cFY+J+wRdWm1OsNx7pFxjibuPB7nIF0x1VbUlc rBvTGgPjT8ET4GBAQzv1WS+yY57HPDtG4UA0Ni+gB8KECgoUcHgSn9Z/A3Bsg1wU nikhoL13yS5ftZczopM/irXcpIpSx8u/klvgdGzjDn0g/lATVme/AXfmRRxKHjr2 p6BMkK2bMpLVSBb0eaqXD7vw0Wo7TE9kpL7glzKmwi2Iu/bpW/DPK8QXf0WIS/zR lkEZQ5sObItkmLz8PFvxc3XN2esGQzJ/2uyL1GHJOzDvIfNOSQW+xaSjcb7qHb1g == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddvfedgieduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgefhtdefueffheekgfffudelffejtdfhvdejkedthfehvdelgfetgfdvtedthfen ucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Sep 2021 09:10:04 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 5d39a4a5 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Sep 2021 13:10:03 +0000 (UTC) Date: Wed, 1 Sep 2021 15:10:02 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v3 6/7] fetch: merge fetching and consuming refs Message-ID: <56ecbfc9c3ac03ae751f83eed7ff0b45966f9297.1630501732.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The functions `fetch_refs()` and `consume_refs()` must always be called together such that we first obtain all missing objects and then update our local refs to match the remote refs. In a subsequent patch, we'll further require that `fetch_refs()` must always be called before `consume_refs()` such that it can correctly assert that we have all objects after the fetch given that we're about to move the connectivity check. Make this requirement explicit by merging both functions into a single `fetch_and_consume_refs()` function. Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 30 +++++++++--------------------- 1 file changed, 9 insertions(+), 21 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index bafeb3d44b..09f7742d0f 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1291,8 +1291,9 @@ static int check_exist_and_connected(struct ref *ref_map) return check_connected(iterate_ref_map, &rm, &opt); } -static int fetch_refs(struct transport *transport, struct ref *ref_map) +static int fetch_and_consume_refs(struct transport *transport, struct ref *ref_map) { + int connectivity_checked; int ret; /* @@ -1308,30 +1309,18 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map) goto out; } - /* - * Keep the new pack's ".keep" file around to allow the caller - * time to update refs to reference the new objects. - */ - return ret; - -out: - transport_unlock_pack(transport); - return ret; -} - -/* Update local refs based on the ref values fetched from a remote */ -static int consume_refs(struct transport *transport, struct ref *ref_map) -{ - int connectivity_checked = transport->smart_options + connectivity_checked = transport->smart_options ? transport->smart_options->connectivity_checked : 0; - int ret; + trace2_region_enter("fetch", "consume_refs", the_repository); ret = store_updated_refs(transport->url, transport->remote->name, connectivity_checked, ref_map); - transport_unlock_pack(transport); trace2_region_leave("fetch", "consume_refs", the_repository); + +out: + transport_unlock_pack(transport); return ret; } @@ -1518,8 +1507,7 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map) transport_set_option(transport, TRANS_OPT_FOLLOWTAGS, NULL); transport_set_option(transport, TRANS_OPT_DEPTH, "0"); transport_set_option(transport, TRANS_OPT_DEEPEN_RELATIVE, NULL); - if (!fetch_refs(transport, ref_map)) - consume_refs(transport, ref_map); + fetch_and_consume_refs(transport, ref_map); if (gsecondary) { transport_disconnect(gsecondary); @@ -1610,7 +1598,7 @@ static int do_fetch(struct transport *transport, transport->url); } } - if (fetch_refs(transport, ref_map) || consume_refs(transport, ref_map)) { + if (fetch_and_consume_refs(transport, ref_map)) { free_refs(ref_map); retcode = 1; goto cleanup; From patchwork Wed Sep 1 13:10:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12469357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3C42C432BE for ; Wed, 1 Sep 2021 13:10:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 990CE60FED for ; Wed, 1 Sep 2021 13:10:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344395AbhIANLM (ORCPT ); Wed, 1 Sep 2021 09:11:12 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:58879 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344396AbhIANLG (ORCPT ); Wed, 1 Sep 2021 09:11:06 -0400 Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 76D545C013C; Wed, 1 Sep 2021 09:10:09 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 01 Sep 2021 09:10:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=GYOU5gPPQF+G93mSqq/ZTCNWbmP RnjC6KRfDlCIwuCo=; b=m6TBa4plOGHZ75u21Utbds2AD2giTJ6445tntTRbxOb b5Hq7dMPRTVV/MFmG1oymySI8NLn5Vih1PfjrMtdI+K5tYQ6a/ZUjQJkiWSrs+SD Yey9m01fBRrIaxJx2egeDBjqr31R/4xtpH7Db7yxIXVjq55/8xJg0IkuyynNtnuN GGMVHX2yx6Vo0HSxcUJCbtUALC9iuci2J99IezqhK1dQa6Nmgtbmd9WrucODIivI +9RIhgld2cWc2zdzp9qTUCgOXf2bi7nWQLFThvpiudrPLBuwWx9GYEGFTy5V4Ji7 hMIoHvuId6M2XMcBtb8Umuqbe3nuFl7DtwOJ0JBq9fQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=GYOU5g PPQF+G93mSqq/ZTCNWbmPRnjC6KRfDlCIwuCo=; b=afkSZVokY3m/yzRrHEI+Vz tMJu5v8b2zpdoC4SsGNreb/J/cq6yrIPyPD5fUdWxoXtJXeMEn25loUNiL7N4WTO Gq+h7oreoU8L20dlOrvH38dUDYW+nej0lp0ORar0G9W0GhrV5xM+32YdRZQeq6mx 3CRpx1CgOL4LZCM5Yp+ZL90wptshuCQ+hO1w1xcQxFdk3KW7mdhNjamcepXuERbk acOESMWg7TuGLJv/7snvMErWE0CfxlTTI5SrC6hybFZEMREze6MxmkKXTwUHmME+ aKPtqOKvC2RpRkFxcKdyw45nozMg+ec30DzZoK7gp151LiBjQwaRL+VGQvmU0ZLw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddvfedgieduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttdejnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgfejueevjeetudehgffffeffvdejfeejiedvkeffgfekuefgheevteeufeelkeen ucevlhhushhtvghrufhiiigvpedvnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Sep 2021 09:10:08 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 9092f07c (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Sep 2021 13:10:08 +0000 (UTC) Date: Wed, 1 Sep 2021 15:10:06 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v3 7/7] fetch: avoid second connectivity check if we already have all objects Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When fetching refs, we are doing two connectivity checks: - The first one is done such that we can skip fetching refs in the case where we already have all objects referenced by the updated set of refs. - The second one verifies that we have all objects after we have fetched objects. We always execute both connectivity checks, but this is wasteful in case the first connectivity check already notices that we have all objects locally available. Skip the second connectivity check in case we already had all objects available. This gives us a nice speedup when doing a mirror-fetch in a repository with about 2.3M refs where the fetching repo already has all objects: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 30.025 s ± 0.081 s [User: 27.070 s, System: 4.933 s] Range (min … max): 29.900 s … 30.111 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 25.574 s ± 0.177 s [User: 22.855 s, System: 4.683 s] Range (min … max): 25.399 s … 25.765 s 5 runs Summary 'HEAD: git-fetch' ran 1.17 ± 0.01 times faster than 'HEAD~: git-fetch' Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 09f7742d0f..6fdd01fb01 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1293,7 +1293,7 @@ static int check_exist_and_connected(struct ref *ref_map) static int fetch_and_consume_refs(struct transport *transport, struct ref *ref_map) { - int connectivity_checked; + int connectivity_checked = 1; int ret; /* @@ -1307,11 +1307,10 @@ static int fetch_and_consume_refs(struct transport *transport, struct ref *ref_m trace2_region_leave("fetch", "fetch_refs", the_repository); if (ret) goto out; + connectivity_checked = transport->smart_options ? + transport->smart_options->connectivity_checked : 0; } - connectivity_checked = transport->smart_options - ? transport->smart_options->connectivity_checked : 0; - trace2_region_enter("fetch", "consume_refs", the_repository); ret = store_updated_refs(transport->url, transport->remote->name,