From patchwork Fri Aug 20 10:08:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12448815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B1ECC4338F for ; Fri, 20 Aug 2021 10:08:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 65861610D2 for ; Fri, 20 Aug 2021 10:08:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237446AbhHTKJR (ORCPT ); Fri, 20 Aug 2021 06:09:17 -0400 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:38183 "EHLO out3-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237267AbhHTKJN (ORCPT ); Fri, 20 Aug 2021 06:09:13 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 204555C0160; Fri, 20 Aug 2021 06:08:35 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Fri, 20 Aug 2021 06:08:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=UHLSZZnC1WCALUjKpySHxFjKozS 5TnlJWoHn7EgNQF4=; b=G6ojw3pPvG22PcGYzAdD+TlsYxqTbrp+Ka+63QTK+0s 8Fr9HFUrHBZmNcgUlULNcZlcg349fe3Fk8gFTMHiRPUK5yyXGUllBd13Y1DnBpjO 6FdIBhdFIiqL/lCB8nx2CnMli5XQbpk5A2Ef50FrAvl2NswSw+V4/qMwianLS9WX RQ99QNrqHwRqrFb5y9mliYH2Ue3wmx3X2TnnLWR+Nxm6jaDPHW1S6dOuhtS/+ABV ia9+0f8ACLXVw4ph21e8zIbLssEK/8SiV/49WeapEV9aYwgKzsgzVzNdPuQZ6H3H 6DoyRIYCFujDD1fJsySYBcNO4dkd6cH+1deOLto4cVQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=UHLSZZ nC1WCALUjKpySHxFjKozS5TnlJWoHn7EgNQF4=; b=in/QuW3Nh9rgh12sZTYsti S21IHg/zOXX515ssysW5EZ2AfOMWsWamlRYtXlkq9u7eADm70nbIHg7EZF4PLKFN efS+Dj5LI2+4IO0NPMKRkND7EskULetqA2RZhxmOJjchPuwJ+HykDFQDILjW944r eSgKVXeEq2bG/2JNqdaXKlvXbjJM8DtQQ3msQptJP0AWscK2uC5oA3gYq9Ycnr1/ hSQd7JsBelTKt5qdXIO05s1p8Io9UdJHbS4ETNmS8Mjhu9FeXtnk6VVOOr2Oaore tk1gM3J2FzGQzb7pcMhyq3UXMe4fchocGDsq0QOCa0Z5IcwB4TVhUGO7lN8+pWHw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrleelgddvvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtderre dttdejnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhk shdrihhmqeenucggtffrrghtthgvrhhnpeehgfejueevjeetudehgffffeffvdejfeejie dvkeffgfekuefgheevteeufeelkeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 20 Aug 2021 06:08:34 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 6180f4b1 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 20 Aug 2021 10:08:29 +0000 (UTC) Date: Fri, 20 Aug 2021 12:08:28 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano Subject: [PATCH 1/6] fetch: speed up lookup of want refs via commit-graph Message-ID: <6872979c4557204821d788dc3f5e1c8bef0a773c.1629452412.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When updating our local refs based on the refs fetched from the remote, we need to iterate through all requested refs and load their respective commits such that we can determine whether they need to be appended to FETCH_HEAD or not. In cases where we're fetching from a remote with exceedingly many refs, resolving these refs can be quite expensive given that we repeatedly need to unpack object headers for each of the referenced objects. Speed this up by opportunistcally trying to resolve object IDs via the commit graph: more likely than not, they're going to be a commit anyway, and this lets us avoid having to unpack object headers completely in case the object is a commit that is part of the commit-graph. This significantly speeds up mirror-fetches in a real-world repository with 2.3M refs: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 56.942 s ± 0.449 s [User: 53.360 s, System: 5.356 s] Range (min … max): 56.372 s … 57.533 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 33.657 s ± 0.167 s [User: 30.302 s, System: 5.181 s] Range (min … max): 33.454 s … 33.844 s 5 runs Summary 'HEAD: git-fetch' ran 1.69 ± 0.02 times faster than 'HEAD~: git-fetch' Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 405afe9bdf..73f5b286d5 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1131,11 +1131,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, continue; } - commit = lookup_commit_reference_gently(the_repository, - &rm->old_oid, - 1); - if (!commit) - rm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE; + commit = lookup_commit_in_graph(the_repository, &rm->old_oid); + if (!commit) { + commit = lookup_commit_reference_gently(the_repository, + &rm->old_oid, + 1); + if (!commit) + rm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE; + } if (rm->fetch_head_status != want_status) continue; From patchwork Fri Aug 20 10:08:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12448819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48EC0C432BE for ; Fri, 20 Aug 2021 10:08:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2C277610F9 for ; Fri, 20 Aug 2021 10:08:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238303AbhHTKJT (ORCPT ); Fri, 20 Aug 2021 06:09:19 -0400 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:51725 "EHLO out3-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237971AbhHTKJN (ORCPT ); Fri, 20 Aug 2021 06:09:13 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id D90F25C0113; Fri, 20 Aug 2021 06:08:35 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Fri, 20 Aug 2021 06:08:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=uIjpf8tR+6ibddGUl+f3r7X7lPV eV6i1QGmt+3Jh2C0=; b=jRlq3SnIwSNnTcyeo4vOsXapDDAQa8wcQf9R4AICb+V YMJp6oubH8rssnDII+kQYjLNEuVdCIvmkvhxNZ4b0JORTJ7b9e89DbZAk80jqXgF Q8iXiMckOx9CswlZy7dtEyLySE8Woiop3Ug1Y+hi167A9weTXIFb6HyYx/zHnrME Q5h+yuUYV/jG8/8+VBVZvboVkxtkyoK+rmD1Bj/YxW3udj2yWODUMC8q8KL5PwyG 9rGTr6aqW/xyGqgPniRMltfEpRkFF0YQ1aoEkiwc6LS7JHjjq5njdNnyhgfqgKuK 6wYY2TrAeDta2wWsCJRG3+x1A4QYmF525tPAOcdEqIQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=uIjpf8 tR+6ibddGUl+f3r7X7lPVeV6i1QGmt+3Jh2C0=; b=gKQbW0xxZP6Rh66vLLn/mD iZBCn49eGKPdabY/Jgc+ZPOhocyHJ/lRsgLnAG+D5h0ngL8Meudw6g5uQs1JwniZ Ad42Banumsftyqt9RR5Wb9xeH+iPI0+2wam4snwtxCziB0KQrMLj49VOmnd9/CWI qWipTVZowkDCJxGORqaKqKwcJ86/mTkvqBH5Vz5YRYCPIXZfhL0RFWX3wHLnMsmO eRQ1fU38Za1JECOww4Z2Yxy5jBXcHqNhK5U5tNxcGazpPT7YyjYA7cYyF6/DcsZB jhkUtKzEAGZPjRwc7wwmGpHLnEhYKMwMvtKP4HCNKlRWnXoWFYdM9u2teGEyhqtg == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrleelgddvfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtderre dttdejnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhk shdrihhmqeenucggtffrrghtthgvrhhnpeehgfejueevjeetudehgffffeffvdejfeejie dvkeffgfekuefgheevteeufeelkeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 20 Aug 2021 06:08:34 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id ea9f842d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 20 Aug 2021 10:08:34 +0000 (UTC) Date: Fri, 20 Aug 2021 12:08:32 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano Subject: [PATCH 2/6] fetch: avoid unpacking headers in object existence check Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When updating local refs after the fetch has transferred all objects, we do an object existence test as a safety guard to avoid updating a ref to an object which we don't have. We do so via `oid_object_info()`: if it returns an error, then we know the object does not exist. One side effect of `oid_object_info()` is that it parses the object's type, and to do so it must unpack the object header. This is completely pointless: we don't care for the type, but only want to assert that the object exists. Refactor the code to use `repo_has_object_file()`, which both makes the code's intent clearer and is also faster because it does not unpack object headers. In a real-world repo with 2.3M refs, this results in a small speedup when doing a mirror-fetch: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 33.686 s ± 0.176 s [User: 30.119 s, System: 5.262 s] Range (min … max): 33.512 s … 33.944 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 31.247 s ± 0.195 s [User: 28.135 s, System: 5.066 s] Range (min … max): 30.948 s … 31.472 s 5 runs Summary 'HEAD: git-fetch' ran 1.08 ± 0.01 times faster than 'HEAD~: git-fetch' Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 73f5b286d5..5fd0f7c791 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -846,13 +846,11 @@ static int update_local_ref(struct ref *ref, int summary_width) { struct commit *current = NULL, *updated; - enum object_type type; struct branch *current_branch = branch_get(NULL); const char *pretty_ref = prettify_refname(ref->name); int fast_forward = 0; - type = oid_object_info(the_repository, &ref->new_oid, NULL); - if (type < 0) + if (!repo_has_object_file(the_repository, &ref->new_oid)) die(_("object %s not found"), oid_to_hex(&ref->new_oid)); if (oideq(&ref->old_oid, &ref->new_oid)) { From patchwork Fri Aug 20 10:08:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12448821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1437C4338F for ; Fri, 20 Aug 2021 10:08:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D6F8F610F9 for ; Fri, 20 Aug 2021 10:08:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238400AbhHTKJV (ORCPT ); Fri, 20 Aug 2021 06:09:21 -0400 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:57065 "EHLO out3-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237943AbhHTKJR (ORCPT ); Fri, 20 Aug 2021 06:09:17 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 06ED05C008E; Fri, 20 Aug 2021 06:08:40 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 20 Aug 2021 06:08:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=LZEdixNdjsqGnzyLCbI1+ntGcH0 xGuRDwcSdmxuUfl4=; b=vPVBWL5LSpsp8/JOe2BvZ+Y9F/oC3uk/3I7pEsi/xzC t9egr3KBMrIQLG3KjLLxowtUA6ooTLPAbkWPLB2rUaAuMPddGjp0XANfjIATbaKG sFku5afoa6RzKxe7JtB3ikh9QjOH4e/37ttNHko6Zpe6b6YhUv37DdJZpiDe1QDE twJlO3xsagiELGZW1DJPT7UQWiLvtJYqY82UEO65TsFsDiQBLU2vsEZkwI4LnMRC Ns7TwWkmar6qOC8zMTQvZs40Sr42HCgqVklDi10XbONJz20uwdsIJWgGFIQTjA5p UsHsKVWeADLSyelaOIeF1QpAcJ3G1mljmrnWwiGCzbw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=LZEdix NdjsqGnzyLCbI1+ntGcH0xGuRDwcSdmxuUfl4=; b=IM+71MQ5oN6ijgfp2a+S9Q vKUka0ghK4yjYMi1Rbs38Pww4eOfGRAQkTb/upF3jkKB0ezK1YqbA7J+Zmir9zZg aJlzht6ABJ4+qdfMw6iAg1DdP5+YNvPQF1Y0TevsMDNDz4f6TYzVMRRHMoCXca19 ftl12a3PsvXk6BUWiVm8PVEOhk16Xjoc1sSh9HQNK2rHaE/PCSEKeRAdSGMqXtze sMQ3mgX+WJxu5CFMUvtEBtDMwSG/TDtVse+CqtmeRvCwrdQKqFEaP7x+tCdSj6wx 3XOdpAXcvTQNKDeb7p55wCabu1hkyZSkPjprzgU3qSU9kNw8NIsIvBffuoW/5spQ == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrleelgddvvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtderre dttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhk shdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtdfhvd ejkedthfehvdelgfetgfdvtedthfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 20 Aug 2021 06:08:39 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 39dda667 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 20 Aug 2021 10:08:38 +0000 (UTC) Date: Fri, 20 Aug 2021 12:08:37 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano Subject: [PATCH 3/6] connected: refactor iterator to return next object ID directly Message-ID: <3bdad7bc8b0debd44138a4d3df5744d5a245475d.1629452412.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The object ID iterator used by the connectivity checks returns the next object ID via an out-parameter and then uses a return code to indicate whether an item was found. This is a bit roundabout: instead of a separate error code, we can just retrun the next object ID directly and use `NULL` pointers as indicator that the iterator got no items left. Furthermore, this avoids a copy of the object ID. Refactor the iterator and all its implementations to return object IDs directly. While I was honestly hoping for a small speedup given that we can now avoid a copy, both versions perform the same. Still, the end result is easier to understand and thus it makes sense to keep this refactoring regardless. Signed-off-by: Patrick Steinhardt --- builtin/clone.c | 8 +++----- builtin/fetch.c | 7 +++---- builtin/receive-pack.c | 17 +++++++---------- connected.c | 15 ++++++++------- connected.h | 2 +- fetch-pack.c | 7 +++---- 6 files changed, 25 insertions(+), 31 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index a74558f30c..817a651936 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -544,7 +544,7 @@ static void write_followtags(const struct ref *refs, const char *msg) } } -static int iterate_ref_map(void *cb_data, struct object_id *oid) +static struct object_id *iterate_ref_map(void *cb_data) { struct ref **rm = cb_data; struct ref *ref = *rm; @@ -555,13 +555,11 @@ static int iterate_ref_map(void *cb_data, struct object_id *oid) */ while (ref && !ref->peer_ref) ref = ref->next; - /* Returning -1 notes "end of list" to the caller. */ if (!ref) - return -1; + return NULL; - oidcpy(oid, &ref->old_oid); *rm = ref->next; - return 0; + return &ref->old_oid; } static void update_remote_refs(const struct ref *refs, diff --git a/builtin/fetch.c b/builtin/fetch.c index 5fd0f7c791..76ce73ccf5 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -962,7 +962,7 @@ static int update_local_ref(struct ref *ref, } } -static int iterate_ref_map(void *cb_data, struct object_id *oid) +static struct object_id *iterate_ref_map(void *cb_data) { struct ref **rm = cb_data; struct ref *ref = *rm; @@ -970,10 +970,9 @@ static int iterate_ref_map(void *cb_data, struct object_id *oid) while (ref && ref->status == REF_STATUS_REJECT_SHALLOW) ref = ref->next; if (!ref) - return -1; /* end of the list */ + return NULL; *rm = ref->next; - oidcpy(oid, &ref->old_oid); - return 0; + return &ref->old_oid; } struct fetch_head { diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index a419de5b38..0abda033bc 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -1307,7 +1307,7 @@ static void refuse_unconfigured_deny_delete_current(void) rp_error("%s", _(refuse_unconfigured_deny_delete_current_msg)); } -static int command_singleton_iterator(void *cb_data, struct object_id *oid); +static struct object_id *command_singleton_iterator(void *cb_data); static int update_shallow_ref(struct command *cmd, struct shallow_info *si) { struct shallow_lock shallow_lock = SHALLOW_LOCK_INIT; @@ -1725,16 +1725,15 @@ static void check_aliased_updates(struct command *commands) string_list_clear(&ref_list, 0); } -static int command_singleton_iterator(void *cb_data, struct object_id *oid) +static struct object_id *command_singleton_iterator(void *cb_data) { struct command **cmd_list = cb_data; struct command *cmd = *cmd_list; if (!cmd || is_null_oid(&cmd->new_oid)) - return -1; /* end of list */ + return NULL; *cmd_list = NULL; /* this returns only one */ - oidcpy(oid, &cmd->new_oid); - return 0; + return &cmd->new_oid; } static void set_connectivity_errors(struct command *commands, @@ -1764,7 +1763,7 @@ struct iterate_data { struct shallow_info *si; }; -static int iterate_receive_command_list(void *cb_data, struct object_id *oid) +static struct object_id *iterate_receive_command_list(void *cb_data) { struct iterate_data *data = cb_data; struct command **cmd_list = &data->cmds; @@ -1775,13 +1774,11 @@ static int iterate_receive_command_list(void *cb_data, struct object_id *oid) /* to be checked in update_shallow_ref() */ continue; if (!is_null_oid(&cmd->new_oid) && !cmd->skip_update) { - oidcpy(oid, &cmd->new_oid); *cmd_list = cmd->next; - return 0; + return &cmd->new_oid; } } - *cmd_list = NULL; - return -1; /* end of list */ + return NULL; } static void reject_updates_to_hidden(struct command *commands) diff --git a/connected.c b/connected.c index b5f9523a5f..374b145355 100644 --- a/connected.c +++ b/connected.c @@ -24,7 +24,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, struct child_process rev_list = CHILD_PROCESS_INIT; FILE *rev_list_in; struct check_connected_options defaults = CHECK_CONNECTED_INIT; - struct object_id oid; + struct object_id *oid; int err = 0; struct packed_git *new_pack = NULL; struct transport *transport; @@ -34,7 +34,8 @@ int check_connected(oid_iterate_fn fn, void *cb_data, opt = &defaults; transport = opt->transport; - if (fn(cb_data, &oid)) { + oid = fn(cb_data); + if (!oid) { if (opt->err_fd) close(opt->err_fd); return err; @@ -73,7 +74,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, for (p = get_all_packs(the_repository); p; p = p->next) { if (!p->pack_promisor) continue; - if (find_pack_entry_one(oid.hash, p)) + if (find_pack_entry_one(oid->hash, p)) goto promisor_pack_found; } /* @@ -83,7 +84,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, goto no_promisor_pack_found; promisor_pack_found: ; - } while (!fn(cb_data, &oid)); + } while ((oid = fn(cb_data)) != NULL); return 0; } @@ -133,12 +134,12 @@ int check_connected(oid_iterate_fn fn, void *cb_data, * are sure the ref is good and not sending it to * rev-list for verification. */ - if (new_pack && find_pack_entry_one(oid.hash, new_pack)) + if (new_pack && find_pack_entry_one(oid->hash, new_pack)) continue; - if (fprintf(rev_list_in, "%s\n", oid_to_hex(&oid)) < 0) + if (fprintf(rev_list_in, "%s\n", oid_to_hex(oid)) < 0) break; - } while (!fn(cb_data, &oid)); + } while ((oid = fn(cb_data)) != NULL); if (ferror(rev_list_in) || fflush(rev_list_in)) { if (errno != EPIPE && errno != EINVAL) diff --git a/connected.h b/connected.h index 8d5a6b3ad6..56cc95be2d 100644 --- a/connected.h +++ b/connected.h @@ -9,7 +9,7 @@ struct transport; * When called after returning the name for the last object, return -1 * to signal EOF, otherwise return 0. */ -typedef int (*oid_iterate_fn)(void *, struct object_id *oid); +typedef struct object_id *(*oid_iterate_fn)(void *); /* * Named-arguments struct for check_connected. All arguments are diff --git a/fetch-pack.c b/fetch-pack.c index 0bf7ed7e47..1a6242cd71 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1912,16 +1912,15 @@ static void update_shallow(struct fetch_pack_args *args, oid_array_clear(&ref); } -static int iterate_ref_map(void *cb_data, struct object_id *oid) +static struct object_id *iterate_ref_map(void *cb_data) { struct ref **rm = cb_data; struct ref *ref = *rm; if (!ref) - return -1; /* end of the list */ + return NULL; *rm = ref->next; - oidcpy(oid, &ref->old_oid); - return 0; + return &ref->old_oid; } struct ref *fetch_pack(struct fetch_pack_args *args, From patchwork Fri Aug 20 10:08:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12448823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 076DAC432BE for ; Fri, 20 Aug 2021 10:08:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E59E16024A for ; Fri, 20 Aug 2021 10:08:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238333AbhHTKJY (ORCPT ); Fri, 20 Aug 2021 06:09:24 -0400 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:46859 "EHLO out3-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238748AbhHTKJW (ORCPT ); Fri, 20 Aug 2021 06:09:22 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 538045C00D0; Fri, 20 Aug 2021 06:08:44 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Fri, 20 Aug 2021 06:08:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=SpJ+4ony9KZCPhpIoQmxliNKfev uC4OHmprV+nP9Q6Y=; b=q0QUdeVANSEJhy2aA289ms4ZjhtbJVIBEsQgtEtgp5g 7vjp3V+LwfK4q/aVp+r8Brw9n7Kvu7+9H+5xYMeJqlbZQk92L4v3IvUiWV7riWsq zi5BEWgCgGPh2q255dqR2ytteLPn04Nlj65kzHjVYOcBjpU/lObJlPoX6Cq3WEl0 a0I1MmmM3/FJHlEmsQKq8T0J66lWnOwPBb+zVEhnHJR/KMqPOjFL/J1e/8jx/TAH Fm4nnh83p/7v+6Fyv0S9zo6lHnItufnV+FN+YKZfO2lSP56iJJJaw2FfdlHryf3c c6m2nGUyBx1Zt8UeRdsvYLje9Iqzp5np68nDr/oSPTw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=SpJ+4o ny9KZCPhpIoQmxliNKfevuC4OHmprV+nP9Q6Y=; b=nWdIpjamcTPCbtbpz7C6Ze /SuNGRfm9ZCI1gj2yMIWl6aFAFQ61RdPr+FHNKK8kChDvWB6xp/I36T+xRbZV/d3 0k83WVqS83QmYkRlRsCYQ0mnCLu0wVXAJFYkcI2mmufiEJvQMr2tA+p/Oe+z0L3T nGJ/95zF4vLzJjDRtFzOso1wUEG1SAaBmV5WIgdLYuX8eJ+PRv7nhaOhX+/AtTuN 4+vxw/gejxTcQCS29vg2T9nYIwurlCHs4bk2pmhwPxy+pTi9NQwVhNQx6e5UUpei A8HEiKgRv6XIE6w8M6S1Vft4O9AkU+gG8TUMSGv1PtJwv0WKW1yezpByNchoHUWA == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrleelgddvfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtderre dttdejnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhk shdrihhmqeenucggtffrrghtthgvrhhnpeehgfejueevjeetudehgffffeffvdejfeejie dvkeffgfekuefgheevteeufeelkeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 20 Aug 2021 06:08:43 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id a8ef8c86 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 20 Aug 2021 10:08:42 +0000 (UTC) Date: Fri, 20 Aug 2021 12:08:41 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano Subject: [PATCH 4/6] fetch-pack: optimize loading of refs via commit graph Message-ID: <67917af7ceeefe41ae0f6edf69cd61e2ee8c0ea3.1629452412.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In order to negotiate a packfile, we need to dereference refs to see which commits we have in common with the remote. To do so, we first look up the object's type -- if it's a tag, we peel until we hit a non-tag object. If we hit a commit eventually, then we return that commit. In case the object ID points to a commit directly, we can avoid the initial lookup of the object type by opportunistically looking up the commit via the commit-graph, if available, which gives us a slight speed bump of about 2% in a huge repository with about 2.3M refs: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 31.634 s ± 0.258 s [User: 28.400 s, System: 5.090 s] Range (min … max): 31.280 s … 31.896 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 31.129 s ± 0.543 s [User: 27.976 s, System: 5.056 s] Range (min … max): 30.172 s … 31.479 s 5 runs Summary 'HEAD: git-fetch' ran 1.02 ± 0.02 times faster than 'HEAD~: git-fetch' In case this fails, we fall back to the old code which peels the objects to a commit. --- fetch-pack.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/fetch-pack.c b/fetch-pack.c index 1a6242cd71..c57faf278f 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -119,6 +119,11 @@ static struct commit *deref_without_lazy_fetch(const struct object_id *oid, { enum object_type type; struct object_info info = { .typep = &type }; + struct commit *commit; + + commit = lookup_commit_in_graph(the_repository, oid); + if (commit) + return commit; while (1) { if (oid_object_info_extended(the_repository, oid, &info, @@ -139,7 +144,7 @@ static struct commit *deref_without_lazy_fetch(const struct object_id *oid, } if (type == OBJ_COMMIT) { - struct commit *commit = lookup_commit(the_repository, oid); + commit = lookup_commit(the_repository, oid); if (!commit || repo_parse_commit(the_repository, commit)) return NULL; return commit; From patchwork Fri Aug 20 10:08:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12448825 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5577FC4338F for ; Fri, 20 Aug 2021 10:08:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 39547610D2 for ; Fri, 20 Aug 2021 10:08:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238882AbhHTKJb (ORCPT ); Fri, 20 Aug 2021 06:09:31 -0400 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:33259 "EHLO out3-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238492AbhHTKJ0 (ORCPT ); Fri, 20 Aug 2021 06:09:26 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 4510D5C008E; Fri, 20 Aug 2021 06:08:48 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Fri, 20 Aug 2021 06:08:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=SfCRg8AncORoKzoD8jgUsNnr77n /s9Y1vMAtQGdGvWM=; b=BgdjiyD1h0SZFGF4RdT0x/yL7iya4ct2/gKwIZCtkCg n8bHAAaooNEXXH6ztjzRTVyxqEQjIb2z1zC0VFOXHF7Nzhpz0LawRi5dMzE47VRZ 21ANaf9eIegAxreRSxUbnrdWrrBqWHWVWB7oBTyx6WTqLI932VfuW95ieqkdTygB TOOCuJClrRPghMu+oagDbuYoTp0N3v3HzHAC7iyr4PbwznBW734zaJXlqk9neTLi FRdxGZU92YJz4AoyGUzOtZyz0+sG8fUWYj8zp72ov5lwKf655R0obPcwoCTjSd5h XF1x+CH6PexXsGmEzzb/S3spn6AURhbMIdyNRAkDq5g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=SfCRg8 AncORoKzoD8jgUsNnr77n/s9Y1vMAtQGdGvWM=; b=LwcsfJIkUmiQrzJoEC03TS 17w/LxMANAc2ZNb4XxtJCHaMNUBY+TGOu5CC8ygZe2Y9z/a5GdLR4IT7CqoaB6wc Wsdrb3Ebm6zZ2SDcTAm60fmj22wL4oLbU7bMpkkzkeQ54OZeJGr10l4PkS7BfCEK /ofsVUb0DnW/7XGWyz/Kbb+akZae5wr+eyKUf2QqV+rY1jmvwpSHJkfRvC+hMG38 WHQnHxbRpQf8/q9UU0zLHdjG597ytPLauNejbv/DTOTb1941IT44OuFqlcjXNZJP RFcP2KNjR+X4ULiYFoeY1Vz6yT0LYdmoc1y4uUmRxYSprIKtX6GCkO20Fs7gp5Vg == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrleelgddvfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtderre dttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhk shdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtdfhvd ejkedthfehvdelgfetgfdvtedthfenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 20 Aug 2021 06:08:47 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id dff0bf99 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 20 Aug 2021 10:08:46 +0000 (UTC) Date: Fri, 20 Aug 2021 12:08:45 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano Subject: [PATCH 5/6] fetch: refactor fetch refs to be more extendable Message-ID: <7653f8eabc1eb9c26e06ad3efa3d7e19dc9547cb.1629452412.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Refactor `fetch_refs()` code to make it more extendable by explicitly handling error cases. The refactored code should behave the same. Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 76ce73ccf5..20fcfe0f45 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1284,20 +1284,29 @@ static int check_exist_and_connected(struct ref *ref_map) static int fetch_refs(struct transport *transport, struct ref *ref_map) { - int ret = check_exist_and_connected(ref_map); - if (ret) { - trace2_region_enter("fetch", "fetch_refs", the_repository); - ret = transport_fetch_refs(transport, ref_map); - trace2_region_leave("fetch", "fetch_refs", the_repository); - } + int ret; + + /* + * We don't need to perform a fetch in case we can already satisfy all + * refs. + */ + ret = check_exist_and_connected(ref_map); if (!ret) - /* - * Keep the new pack's ".keep" file around to allow the caller - * time to update refs to reference the new objects. - */ return 0; - transport_unlock_pack(transport); - return ret; + + trace2_region_enter("fetch", "fetch_refs", the_repository); + ret = transport_fetch_refs(transport, ref_map); + trace2_region_leave("fetch", "fetch_refs", the_repository); + if (ret) { + transport_unlock_pack(transport); + return ret; + } + + /* + * Keep the new pack's ".keep" file around to allow the caller + * time to update refs to reference the new objects. + */ + return 0; } /* Update local refs based on the ref values fetched from a remote */ From patchwork Fri Aug 20 10:08:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12448827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEA85C432BE for ; Fri, 20 Aug 2021 10:08:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 99CC1610D2 for ; Fri, 20 Aug 2021 10:08:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238879AbhHTKJd (ORCPT ); Fri, 20 Aug 2021 06:09:33 -0400 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:41837 "EHLO out3-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238994AbhHTKJa (ORCPT ); Fri, 20 Aug 2021 06:09:30 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 429EC5C0048; Fri, 20 Aug 2021 06:08:52 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Fri, 20 Aug 2021 06:08:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=mBohNgS478ywCUXq1+GD9AOmMK6 cqqhS2cUnOebg/Sc=; b=ub5Ib3xpDx6EGRro+Y5XK94qIBkV0eDkgJde2fF+5Pu Dic0mf0/a29EDpkS9/Eqm6H+r2qYI2ur/iIFJVqh2iLOz/TSh+4bfaHRrkjjq4KA POXacWtEUbjrO+EpFfigod4DteZITkdCl9k48qa8nr6uFLH3R6ASkJfTuC9OEVNv 9WTABso+DAfz8H8U33d+VU4I9uclzBGUNrC7aKWkUWFXtAEdIEc0P115QM118FlB TJyg6L8/10mVpXWbrdl94vDp5GLNq4TEW01Yye7F/00X6ffM3fgXElnLQlpmmhYm 1m+RYPiBLfTziZiXDG3Ot71Wipt2BnziUWx542++e7Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=mBohNg S478ywCUXq1+GD9AOmMK6cqqhS2cUnOebg/Sc=; b=v/8AyZZA49zMQ7050v2/3F 03Rmgrb0Vquf/NmRCG1eNKiiQxv9jy5zvSP8eKAwVzz7dwSpnmRyiW2HrxMaH3Tb 4DioI70fVfrtwWTbXerIVdCmUG64pQeVPIGg1RLwZrxlSRXi5KhtuYnDlMrMaPpX uz9lOZnxgnapjxlfAEplf5GJRpubpWZRiWw1NmBvVrvPDSyntUwGPN3nQcAuTXWe faCPZStoT3TunRLKjf0OHQzey+KhKHrttSDTlMvxXe3nvR+QmWU9cJy6GeUu0lBd 3Ken4No/czGNsSK/14n/X7hPkg2ONNO0HkTa4IbDbNmPvbglVjRydgDjffrJCq3A == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrleelgddvfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtderre dttdejnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhk shdrihhmqeenucggtffrrghtthgvrhhnpeehgfejueevjeetudehgffffeffvdejfeejie dvkeffgfekuefgheevteeufeelkeenucevlhhushhtvghrufhiiigvpedunecurfgrrhgr mhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 20 Aug 2021 06:08:51 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 85dbd786 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 20 Aug 2021 10:08:50 +0000 (UTC) Date: Fri, 20 Aug 2021 12:08:49 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano Subject: [PATCH 6/6] fetch: avoid second connectivity check if we already have all objects Message-ID: <646ac90e62aab4e4aec595d6848b60233bbe8c77.1629452412.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When fetching refs, we are doing two connectivity checks: - The first one in `fetch_refs()` is done such that we can short-circuit the case where we already have all objects referenced by the updated set of refs. - The second one in `store_updated_refs()` does a sanity check that we have all objects after we have fetched the packfile. We always execute both connectivity checks, but this is wasteful in case the first connectivity check already notices that we have all objects locally available. Refactor the code to do both connectivity checks in `fetch_refs()`, which allows us to easily skip the second connectivity check if we already have all objects available. This refactoring is safe to do given that we always call `fetch_refs()` followed by `consume_refs()`, which is the only caller of `store_updated_refs()`. This gives us a nice speedup when doing a mirror-fetch in a repository with about 2.3M refs where the fetching repo already has all objects: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 31.232 s ± 0.082 s [User: 27.901 s, System: 5.178 s] Range (min … max): 31.118 s … 31.301 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 26.616 s ± 0.100 s [User: 23.675 s, System: 4.752 s] Range (min … max): 26.544 s … 26.788 s 5 runs Summary 'HEAD: git-fetch' ran 1.17 ± 0.01 times faster than 'HEAD~: git-fetch' Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 20fcfe0f45..088a8af13b 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1068,7 +1068,7 @@ N_("It took %.2f seconds to check forced updates. You can use\n" " to avoid this check.\n"); static int store_updated_refs(const char *raw_url, const char *remote_name, - int connectivity_checked, struct ref *ref_map) + struct ref *ref_map) { struct fetch_head fetch_head; struct commit *commit; @@ -1090,16 +1090,6 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, else url = xstrdup("foreign"); - if (!connectivity_checked) { - struct check_connected_options opt = CHECK_CONNECTED_INIT; - - rm = ref_map; - if (check_connected(iterate_ref_map, &rm, &opt)) { - rc = error(_("%s did not send all necessary objects\n"), url); - goto abort; - } - } - if (atomic_fetch) { transaction = ref_transaction_begin(&err); if (!transaction) { @@ -1302,6 +1292,18 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map) return ret; } + /* + * If the transport didn't yet check for us, we need to verify + * ourselves that we have obtained all missing objects now. + */ + if (!transport->smart_options || !transport->smart_options->connectivity_checked) { + if (check_connected(iterate_ref_map, &ref_map, NULL)) { + ret = error(_("remote did not send all necessary objects\n")); + transport_unlock_pack(transport); + return ret; + } + } + /* * Keep the new pack's ".keep" file around to allow the caller * time to update refs to reference the new objects. @@ -1312,13 +1314,10 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map) /* Update local refs based on the ref values fetched from a remote */ static int consume_refs(struct transport *transport, struct ref *ref_map) { - int connectivity_checked = transport->smart_options - ? transport->smart_options->connectivity_checked : 0; int ret; trace2_region_enter("fetch", "consume_refs", the_repository); ret = store_updated_refs(transport->url, transport->remote->name, - connectivity_checked, ref_map); transport_unlock_pack(transport); trace2_region_leave("fetch", "consume_refs", the_repository);