From patchwork Tue Aug 24 10:36:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12454595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15226C4338F for ; Tue, 24 Aug 2021 10:37:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E8A1761008 for ; Tue, 24 Aug 2021 10:37:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236188AbhHXKhx (ORCPT ); Tue, 24 Aug 2021 06:37:53 -0400 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:56303 "EHLO out4-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236175AbhHXKhm (ORCPT ); Tue, 24 Aug 2021 06:37:42 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 8970B5C014D; Tue, 24 Aug 2021 06:36:58 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Tue, 24 Aug 2021 06:36:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=MVoPtfxc8OADHhfUfTIB5cKU4jF DTPrEfIRAf/D+U5c=; b=atJlw8Kd5PeodWHqki3FqCJwTJSQ0U0t9lH9qu4LUNF evS1P6SlhsZsbi2sPRm3UbNxbZOqYs/JdzobhAhREbBuL28mEF9pdBRq6AouZ4Kj N6gGKY7BUZPjUDYCjFGfOraJCV4RGGcDXn4DACNMkuNxyvJViaF0vuSq5ba93vIX BxqhTnOXqgHJ9LOBqM8gzeCQeOA5pZoKRactjxOs98JM+hOGscgNLp/o1N3MZY8m jTZ7wi3vGh24ibsxO5llms8O7Dw35OYWGD5FgIBtobvnUi/AE0lhybDRDDOiW7iq 4p92EJCYbLE1HDS0QfauKZpkCmgqrcKQPEp5C8LW3qg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=MVoPtf xc8OADHhfUfTIB5cKU4jFDTPrEfIRAf/D+U5c=; b=iyswFxxJZ4g8+FyhSJylBe nRmbrWzNXfLeK2p3wGJ/kOuEumCVbdj5QDSPeG+CArmtbYj0O03Oj8TJmzDKkKj0 2axuAsxQkQM9CB6Wgx5KXHcdbwgYpxnP3YPXJg+80ylJWY3SPAMWABTJghf47u6W L0b8NKUOdvzvNNudU+bfIlWUk42AFx+o8lWzd9ZMD2VYMO2x/bKFwmr/+RS9UPqS qBW5t+nLAK8Bhc+tzyFv4xK0XAHwa+NxGxnRSxjLLCBXJZ43bTJXX/3sj9ZNstf7 Mny+zH4nsl6Lwp8srZY8uxepKv/X2TTEU0uIsk/Od67G2JrUtm1LDwYGdSNoxWSQ == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddtjedgfedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttdejnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgfejueevjeetudehgffffeffvdejfeejiedvkeffgfekuefgheevteeufeelkeen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Aug 2021 06:36:57 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 13460ce9 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 24 Aug 2021 10:36:56 +0000 (UTC) Date: Tue, 24 Aug 2021 12:36:55 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v2 1/7] fetch: speed up lookup of want refs via commit-graph Message-ID: <4a819a68309bf03db2d9a5e5be070e52c3542af8.1629800774.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When updating our local refs based on the refs fetched from the remote, we need to iterate through all requested refs and load their respective commits such that we can determine whether they need to be appended to FETCH_HEAD or not. In cases where we're fetching from a remote with exceedingly many refs, resolving these refs can be quite expensive given that we repeatedly need to unpack object headers for each of the referenced objects. Speed this up by opportunistcally trying to resolve object IDs via the commit graph. We only do so for any refs which are not in "refs/tags": more likely than not, these are going to be a commit anyway, and this lets us avoid having to unpack object headers completely in case the object is a commit that is part of the commit-graph. This significantly speeds up mirror-fetches in a real-world repository with 2.3M refs: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 56.482 s ± 0.384 s [User: 53.340 s, System: 5.365 s] Range (min … max): 56.050 s … 57.045 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 33.727 s ± 0.170 s [User: 30.252 s, System: 5.194 s] Range (min … max): 33.452 s … 33.871 s 5 runs Summary 'HEAD: git-fetch' ran 1.67 ± 0.01 times faster than 'HEAD~: git-fetch' Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index e064687dbd..91d1301613 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1074,7 +1074,6 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, int connectivity_checked, struct ref *ref_map) { struct fetch_head fetch_head; - struct commit *commit; int url_len, i, rc = 0; struct strbuf note = STRBUF_INIT, err = STRBUF_INIT; struct ref_transaction *transaction = NULL; @@ -1122,6 +1121,7 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, want_status <= FETCH_HEAD_IGNORE; want_status++) { for (rm = ref_map; rm; rm = rm->next) { + struct commit *commit = NULL; struct ref *ref = NULL; if (rm->status == REF_STATUS_REJECT_SHALLOW) { @@ -1131,11 +1131,23 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, continue; } - commit = lookup_commit_reference_gently(the_repository, - &rm->old_oid, - 1); - if (!commit) - rm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE; + /* + * References in "refs/tags/" are often going to point + * to annotated tags, which are not part of the + * commit-graph. We thus only try to look up refs in + * the graph which are not in that namespace to not + * regress performance in repositories with many + * annotated tags. + */ + if (!starts_with(rm->name, "refs/tags/")) + commit = lookup_commit_in_graph(the_repository, &rm->old_oid); + if (!commit) { + commit = lookup_commit_reference_gently(the_repository, + &rm->old_oid, + 1); + if (!commit) + rm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE; + } if (rm->fetch_head_status != want_status) continue; From patchwork Tue Aug 24 10:37:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12454597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BAF7C4320A for ; Tue, 24 Aug 2021 10:37:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 01D8B61008 for ; Tue, 24 Aug 2021 10:37:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236207AbhHXKhx (ORCPT ); Tue, 24 Aug 2021 06:37:53 -0400 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:55497 "EHLO out4-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236261AbhHXKhu (ORCPT ); Tue, 24 Aug 2021 06:37:50 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id F39065C0121; Tue, 24 Aug 2021 06:37:05 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Tue, 24 Aug 2021 06:37:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=VfWBHeI3sZurx+ImAqL6erq3gkM r4MWLfUx7+IjLUPY=; b=PMJeRYFx5Z78vn3LwVF39sgrynHmuLSDe+GeEr+3KT2 CUYNAcQVs3I9cybQFYjmM4+657JDgjAWjZVHUU9r+kegbB1A1uWTVxMHGBnq1S4s Dy55DX8Br1WrPqJqFg87EZhLX8F2kNW/hZ1TlzbY9C5Vx8d+8XCM1RC0BnWMRalL pGnMawCelqsncZO3JIx761O+Vuz/8b/kskXD2y75UVsYXPnLdCW/EH42V7cJCBNn uJNcwYjgC9iaaFnDlTz3S+gmsy2PKCIKN9M01wZej/xUHSS/uHzqQrbTMkCUyvJ8 8yhxtJnh8+ndY3KAI2lkmH4s0B/KyYg66qsJnM2yfOQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=VfWBHe I3sZurx+ImAqL6erq3gkMr4MWLfUx7+IjLUPY=; b=S6KVIun9c0pp1BxvBcb2Od 2LHfOnMustrO+RZwxovV4K6evEA06fVC4u8+0/J3XisID5jcJaMxfigz3dlJ4zIn s0VEzKWjGM/hTAZ3uZlj6gm4PJZRS9ennEoeGLc3TthrYqbQXQxpWmY8CfIwTdIe RUXGOeJeGJVsZJI7uvLKfYu2pLqyq6Wl6UvXVbkc9lD0CpQBVevX4oS6Q/LTj+/T ap9vsFCVS6QVTrMuseRkyfNdx3hUCWXGE1RKsHKfEdIONUg3RbAyRv9baYZ4XpGI VLfddlw6a/mNB0YA3zF1sEi2PCoqniB2ftiYIS86hBri55xZ1Klzc7EDM+c1dHAw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddtjedgfedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttdejnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgfejueevjeetudehgffffeffvdejfeejiedvkeffgfekuefgheevteeufeelkeen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Aug 2021 06:37:04 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id af769317 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 24 Aug 2021 10:37:01 +0000 (UTC) Date: Tue, 24 Aug 2021 12:37:00 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v2 2/7] fetch: avoid unpacking headers in object existence check Message-ID: <81ebadabe8e861269336f6446f87acac88371042.1629800774.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When updating local refs after the fetch has transferred all objects, we do an object existence test as a safety guard to avoid updating a ref to an object which we don't have. We do so via `oid_object_info()`: if it returns an error, then we know the object does not exist. One side effect of `oid_object_info()` is that it parses the object's type, and to do so it must unpack the object header. This is completely pointless: we don't care for the type, but only want to assert that the object exists. Refactor the code to use `repo_has_object_file()`, which both makes the code's intent clearer and is also faster because it does not unpack object headers. In a real-world repo with 2.3M refs, this results in a small speedup when doing a mirror-fetch: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 33.686 s ± 0.176 s [User: 30.119 s, System: 5.262 s] Range (min … max): 33.512 s … 33.944 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 31.247 s ± 0.195 s [User: 28.135 s, System: 5.066 s] Range (min … max): 30.948 s … 31.472 s 5 runs Summary 'HEAD: git-fetch' ran 1.08 ± 0.01 times faster than 'HEAD~: git-fetch' Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 91d1301613..01513e6aea 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -846,13 +846,11 @@ static int update_local_ref(struct ref *ref, int summary_width) { struct commit *current = NULL, *updated; - enum object_type type; struct branch *current_branch = branch_get(NULL); const char *pretty_ref = prettify_refname(ref->name); int fast_forward = 0; - type = oid_object_info(the_repository, &ref->new_oid, NULL); - if (type < 0) + if (!repo_has_object_file(the_repository, &ref->new_oid)) die(_("object %s not found"), oid_to_hex(&ref->new_oid)); if (oideq(&ref->old_oid, &ref->new_oid)) { From patchwork Tue Aug 24 10:37:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12454601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3D07C4338F for ; Tue, 24 Aug 2021 10:37:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DC85661008 for ; Tue, 24 Aug 2021 10:37:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236261AbhHXKh6 (ORCPT ); Tue, 24 Aug 2021 06:37:58 -0400 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:42583 "EHLO out4-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236262AbhHXKhv (ORCPT ); Tue, 24 Aug 2021 06:37:51 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 8C9395C0079; Tue, 24 Aug 2021 06:37:07 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Tue, 24 Aug 2021 06:37:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=cY7FgXwppEBRjAqOvDLiX2PYvAb KZZWADqXAHCXSxMc=; b=QM3bmkt5kp+cObnC5PcN8iNgdPjkc8gfjvZPQBPm3zZ B+di+73NMNR02KPvpHHbaEzCo+0DYXnSucVA9ZmThwCltdToN2NUKp3qDti2er5q hvCVk/AqgCiyZMdoALqeiv2EEDFKHTixOw4Qp139oIH8jCdrIvbPXEhjz5Iw7fCE np3edPtMzgs5rJaSpr6KGOTyZclPEmPOednMtj2bulZ8LNCZ53qFgdpJV4quKpy2 jn3YtM2dLnkNVm/k0vqL4BnN3JogcLg94WMK7yOgeEoG9qel4KgdovUPsk7BHdJz WEb7DpoLyAPUEggTOiboS3kxXFXzk4WINjI18CwQStw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=cY7FgX wppEBRjAqOvDLiX2PYvAbKZZWADqXAHCXSxMc=; b=f/GHZAhLSHm9zE6E6ro5+y om3LqcFAuhz9pm4GcCQXBUnsKCF51k2uJAkTVLG+l4GHzZh3OF6GXLZTiVeJ9q9O jAA6buwZ0ydIIhxwR5uoDQP6bv6PuEnBvFfvuR5uQ/MtIDYqJBP8iVoTSzG4u0t0 A43XTkqc3X+1iKNRNOLYDxs8MAH4JRF6sase5lU6Nhno8GiChPGQgyyTmkWOoVm6 EZmJ5gwTCCGpDE0hxs/KGdQSBzRg9sqsBbImGUNcXMnz9IIE18wjA2bVRGnytHQx AG1Is0/tLvrztSNxoRbI5uj+LmaOsbA6usSgoTb9KlhwLfG+++f0NCFYBYbUaqKw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddtjedgfedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttdejnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgfejueevjeetudehgffffeffvdejfeejiedvkeffgfekuefgheevteeufeelkeen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Aug 2021 06:37:06 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 72429783 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 24 Aug 2021 10:37:05 +0000 (UTC) Date: Tue, 24 Aug 2021 12:37:04 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v2 3/7] connected: refactor iterator to return next object ID directly Message-ID: <98e981ced94b2e75ac043d0707ef025fbdc9542b.1629800774.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The object ID iterator used by the connectivity checks returns the next object ID via an out-parameter and then uses a return code to indicate whether an item was found. This is a bit roundabout: instead of a separate error code, we can just return the next object ID directly and use `NULL` pointers as indicator that the iterator got no items left. Furthermore, this avoids a copy of the object ID. Refactor the iterator and all its implementations to return object IDs directly. This brings a tiny performance improvement when doing a mirror-fetch of a repository with about 2.3M refs: Benchmark #1: 328dc58b49919c43897240f2eabfa30be2ce32a4~: git-fetch Time (mean ± σ): 30.110 s ± 0.148 s [User: 27.161 s, System: 5.075 s] Range (min … max): 29.934 s … 30.406 s 10 runs Benchmark #2: 328dc58b49919c43897240f2eabfa30be2ce32a4: git-fetch Time (mean ± σ): 29.899 s ± 0.109 s [User: 26.916 s, System: 5.104 s] Range (min … max): 29.696 s … 29.996 s 10 runs Summary '328dc58b49919c43897240f2eabfa30be2ce32a4: git-fetch' ran 1.01 ± 0.01 times faster than '328dc58b49919c43897240f2eabfa30be2ce32a4~: git-fetch' While this 1% speedup could be labelled as statistically insignificant, the speedup is consistent on my machine. Furthermore, this is an end to end test, so it is expected that the improvement in the connectivity check itself is more significant. Signed-off-by: Patrick Steinhardt --- builtin/clone.c | 8 +++----- builtin/fetch.c | 7 +++---- builtin/receive-pack.c | 17 +++++++---------- connected.c | 15 ++++++++------- connected.h | 2 +- fetch-pack.c | 7 +++---- 6 files changed, 25 insertions(+), 31 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 66fe66679c..4a1056fcc2 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -657,7 +657,7 @@ static void write_followtags(const struct ref *refs, const char *msg) } } -static int iterate_ref_map(void *cb_data, struct object_id *oid) +static const struct object_id *iterate_ref_map(void *cb_data) { struct ref **rm = cb_data; struct ref *ref = *rm; @@ -668,13 +668,11 @@ static int iterate_ref_map(void *cb_data, struct object_id *oid) */ while (ref && !ref->peer_ref) ref = ref->next; - /* Returning -1 notes "end of list" to the caller. */ if (!ref) - return -1; + return NULL; - oidcpy(oid, &ref->old_oid); *rm = ref->next; - return 0; + return &ref->old_oid; } static void update_remote_refs(const struct ref *refs, diff --git a/builtin/fetch.c b/builtin/fetch.c index 01513e6aea..cdf0d0d671 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -962,7 +962,7 @@ static int update_local_ref(struct ref *ref, } } -static int iterate_ref_map(void *cb_data, struct object_id *oid) +static const struct object_id *iterate_ref_map(void *cb_data) { struct ref **rm = cb_data; struct ref *ref = *rm; @@ -970,10 +970,9 @@ static int iterate_ref_map(void *cb_data, struct object_id *oid) while (ref && ref->status == REF_STATUS_REJECT_SHALLOW) ref = ref->next; if (!ref) - return -1; /* end of the list */ + return NULL; *rm = ref->next; - oidcpy(oid, &ref->old_oid); - return 0; + return &ref->old_oid; } struct fetch_head { diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index 2d1f97e1ca..041e915454 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -1306,7 +1306,7 @@ static void refuse_unconfigured_deny_delete_current(void) rp_error("%s", _(refuse_unconfigured_deny_delete_current_msg)); } -static int command_singleton_iterator(void *cb_data, struct object_id *oid); +static const struct object_id *command_singleton_iterator(void *cb_data); static int update_shallow_ref(struct command *cmd, struct shallow_info *si) { struct shallow_lock shallow_lock = SHALLOW_LOCK_INIT; @@ -1731,16 +1731,15 @@ static void check_aliased_updates(struct command *commands) string_list_clear(&ref_list, 0); } -static int command_singleton_iterator(void *cb_data, struct object_id *oid) +static const struct object_id *command_singleton_iterator(void *cb_data) { struct command **cmd_list = cb_data; struct command *cmd = *cmd_list; if (!cmd || is_null_oid(&cmd->new_oid)) - return -1; /* end of list */ + return NULL; *cmd_list = NULL; /* this returns only one */ - oidcpy(oid, &cmd->new_oid); - return 0; + return &cmd->new_oid; } static void set_connectivity_errors(struct command *commands, @@ -1770,7 +1769,7 @@ struct iterate_data { struct shallow_info *si; }; -static int iterate_receive_command_list(void *cb_data, struct object_id *oid) +static const struct object_id *iterate_receive_command_list(void *cb_data) { struct iterate_data *data = cb_data; struct command **cmd_list = &data->cmds; @@ -1781,13 +1780,11 @@ static int iterate_receive_command_list(void *cb_data, struct object_id *oid) /* to be checked in update_shallow_ref() */ continue; if (!is_null_oid(&cmd->new_oid) && !cmd->skip_update) { - oidcpy(oid, &cmd->new_oid); *cmd_list = cmd->next; - return 0; + return &cmd->new_oid; } } - *cmd_list = NULL; - return -1; /* end of list */ + return NULL; } static void reject_updates_to_hidden(struct command *commands) diff --git a/connected.c b/connected.c index b5f9523a5f..cf68e37a97 100644 --- a/connected.c +++ b/connected.c @@ -24,7 +24,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, struct child_process rev_list = CHILD_PROCESS_INIT; FILE *rev_list_in; struct check_connected_options defaults = CHECK_CONNECTED_INIT; - struct object_id oid; + const struct object_id *oid; int err = 0; struct packed_git *new_pack = NULL; struct transport *transport; @@ -34,7 +34,8 @@ int check_connected(oid_iterate_fn fn, void *cb_data, opt = &defaults; transport = opt->transport; - if (fn(cb_data, &oid)) { + oid = fn(cb_data); + if (!oid) { if (opt->err_fd) close(opt->err_fd); return err; @@ -73,7 +74,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, for (p = get_all_packs(the_repository); p; p = p->next) { if (!p->pack_promisor) continue; - if (find_pack_entry_one(oid.hash, p)) + if (find_pack_entry_one(oid->hash, p)) goto promisor_pack_found; } /* @@ -83,7 +84,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, goto no_promisor_pack_found; promisor_pack_found: ; - } while (!fn(cb_data, &oid)); + } while ((oid = fn(cb_data)) != NULL); return 0; } @@ -133,12 +134,12 @@ int check_connected(oid_iterate_fn fn, void *cb_data, * are sure the ref is good and not sending it to * rev-list for verification. */ - if (new_pack && find_pack_entry_one(oid.hash, new_pack)) + if (new_pack && find_pack_entry_one(oid->hash, new_pack)) continue; - if (fprintf(rev_list_in, "%s\n", oid_to_hex(&oid)) < 0) + if (fprintf(rev_list_in, "%s\n", oid_to_hex(oid)) < 0) break; - } while (!fn(cb_data, &oid)); + } while ((oid = fn(cb_data)) != NULL); if (ferror(rev_list_in) || fflush(rev_list_in)) { if (errno != EPIPE && errno != EINVAL) diff --git a/connected.h b/connected.h index 8d5a6b3ad6..6e59c92aa3 100644 --- a/connected.h +++ b/connected.h @@ -9,7 +9,7 @@ struct transport; * When called after returning the name for the last object, return -1 * to signal EOF, otherwise return 0. */ -typedef int (*oid_iterate_fn)(void *, struct object_id *oid); +typedef const struct object_id *(*oid_iterate_fn)(void *); /* * Named-arguments struct for check_connected. All arguments are diff --git a/fetch-pack.c b/fetch-pack.c index 0bf7ed7e47..e6ec79f81a 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1912,16 +1912,15 @@ static void update_shallow(struct fetch_pack_args *args, oid_array_clear(&ref); } -static int iterate_ref_map(void *cb_data, struct object_id *oid) +static const struct object_id *iterate_ref_map(void *cb_data) { struct ref **rm = cb_data; struct ref *ref = *rm; if (!ref) - return -1; /* end of the list */ + return NULL; *rm = ref->next; - oidcpy(oid, &ref->old_oid); - return 0; + return &ref->old_oid; } struct ref *fetch_pack(struct fetch_pack_args *args, From patchwork Tue Aug 24 10:37:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12454603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BFEAC4338F for ; Tue, 24 Aug 2021 10:37:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6E36D61265 for ; Tue, 24 Aug 2021 10:37:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236243AbhHXKiB (ORCPT ); Tue, 24 Aug 2021 06:38:01 -0400 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:42105 "EHLO out4-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236232AbhHXKh4 (ORCPT ); Tue, 24 Aug 2021 06:37:56 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 3826C5C0079; Tue, 24 Aug 2021 06:37:12 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Tue, 24 Aug 2021 06:37:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=UWWOdz6xMcmICbFqoU/c4SXkFdY tjN6tbnvPKpberVs=; b=Bsn3zcUryceQ/H9vqpmZVR5FZHuZyImL7DpXvEPG0B5 4sX+/a37bttd/2zBYu51/1IKCb4XUh7HBKS/xD73dXwLHEfcBIPciJ74+hGeZS2x btYxmZVz+ZfK5QwrzOWHTjgYKDwSjW3rRHwd7wKxroZW6XdgYN+PPbpGzt4AYek/ RFI91To5IlvSewXSuhPcwsYjURcL7OjWhEGoFPKz22dgONawT7UrgV77iOPv7b1g +Z6mvJIQbWJ1U6+ZP6WU1SOnyJhaJoPwBGcqVfQsObSluieM6hJgAyEjg0XgB6ux 36hMlm9S8O8XG0nVcb5WB5UttSXtgToIEhI0792p10A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=UWWOdz 6xMcmICbFqoU/c4SXkFdYtjN6tbnvPKpberVs=; b=La0KxQ/pkVWrkHLcL+8/kk L1B0kvMVs085AHkUg5GFnzlgFKrnBDge/YEdJuMw2WlxmERFwqfSB8CPvcZcwNPm OG6GUEU3X/zZpnQo/l1rBBzzkXb6TwviP3vAB0BOCfhwYHuFwZQlaOX46kv1ncvd NuNzdiaMrOAdk6RmScf0E6xCOq1A5v5CvPb5DpWeBgrq7L9Ir7KOmRjnb6yb4n8w Uc43n8aIrOv6Tg4OWmDC62paPb3iAVInoPlrkfaxdiOsMl26gc2+ta/Qi92Y0PJG kOU58MiqKS5fFHWhcG/LTsFYysH4IQ1xvteRhomvCxoWu/955NdOoCPMxv7Qm5oA == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddtjedgfedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttdejnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgfejueevjeetudehgffffeffvdejfeejiedvkeffgfekuefgheevteeufeelkeen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Aug 2021 06:37:10 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 96b2a443 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 24 Aug 2021 10:37:10 +0000 (UTC) Date: Tue, 24 Aug 2021 12:37:09 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v2 4/7] fetch-pack: optimize loading of refs via commit graph Message-ID: <6311203f08219a8fa3884af5d288340a3fa5f47a.1629800774.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In order to negotiate a packfile, we need to dereference refs to see which commits we have in common with the remote. To do so, we first look up the object's type -- if it's a tag, we peel until we hit a non-tag object. If we hit a commit eventually, then we return that commit. In case the object ID points to a commit directly, we can avoid the initial lookup of the object type by opportunistically looking up the commit via the commit-graph, if available, which gives us a slight speed bump of about 2% in a huge repository with about 2.3M refs: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 31.634 s ± 0.258 s [User: 28.400 s, System: 5.090 s] Range (min … max): 31.280 s … 31.896 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 31.129 s ± 0.543 s [User: 27.976 s, System: 5.056 s] Range (min … max): 30.172 s … 31.479 s 5 runs Summary 'HEAD: git-fetch' ran 1.02 ± 0.02 times faster than 'HEAD~: git-fetch' In case this fails, we fall back to the old code which peels the objects to a commit. Signed-off-by: Patrick Steinhardt --- fetch-pack.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/fetch-pack.c b/fetch-pack.c index e6ec79f81a..dc800879cb 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -119,6 +119,11 @@ static struct commit *deref_without_lazy_fetch(const struct object_id *oid, { enum object_type type; struct object_info info = { .typep = &type }; + struct commit *commit; + + commit = lookup_commit_in_graph(the_repository, oid); + if (commit) + return commit; while (1) { if (oid_object_info_extended(the_repository, oid, &info, @@ -139,7 +144,7 @@ static struct commit *deref_without_lazy_fetch(const struct object_id *oid, } if (type == OBJ_COMMIT) { - struct commit *commit = lookup_commit(the_repository, oid); + commit = lookup_commit(the_repository, oid); if (!commit || repo_parse_commit(the_repository, commit)) return NULL; return commit; From patchwork Tue Aug 24 10:37:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12454605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B37BC4338F for ; Tue, 24 Aug 2021 10:37:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1377E61008 for ; Tue, 24 Aug 2021 10:37:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236342AbhHXKi0 (ORCPT ); Tue, 24 Aug 2021 06:38:26 -0400 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:52065 "EHLO out4-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236285AbhHXKiA (ORCPT ); Tue, 24 Aug 2021 06:38:00 -0400 Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 656BB5C013B; Tue, 24 Aug 2021 06:37:16 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Tue, 24 Aug 2021 06:37:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=ydwJQMST+meK/8kus6/KfBJhg1U n+26LX7CfoObwmbc=; b=UWaBOPIDJpUC8BggSOHnJ5TARL4VCvAjC2kPcqc2byS R/rBOfbAN0SZs2XLXTZnIRIVYd2rLwPn5oYy4tG1a3syRTPyDo1Ec4+TpvGhicP7 qiby0bW2DeNu9Jl9auffvTetLr8irvYtc86Lk1FYyvZgQYmczJhncXTOzHVTw53+ Lk84eq8oVSrZNOFZ9VAVuHBMBSl0atINNrQR5nB9meOpSaJZBVeabTMfDcR5pk3K GdC+o/mVIK5DrnFFVP4lbAQ/Rk8cUTEzfa2e2CBozgwUvdYocHkKbOw3t9gZJXsL uFNpgsPGNUYlp+GSFB+VBvAYsyy1GSgfKY+ZlYEIFvQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=ydwJQM ST+meK/8kus6/KfBJhg1Un+26LX7CfoObwmbc=; b=ILk+mvAMiJsFGZ0JgcY0iI At09Kby4cODHH8/3G1BuvLcnJXEewYjPkArw3onR3fVonudwb9I89FDv9fDzvSV8 x2k8qbjcA1pzNjD4wW8HewFQPcx7jVkB/jVZsu5YAIT+kwUSZ2UOG1lMRkxuOxQI Okolub+RtLU6Q84wKF45wdCje3Ny3d87SnpNCcqJ2iHFnngE90PV19zFqkm1NUrZ 6gdwk8yA3FUSiQDpLZTK32jfMFse/tQcHAAdYGPCPhoNB0rORWNbJpJxPpAtvcTL vApEAxu1Sh4fn4gi0N1WStjfVh4Eol/j4gTnekXEss9Pi6R6B03EyhkTZffIUFXw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddtjedgfedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgefhtdefueffheekgfffudelffejtdfhvdejkedthfehvdelgfetgfdvtedthfen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Aug 2021 06:37:15 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id dd71eb36 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 24 Aug 2021 10:37:14 +0000 (UTC) Date: Tue, 24 Aug 2021 12:37:13 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v2 5/7] fetch: refactor fetch refs to be more extendable Message-ID: <56a9158ac331f9911a4347d7d4afc2bbd2cf4d33.1629800774.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Refactor `fetch_refs()` code to make it more extendable by explicitly handling error cases. The refactored code should behave the same. Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index cdf0d0d671..da0e283288 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1293,20 +1293,28 @@ static int check_exist_and_connected(struct ref *ref_map) static int fetch_refs(struct transport *transport, struct ref *ref_map) { - int ret = check_exist_and_connected(ref_map); + int ret; + + /* + * We don't need to perform a fetch in case we can already satisfy all + * refs. + */ + ret = check_exist_and_connected(ref_map); if (ret) { trace2_region_enter("fetch", "fetch_refs", the_repository); ret = transport_fetch_refs(transport, ref_map); trace2_region_leave("fetch", "fetch_refs", the_repository); + if (ret) { + transport_unlock_pack(transport); + return ret; + } } - if (!ret) - /* - * Keep the new pack's ".keep" file around to allow the caller - * time to update refs to reference the new objects. - */ - return 0; - transport_unlock_pack(transport); - return ret; + + /* + * Keep the new pack's ".keep" file around to allow the caller + * time to update refs to reference the new objects. + */ + return 0; } /* Update local refs based on the ref values fetched from a remote */ From patchwork Tue Aug 24 10:37:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12454609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD968C432BE for ; Tue, 24 Aug 2021 10:37:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BA7BE61008 for ; Tue, 24 Aug 2021 10:37:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236265AbhHXKi1 (ORCPT ); Tue, 24 Aug 2021 06:38:27 -0400 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:48837 "EHLO out4-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236310AbhHXKiE (ORCPT ); Tue, 24 Aug 2021 06:38:04 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id A510F5C006F; Tue, 24 Aug 2021 06:37:20 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Tue, 24 Aug 2021 06:37:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=/TovEeBDFb13LAwZ91p0bt3lWv+ rkQWZXQq6biNwZTE=; b=GRnTv1Y+TRZiKo2saQYxcUua9wLP0Xq2aQcVTlmZXct cQh+8h8srPLZe2z4SxOFb2c5yaIGdBAiJLsAULSoGtmolGW6D+Qra/KKZ4/cLozM LIOFQBHJ9ymc1CikCrbA3zXX8yp3VtvVzeie4+3XW4A4iRufYCum6E/d8+yCqQxl kfd7Vuze0khCtFRjHywWjM9lzDmyZ+/yfVtMufk1fkF85zADT2UxnmvrcuKFg6vf euW8giROgUatOnqDIBSLqBrx+Y6bPmyOB/2/dM+2D/Jr6b1zdwPSEZFi/3HhS5SU 1ErgAnw7qIb2VECq3nfaDloBPgBf+NjhXrcQnMpHoVw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=/TovEe BDFb13LAwZ91p0bt3lWv+rkQWZXQq6biNwZTE=; b=izmk3jPgjwd0MG3Rl3McKB vjPmH3VueabQQ5EvIginP45Fjgzv4ivZ+moSaXMQgzGmEjIGYIKg0GugVHnbSaf3 QgKv90hwxG6+CjgGJPXFReKkpUmA40xdbmKTK/oKl5rMGFrISw0oBq2DMKvIWQlD 4125HaJ5SCZqL7AFSgV/3x0DW3ZwjieyTHFBq4dXhmmI5ZOdsfJ5xIqYUA4gR9UL viJGPt9WviDF0Bmol383+TruYQ3YZuluZT2HtOqB5K7b4Bzyp27b0qmZuYZM8yPv Rw5DTOwvLeslwjedvKyYmojUkLqUgXGrEx4Im7kfOnZ8RCghwn64MXdZx414R+BA == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddtjedgfedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgefhtdefueffheekgfffudelffejtdfhvdejkedthfehvdelgfetgfdvtedthfen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Aug 2021 06:37:19 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 37514157 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 24 Aug 2021 10:37:18 +0000 (UTC) Date: Tue, 24 Aug 2021 12:37:17 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v2 6/7] fetch: merge fetching and consuming refs Message-ID: <31d9f72edf5c178b2e80c30bb7c0a9bc164ca5eb.1629800774.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The functions `fetch_refs()` and `consume_refs()` must always be called together such that we first obtain all missing objects and then update our local refs to match the remote refs. In a subsequent patch, we'll further require that `fetch_refs()` must always be called before `consume_refs()` such that it can correctly assert that we have all objects after the fetch given that we're about to move the connectivity check. Make this requirement explicit by merging both functions into a single `fetch_and_consume_refs()` function. Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 32 +++++++++++--------------------- 1 file changed, 11 insertions(+), 21 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index da0e283288..a1e17edd8b 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1291,8 +1291,9 @@ static int check_exist_and_connected(struct ref *ref_map) return check_connected(iterate_ref_map, &rm, &opt); } -static int fetch_refs(struct transport *transport, struct ref *ref_map) +static int fetch_and_consume_refs(struct transport *transport, struct ref *ref_map) { + int connectivity_checked; int ret; /* @@ -1304,32 +1305,22 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map) trace2_region_enter("fetch", "fetch_refs", the_repository); ret = transport_fetch_refs(transport, ref_map); trace2_region_leave("fetch", "fetch_refs", the_repository); - if (ret) { - transport_unlock_pack(transport); - return ret; - } + if (ret) + goto out; } - /* - * Keep the new pack's ".keep" file around to allow the caller - * time to update refs to reference the new objects. - */ - return 0; -} - -/* Update local refs based on the ref values fetched from a remote */ -static int consume_refs(struct transport *transport, struct ref *ref_map) -{ - int connectivity_checked = transport->smart_options + connectivity_checked = transport->smart_options ? transport->smart_options->connectivity_checked : 0; - int ret; + trace2_region_enter("fetch", "consume_refs", the_repository); ret = store_updated_refs(transport->url, transport->remote->name, connectivity_checked, ref_map); - transport_unlock_pack(transport); trace2_region_leave("fetch", "consume_refs", the_repository); + +out: + transport_unlock_pack(transport); return ret; } @@ -1518,8 +1509,7 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map) transport_set_option(transport, TRANS_OPT_FOLLOWTAGS, NULL); transport_set_option(transport, TRANS_OPT_DEPTH, "0"); transport_set_option(transport, TRANS_OPT_DEEPEN_RELATIVE, NULL); - if (!fetch_refs(transport, ref_map)) - consume_refs(transport, ref_map); + fetch_and_consume_refs(transport, ref_map); if (gsecondary) { transport_disconnect(gsecondary); @@ -1610,7 +1600,7 @@ static int do_fetch(struct transport *transport, transport->url); } } - if (fetch_refs(transport, ref_map) || consume_refs(transport, ref_map)) { + if (fetch_and_consume_refs(transport, ref_map)) { free_refs(ref_map); retcode = 1; goto cleanup; From patchwork Tue Aug 24 10:37:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12454607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DC00C4320A for ; Tue, 24 Aug 2021 10:37:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 20EB761265 for ; Tue, 24 Aug 2021 10:37:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236277AbhHXKi0 (ORCPT ); Tue, 24 Aug 2021 06:38:26 -0400 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:36049 "EHLO out4-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236330AbhHXKiJ (ORCPT ); Tue, 24 Aug 2021 06:38:09 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id E0F7F5C0079; Tue, 24 Aug 2021 06:37:24 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Tue, 24 Aug 2021 06:37:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=k18Br12CNucwD7YofIHElFZk1j0 GF96LxxbqLusIaug=; b=GRvVWx+1B+puwpbsrwW5v1fi3e+g8Q68umk6hxgc7oz sHzdrQqRa8cmnbgcTOMWGJvvy93Qn421WD348uBurG5X4QtRfUCXjtPPZEgwKBRp c/lpoDCQw/yl3QWEbfCdpZGuWOCzS47EYbBf2L8QAUto8BExKtTmBunbGt4BsxjG AiqHutmjThQ20l1amoalHMbp8mknPeJe+mgLjugCVz4ySty/qDe6OnWQ2cSsvhGG 6z5UjUspIJPIwWFwZJGqCeNWq6x2jfjb1OwsCHvVJd1p5FMNqD9/WJFelq/Ji0fC LiE1uuuxFj/oR62QXuQ7ks165y3Wd0eVLauGdicQmfA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=k18Br1 2CNucwD7YofIHElFZk1j0GF96LxxbqLusIaug=; b=eZrZqxoKx+9wDl1RCr0Tzq N8u7Sxsm7P08qY+uwN7diuMKcgIF52Mwk1+PHYwBVr0mLtL2vAnkjOtVSmUAJkn/ N4R1IpNH4Spa7kTw4NBOx1T4ikHtmUdAcg5F5r9/ziN+HBFTN/1kHF3EQo6nDjsF I7zZhXCOnf4BEV2KNzq94mt2Oj3oeiacByd7yEdbCaZ3D/Nq/pfc9eyljsh5yG+N wsXkoqwyQJW+Lxjqs1LHgxIdxJjHnUElkKzdcaNJ1hsNMeQW3+OOoEy22uz6RTYt dZn0JFgBlAOcvDcBXpQNCKwRSjAcyet29WlQnZ2+Ziu4L//3BB8oWKd2LAzYiwNw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddtjedgfedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttdejnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgfejueevjeetudehgffffeffvdejfeejiedvkeffgfekuefgheevteeufeelkeen ucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 24 Aug 2021 06:37:23 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id c27ad34c (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 24 Aug 2021 10:37:22 +0000 (UTC) Date: Tue, 24 Aug 2021 12:37:21 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v2 7/7] fetch: avoid second connectivity check if we already have all objects Message-ID: <84e39c847f8258f0a0b392a2752dab00237c9e8f.1629800774.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When fetching refs, we are doing two connectivity checks: - The first one is done such that we can skip fetching refs in the case where we already have all objects referenced by the updated set of refs. - The second one verifies that we have all objects after we have fetched objects. We always execute both connectivity checks, but this is wasteful in case the first connectivity check already notices that we have all objects locally available. Skip the second connectivity check in case we already had all objects available. This gives us a nice speedup when doing a mirror-fetch in a repository with about 2.3M refs where the fetching repo already has all objects: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 30.025 s ± 0.081 s [User: 27.070 s, System: 4.933 s] Range (min … max): 29.900 s … 30.111 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 25.574 s ± 0.177 s [User: 22.855 s, System: 4.683 s] Range (min … max): 25.399 s … 25.765 s 5 runs Summary 'HEAD: git-fetch' ran 1.17 ± 0.01 times faster than 'HEAD~: git-fetch' Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index a1e17edd8b..e2c952ec67 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1293,7 +1293,7 @@ static int check_exist_and_connected(struct ref *ref_map) static int fetch_and_consume_refs(struct transport *transport, struct ref *ref_map) { - int connectivity_checked; + int connectivity_checked = 1; int ret; /* @@ -1307,11 +1307,10 @@ static int fetch_and_consume_refs(struct transport *transport, struct ref *ref_m trace2_region_leave("fetch", "fetch_refs", the_repository); if (ret) goto out; + connectivity_checked = transport->smart_options ? + transport->smart_options->connectivity_checked : 0; } - connectivity_checked = transport->smart_options - ? transport->smart_options->connectivity_checked : 0; - trace2_region_enter("fetch", "consume_refs", the_repository); ret = store_updated_refs(transport->url, transport->remote->name,