From patchwork Wed Sep 1 13:09:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12469343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26566C4320E for ; Wed, 1 Sep 2021 13:09:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0AECD60231 for ; Wed, 1 Sep 2021 13:09:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344262AbhIANKj (ORCPT ); Wed, 1 Sep 2021 09:10:39 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:41391 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244182AbhIANKi (ORCPT ); Wed, 1 Sep 2021 09:10:38 -0400 Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id E12F35C0200; Wed, 1 Sep 2021 09:09:40 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Wed, 01 Sep 2021 09:09:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=/1RpOEwJMrWz1nb/3Eg9Z84/a4I iIz+XEbkRA4d+bMg=; b=czcMyoa9L28fciqBG8uYLwNGfbeCQd7PkSAe2RApgJM fIdJE4XgdATv0fmx0tOKMyPdFakZc+j1PRCCoMYimDzNMvCt72OrySN8qBXhIAGf dYEg8eScnWQ7sm50eJYuXvNI8Mr8xHN/y48jj0RG4OlGAiPV4ipadrn4WWruNGka XPMwv2veipWps6RM2zVEbesxxg9LZmTPIrkimJFu1ddMkqQUEYV6sD93Bj+9Ga93 1KwGYFP9KxuwmXj2vEu8YZ08JObeYKY6z6ecQJm1fV6i11RXnU0SyVJiOr/q2xZw 1ehwrzsSiQogw1qNokvIKuRQtjqpVq1qHdKTSRCqEAw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=/1RpOE wJMrWz1nb/3Eg9Z84/a4IiIz+XEbkRA4d+bMg=; b=Nx0rp0eF3foSyNXSMM/9k0 gbgp6utxmFWB8hnBrA72JtTjMRQRZbl6QZU+K32WfnY6qouyecums8Ub+C/RHY/4 59HeAuc5Wh1At590bOnidm6+6qJaSs7/0Cj96djWWPJ5DX0SACGC32OD9W1Bw+e6 F74GmhrPHNhGcE1byU1pBFgkYkvjdhL2LdTnQPdF3vn+ufC8JM2cU8mVDqBNlydN WRb2Kw1WK7phWyMC94jAvv9MP9CvEicIYQfZSxZ4sGflrbNUTC/r8rpC3za6shWr LPvGPsSD2A5v20Mv0QZRppAwzWhmESKUvsqff0IeJCXkW1FaeV4TVqUZRNvXJEZQ == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddruddvfedgieduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgefhtdefueffheekgfffudelffejtdfhvdejkedthfehvdelgfetgfdvtedthfen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Sep 2021 09:09:38 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id d65b90a2 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 1 Sep 2021 13:09:37 +0000 (UTC) Date: Wed, 1 Sep 2021 15:09:36 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano , Derrick Stolee , =?iso-8859-1?q?Ren=E9?= Scharfe Subject: [PATCH v3 0/7] Speed up mirror-fetches with many refs Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Hi, this is the third version of my patch series to speed up mirror-fetches with many refs. This patch series applies on top of master with ps/connectivity-optim merged into it. There's only some smallish changes based on Stolee's feedback (thanks for that!): - A small typo in 1/7. - A confict fix in 4/7 required now because it's based on master instead of directly on my merged topic. - I've adjusted patch 5/7 such that I don't have to re-touch the logic in 6/7. Patrick Patrick Steinhardt (7): fetch: speed up lookup of want refs via commit-graph fetch: avoid unpacking headers in object existence check connected: refactor iterator to return next object ID directly fetch-pack: optimize loading of refs via commit graph fetch: refactor fetch refs to be more extendable fetch: merge fetching and consuming refs fetch: avoid second connectivity check if we already have all objects builtin/clone.c | 8 ++--- builtin/fetch.c | 74 +++++++++++++++++++++++------------------- builtin/receive-pack.c | 17 ++++------ connected.c | 15 +++++---- connected.h | 2 +- fetch-pack.c | 12 ++++--- 6 files changed, 67 insertions(+), 61 deletions(-) Range-diff against v2: 1: 4a819a6830 ! 1: 8214f04971 fetch: speed up lookup of want refs via commit-graph @@ Commit message that we repeatedly need to unpack object headers for each of the referenced objects. - Speed this up by opportunistcally trying to resolve object IDs via the + Speed this up by opportunistically trying to resolve object IDs via the commit graph. We only do so for any refs which are not in "refs/tags": more likely than not, these are going to be a commit anyway, and this lets us avoid having to unpack object headers completely in case the 2: 81ebadabe8 = 2: 991a27cb82 fetch: avoid unpacking headers in object existence check 3: 98e981ced9 = 3: ba834803ab connected: refactor iterator to return next object ID directly 4: 6311203f08 ! 4: 99d3316d48 fetch-pack: optimize loading of refs via commit graph @@ fetch-pack.c: static struct commit *deref_without_lazy_fetch(const struct object while (1) { if (oid_object_info_extended(the_repository, oid, &info, -@@ fetch-pack.c: static struct commit *deref_without_lazy_fetch(const struct object_id *oid, - } - - if (type == OBJ_COMMIT) { -- struct commit *commit = lookup_commit(the_repository, oid); -+ commit = lookup_commit(the_repository, oid); - if (!commit || repo_parse_commit(the_repository, commit)) - return NULL; - return commit; 5: 56a9158ac3 ! 5: d64888e072 fetch: refactor fetch refs to be more extendable @@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map) trace2_region_enter("fetch", "fetch_refs", the_repository); ret = transport_fetch_refs(transport, ref_map); trace2_region_leave("fetch", "fetch_refs", the_repository); -+ if (ret) { -+ transport_unlock_pack(transport); -+ return ret; -+ } ++ if (ret) ++ goto out; } - if (!ret) - /* @@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map) - * time to update refs to reference the new objects. - */ - return 0; -- transport_unlock_pack(transport); -- return ret; + + /* + * Keep the new pack's ".keep" file around to allow the caller + * time to update refs to reference the new objects. + */ -+ return 0; ++ return ret; ++ ++out: + transport_unlock_pack(transport); + return ret; } - - /* Update local refs based on the ref values fetched from a remote */ 6: 31d9f72edf ! 6: 56ecbfc9c3 fetch: merge fetching and consuming refs @@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map) /* @@ builtin/fetch.c: static int fetch_refs(struct transport *transport, struct ref *ref_map) - trace2_region_enter("fetch", "fetch_refs", the_repository); - ret = transport_fetch_refs(transport, ref_map); - trace2_region_leave("fetch", "fetch_refs", the_repository); -- if (ret) { -- transport_unlock_pack(transport); -- return ret; -- } -+ if (ret) -+ goto out; + goto out; } - /* - * Keep the new pack's ".keep" file around to allow the caller - * time to update refs to reference the new objects. - */ -- return 0; +- return ret; +- +-out: +- transport_unlock_pack(transport); +- return ret; -} - -/* Update local refs based on the ref values fetched from a remote */ 7: 84e39c847f = 7: c342fc0c69 fetch: avoid second connectivity check if we already have all objects