From patchwork Fri Aug 20 10:08:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12448815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B1ECC4338F for ; Fri, 20 Aug 2021 10:08:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 65861610D2 for ; Fri, 20 Aug 2021 10:08:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237446AbhHTKJR (ORCPT ); Fri, 20 Aug 2021 06:09:17 -0400 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:38183 "EHLO out3-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237267AbhHTKJN (ORCPT ); Fri, 20 Aug 2021 06:09:13 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 204555C0160; Fri, 20 Aug 2021 06:08:35 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Fri, 20 Aug 2021 06:08:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm1; bh=UHLSZZnC1WCALUjKpySHxFjKozS 5TnlJWoHn7EgNQF4=; b=G6ojw3pPvG22PcGYzAdD+TlsYxqTbrp+Ka+63QTK+0s 8Fr9HFUrHBZmNcgUlULNcZlcg349fe3Fk8gFTMHiRPUK5yyXGUllBd13Y1DnBpjO 6FdIBhdFIiqL/lCB8nx2CnMli5XQbpk5A2Ef50FrAvl2NswSw+V4/qMwianLS9WX RQ99QNrqHwRqrFb5y9mliYH2Ue3wmx3X2TnnLWR+Nxm6jaDPHW1S6dOuhtS/+ABV ia9+0f8ACLXVw4ph21e8zIbLssEK/8SiV/49WeapEV9aYwgKzsgzVzNdPuQZ6H3H 6DoyRIYCFujDD1fJsySYBcNO4dkd6cH+1deOLto4cVQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=UHLSZZ nC1WCALUjKpySHxFjKozS5TnlJWoHn7EgNQF4=; b=in/QuW3Nh9rgh12sZTYsti S21IHg/zOXX515ssysW5EZ2AfOMWsWamlRYtXlkq9u7eADm70nbIHg7EZF4PLKFN efS+Dj5LI2+4IO0NPMKRkND7EskULetqA2RZhxmOJjchPuwJ+HykDFQDILjW944r eSgKVXeEq2bG/2JNqdaXKlvXbjJM8DtQQ3msQptJP0AWscK2uC5oA3gYq9Ycnr1/ hSQd7JsBelTKt5qdXIO05s1p8Io9UdJHbS4ETNmS8Mjhu9FeXtnk6VVOOr2Oaore tk1gM3J2FzGQzb7pcMhyq3UXMe4fchocGDsq0QOCa0Z5IcwB4TVhUGO7lN8+pWHw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrleelgddvvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtderre dttdejnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhk shdrihhmqeenucggtffrrghtthgvrhhnpeehgfejueevjeetudehgffffeffvdejfeejie dvkeffgfekuefgheevteeufeelkeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 20 Aug 2021 06:08:34 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 6180f4b1 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 20 Aug 2021 10:08:29 +0000 (UTC) Date: Fri, 20 Aug 2021 12:08:28 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King , =?iso-8859-1?q?=C6var_Arnfj=F6r=F0?= Bjarmason , Junio C Hamano Subject: [PATCH 1/6] fetch: speed up lookup of want refs via commit-graph Message-ID: <6872979c4557204821d788dc3f5e1c8bef0a773c.1629452412.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When updating our local refs based on the refs fetched from the remote, we need to iterate through all requested refs and load their respective commits such that we can determine whether they need to be appended to FETCH_HEAD or not. In cases where we're fetching from a remote with exceedingly many refs, resolving these refs can be quite expensive given that we repeatedly need to unpack object headers for each of the referenced objects. Speed this up by opportunistcally trying to resolve object IDs via the commit graph: more likely than not, they're going to be a commit anyway, and this lets us avoid having to unpack object headers completely in case the object is a commit that is part of the commit-graph. This significantly speeds up mirror-fetches in a real-world repository with 2.3M refs: Benchmark #1: HEAD~: git-fetch Time (mean ± σ): 56.942 s ± 0.449 s [User: 53.360 s, System: 5.356 s] Range (min … max): 56.372 s … 57.533 s 5 runs Benchmark #2: HEAD: git-fetch Time (mean ± σ): 33.657 s ± 0.167 s [User: 30.302 s, System: 5.181 s] Range (min … max): 33.454 s … 33.844 s 5 runs Summary 'HEAD: git-fetch' ran 1.69 ± 0.02 times faster than 'HEAD~: git-fetch' Signed-off-by: Patrick Steinhardt --- builtin/fetch.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 405afe9bdf..73f5b286d5 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1131,11 +1131,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, continue; } - commit = lookup_commit_reference_gently(the_repository, - &rm->old_oid, - 1); - if (!commit) - rm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE; + commit = lookup_commit_in_graph(the_repository, &rm->old_oid); + if (!commit) { + commit = lookup_commit_reference_gently(the_repository, + &rm->old_oid, + 1); + if (!commit) + rm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE; + } if (rm->fetch_head_status != want_status) continue;