From patchwork Wed Feb 23 12:35:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12756872 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A694BC433EF for ; Wed, 23 Feb 2022 12:35:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240488AbiBWMgT (ORCPT ); Wed, 23 Feb 2022 07:36:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240491AbiBWMgQ (ORCPT ); Wed, 23 Feb 2022 07:36:16 -0500 Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com [64.147.123.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6CB0FA27B1 for ; Wed, 23 Feb 2022 04:35:46 -0800 (PST) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id E90D6320187F for ; Wed, 23 Feb 2022 07:35:45 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Wed, 23 Feb 2022 07:35:46 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm3; bh=EOnloiPcWIzchY5d3Qqpa8v/o0Iw8YS5SpJsLi NtTL4=; b=AOyB6P5eN/Gq/verw8TIyftJAmlkku2YLVbQHgZQgywDEDvMt2TPlb cgyy5bp++mpWUddSmuxZTTdcejNRUVW7Xd7+5+yT4BN/1NEbzM4gw6uOhdS8GuuW Y5I2DvWkRhc8IJwXsNZuB9h6gE9It95HQMwFidmi2Y+tBz9FTbk7tkW9xJ2pBx8y 36w+fihUYpGSmODNeEgkJpj+4JB4dm3zC9hIOOlDhRxgzpj4GfuEzZnlx8TjGdZX qgU6AURTbBCEQdZmd1npmC8PumNJCZoHjbAUnvegdvT5OiRc6IFSFEctKpigCl01 eFdGDjvZMaJu0QRCEvdEu5zhw4oZtZ8g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=EOnloiPcWIzchY5d3 Qqpa8v/o0Iw8YS5SpJsLiNtTL4=; b=IIre5g0WNO4E5VoRHEftkPHtzQue+QKtB LjLG0ocVW1BOTm8jt+RSflE2gLfHhrC8pvnCUicaF8/WvHYszNB6iKJdgwoRW9c1 3hBBe+SGqK7Qj+wPKY2KvaYIkOrviQNM+qqqvueQ25yDhEn10uizu7ZDvGN3Clxd kTp0IPA61AetjfizLEYoRJ2MxisS99+gXx3uOY1hzjSM5ZGv423kFeexE72YBulK kHcE2MPznWk9QbAhc6gAWXQvuw5bNgJczb3VtpBhydiEzYzChKELe/sWXQ91hH4+ OtFOIFv3mgZekuGFzWChFi5MMz1EiYouGwwKoh1gWTyZJGxEjyZlQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrledtgdegtdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtderre dttdejnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhk shdrihhmqeenucggtffrrghtthgvrhhnpeehgfejueevjeetudehgffffeffvdejfeejie dvkeffgfekuefgheevteeufeelkeenucevlhhushhtvghrufhiiigvpedunecurfgrrhgr mhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 23 Feb 2022 07:35:44 -0500 (EST) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 8da4dcec (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Wed, 23 Feb 2022 12:35:44 +0000 (UTC) Date: Wed, 23 Feb 2022 13:35:43 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 5/5] refs/files-backend: optimize reading of symbolic refs Message-ID: <7213ffdbddc7b14c8665974e41bda9449f66de4a.1645619224.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When reading references via `files_read_raw_ref()` we always consult both the loose reference, and if that wasn't found, we also consult the packed-refs file. While this makes sense to read a generic reference, it is wasteful in the case where we only care about symbolic references: the packed-refs backend does not support them, and thus it cannot ever return one for us. Special-case reading of symbolic references for the files backend such that we always skip asking the packed-refs backend. We use `refs_read_symbolic_ref()` extensively to determine whether we need to skip updating local symbolic references during a fetch, which is why the change results in a significant speedup when doing fetches in repositories with huge numbers of references. The following benchmark executes a mirror-fetch in a repository with about 2 million references: Benchmark 1: git fetch --prune --no-write-fetch-head +refs/*:refs/* (HEAD~) Time (mean ± σ): 68.372 s ± 2.344 s [User: 65.629 s, System: 8.786 s] Range (min … max): 65.745 s … 70.246 s 3 runs Benchmark 2: git fetch --prune --no-write-fetch-head +refs/*:refs/* (HEAD) Time (mean ± σ): 60.259 s ± 0.343 s [User: 61.019 s, System: 7.245 s] Range (min … max): 60.003 s … 60.649 s 3 runs Summary 'git fetch --prune --no-write-fetch-head +refs/*:refs/* (HEAD)' ran 1.13 ± 0.04 times faster than 'git fetch --prune --no-write-fetch-head +refs/*:refs/* (HEAD~)' Signed-off-by: Patrick Steinhardt --- refs/files-backend.c | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/refs/files-backend.c b/refs/files-backend.c index f3428a9f12..0457ecdb42 100644 --- a/refs/files-backend.c +++ b/refs/files-backend.c @@ -338,9 +338,9 @@ static struct ref_cache *get_loose_ref_cache(struct files_ref_store *refs) return refs->loose; } -static int files_read_raw_ref(struct ref_store *ref_store, const char *refname, - struct object_id *oid, struct strbuf *referent, - unsigned int *type, int *failure_errno) +static int read_ref_internal(struct ref_store *ref_store, const char *refname, + struct object_id *oid, struct strbuf *referent, + unsigned int *type, int *failure_errno, int skip_packed_refs) { struct files_ref_store *refs = files_downcast(ref_store, REF_STORE_READ, "read_raw_ref"); @@ -381,7 +381,7 @@ static int files_read_raw_ref(struct ref_store *ref_store, const char *refname, if (lstat(path, &st) < 0) { int ignore_errno; myerr = errno; - if (myerr != ENOENT) + if (myerr != ENOENT || skip_packed_refs) goto out; if (refs_read_raw_ref(refs->packed_ref_store, refname, oid, referent, type, &ignore_errno)) { @@ -425,7 +425,8 @@ static int files_read_raw_ref(struct ref_store *ref_store, const char *refname, * ref is supposed to be, there could still be a * packed ref: */ - if (refs_read_raw_ref(refs->packed_ref_store, refname, oid, + if (skip_packed_refs || + refs_read_raw_ref(refs->packed_ref_store, refname, oid, referent, type, &ignore_errno)) { myerr = EISDIR; goto out; @@ -470,6 +471,27 @@ static int files_read_raw_ref(struct ref_store *ref_store, const char *refname, return ret; } +static int files_read_raw_ref(struct ref_store *ref_store, const char *refname, + struct object_id *oid, struct strbuf *referent, + unsigned int *type, int *failure_errno) +{ + return read_ref_internal(ref_store, refname, oid, referent, type, failure_errno, 0); +} + +static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refname, + struct strbuf *referent) +{ + struct object_id oid; + int failure_errno, ret; + unsigned int type; + + ret = read_ref_internal(ref_store, refname, &oid, referent, &type, &failure_errno, 1); + if (ret) + return ret; + + return !(type & REF_ISSYMREF); +} + int parse_loose_ref_contents(const char *buf, struct object_id *oid, struct strbuf *referent, unsigned int *type, int *failure_errno) @@ -3286,7 +3308,7 @@ struct ref_storage_be refs_be_files = { files_ref_iterator_begin, files_read_raw_ref, - NULL, + files_read_symbolic_ref, files_reflog_iterator_begin, files_for_each_reflog_ent,