From patchwork Tue Mar 1 09:33:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12764398 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCFC0C433F5 for ; Tue, 1 Mar 2022 09:34:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232895AbiCAJel (ORCPT ); Tue, 1 Mar 2022 04:34:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52958 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232098AbiCAJel (ORCPT ); Tue, 1 Mar 2022 04:34:41 -0500 Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E42DF5F4D8 for ; Tue, 1 Mar 2022 01:33:45 -0800 (PST) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 6EEA95C0163; Tue, 1 Mar 2022 04:33:40 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Tue, 01 Mar 2022 04:33:40 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm3; bh=kOuViQxE6N2JUrM7hk0z5TXZTTF5+5+eGzXYGg EeoQo=; b=kx1hpmcXfiId8qiBhPgfDFhPKUltXXDJMSBGKRebLSM0imVU1oAPs8 f5Z7kc2NOFSAphPMNYWFUJkfg7gypmVCeY3WVISRiRvZLh7W1KAqjH/O8m6lCD2L uEQCpArvn3n2Y5q7LxkKRzrAvvWqmwNSLxAQ5NL7bnSXwpNSEVDD8QmGDSlI6g4r iKZeJiUDZBR1ctTzzKrc3zMXLVC/KPLip7SvIgWBp33Xl3FYjrc0UTghajQ54g35 lTU5o291EouA4Qn2uASWTjyhfZdgfo2a5MQMGY0g1FHsFS6BG4XX9As/18tb1xxp o1r9L6qr3GpOz+KaqehZo0stJfijclpg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=kOuViQxE6N2JUrM7h k0z5TXZTTF5+5+eGzXYGgEeoQo=; b=edGMequtnzSTFfhoeh1ja4z2jyhHVcvpC 8dYZaMvD6FCsnpcxZz2iZBO9vXZJHf8fufP5xY9tog7PgAZBbHcW57D4/Y2qQbO5 81nKjRYZPbemfxM+xGVuaZL3tVfb5z9SZyp50LGNv48wcn4aS1Xf5zGy7rKrzHWu gAwnOfCgFVfaquP5s2qMqECt9s29MxkyVDXnOi8NdU0SsTbSZKQwLxA2q2YHvgf9 Aml82lsaXUiuCWCzM5Fn4TvpybWraYjK50FXtfr+UPQEAwJQGrcVwtmOKDparVsO UttocvquPH1cZ9FeMXS8+9pPeFvtw8nJht+5tkU4WoeBdSKEibucQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddruddtvddgtdefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttdejnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgfejueevjeetudehgffffeffvdejfeejiedvkeffgfekuefgheevteeufeelkeen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 1 Mar 2022 04:33:39 -0500 (EST) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id f60d94ff (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 1 Mar 2022 09:33:39 +0000 (UTC) Date: Tue, 1 Mar 2022 10:33:37 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH v2 1/5] upload-pack: look up "want" lines via commit-graph Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org During packfile negotiation the client will send "want" and "want-ref" lines to the server to tell it which objects it is interested in. The server-side parses each of those and looks them up to see whether it actually has requested objects. This lookup is performed by calling `parse_object()` directly, which thus hits the object database. In the general case though most of the objects the client requests will be commits. We can thus try to look up the object via the commit-graph opportunistically, which is much faster than doing the same via the object database. Refactor parsing of both "want" and "want-ref" lines to do so. The following benchmark is executed in a repository with a huge number of references. It uses cached request from git-fetch(1) as input to git-upload-pack(1) that contains about 876,000 "want" lines: Benchmark 1: HEAD~ Time (mean ± σ): 7.113 s ± 0.028 s [User: 6.900 s, System: 0.662 s] Range (min … max): 7.072 s … 7.168 s 10 runs Benchmark 2: HEAD Time (mean ± σ): 6.622 s ± 0.061 s [User: 6.452 s, System: 0.650 s] Range (min … max): 6.535 s … 6.727 s 10 runs Summary 'HEAD' ran 1.07 ± 0.01 times faster than 'HEAD~' Signed-off-by: Patrick Steinhardt --- upload-pack.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/upload-pack.c b/upload-pack.c index 8acc98741b..3a851b3606 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -1400,13 +1400,19 @@ static int parse_want(struct packet_writer *writer, const char *line, const char *arg; if (skip_prefix(line, "want ", &arg)) { struct object_id oid; + struct commit *commit; struct object *o; if (get_oid_hex(arg, &oid)) die("git upload-pack: protocol error, " "expected to get oid, not '%s'", line); - o = parse_object(the_repository, &oid); + commit = lookup_commit_in_graph(the_repository, &oid); + if (commit) + o = &commit->object; + else + o = parse_object(the_repository, &oid); + if (!o) { packet_writer_error(writer, "upload-pack: not our ref %s", @@ -1434,7 +1440,7 @@ static int parse_want_ref(struct packet_writer *writer, const char *line, if (skip_prefix(line, "want-ref ", &refname_nons)) { struct object_id oid; struct string_list_item *item; - struct object *o; + struct object *o = NULL; struct strbuf refname = STRBUF_INIT; strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons); @@ -1448,7 +1454,15 @@ static int parse_want_ref(struct packet_writer *writer, const char *line, item = string_list_append(wanted_refs, refname_nons); item->util = oiddup(&oid); - o = parse_object_or_die(&oid, refname_nons); + if (!starts_with(refname_nons, "refs/tags/")) { + struct commit *commit = lookup_commit_in_graph(the_repository, &oid); + if (commit) + o = &commit->object; + } + + if (!o) + o = parse_object_or_die(&oid, refname_nons); + if (!(o->flags & WANTED)) { o->flags |= WANTED; add_object_array(o, NULL, want_obj);