From patchwork Wed May 19 19:13:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12268265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7D5AC433B4 for ; Wed, 19 May 2021 19:13:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8512261355 for ; Wed, 19 May 2021 19:13:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231862AbhESTOs (ORCPT ); Wed, 19 May 2021 15:14:48 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:55749 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231131AbhESTOq (ORCPT ); Wed, 19 May 2021 15:14:46 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 35AA35C01A4 for ; Wed, 19 May 2021 15:13:26 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Wed, 19 May 2021 15:13:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:subject:message-id:references:mime-version:content-type :in-reply-to; s=fm1; bh=WPwqV4cbdXpdl3f66TQVRtKa4kULWTJ9FHmjBq0R X2A=; b=ZwlQ18ci/7qyNPbLRpfNshWZ6U3D3rl9gTITjyOUJ+DQnZc0CiY0MJOB XBhua0NsWQO6po77k4IHTN2+4T35/7hv46XHBrEx7gj07OdV+iHySITKS3qbY5ta eCuzEfk30g0R4MCS8d3ZmD2xvjEJMIPUGjF2ULA1ZH/YF31h3Ex4Iv3gx0k28jFS Fuf0Vxg2cyESrNfCnPozUzpssg50qFO/VtYVLc6UY8sxsr3Hb1n+84mFjQL23KBY r1EnjtYIP7/KxoAD6uF+4795caTQBsoDTpWKLJVvF9BpqtcfynHCDlSezo/zAhdB 6OfCx3CQqa6EGlIkVuwaTz4hs4kvQA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=WPwqV4 cbdXpdl3f66TQVRtKa4kULWTJ9FHmjBq0RX2A=; b=JULA/QwVatryDpB4BOiOhG G430lzZIKR6RsQHqv1RwnKPE+FDOrj3kmAU/xOPDeJTgvQD+o9jzD5gceys4leCZ xTvRbflQ1+olVlPkn5gsN7iIHzYOyfjEcBNAkpRSmrCpgheUsPAoWTK40omHUeAs azksnbYSM45YXOmLaYQLRuylscCY6hs6cX5whu+EqXwvMEiw0ofh5pwP3egX6CXd w4qqRV4HJoyO0y0gxjJ5cZGpu+zX5fJR9irpUyW1UsNc7Q25oun1a1GkptoTWHHR 5nnJE4wiAhoCltgMz4MW4bhcHMoKufPUaR6Y9Qfpi0GAMoru2N0TiJkiYYekSC4Q == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdeiledgudefudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucfkphepjeejrddukeefrdefjedrudehieen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: from vm-mail.pks.im (x4db7259c.dyn.telefonica.de [77.183.37.156]) by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 19 May 2021 15:13:25 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id eb83acad (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Wed, 19 May 2021 19:13:23 +0000 (UTC) Date: Wed, 19 May 2021 21:13:22 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 1/8] perf: fix when running with TEST_OUTPUT_DIRECTORY Message-ID: <5488f0f4888fbc46378e2777d7daba1ec627b6c6.1621451532.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When the TEST_OUTPUT_DIRECTORY is defined, then all test data will be written in that directory instead of the default directory located in "t/". While this works as expected for our normal tests, performance tests fail to locate and aggregate performance data because they don't know to handle TEST_OUTPUT_DIRECTORY correctly and always look at the default location. Fix the issue by adding a `--results-dir` parameter to "aggregate.perl" which identifies the directory where results are and by making the "run" script awake of the TEST_OUTPUT_DIRECTORY variable. Signed-off-by: Patrick Steinhardt --- t/perf/aggregate.perl | 8 ++++++-- t/perf/perf-lib.sh | 4 ++-- t/perf/run | 25 ++++++++++++++++--------- 3 files changed, 24 insertions(+), 13 deletions(-) diff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl index 14e4cda287..5d4964c5c6 100755 --- a/t/perf/aggregate.perl +++ b/t/perf/aggregate.perl @@ -58,6 +58,7 @@ sub usage { Options: --codespeed * Format output for Codespeed --reponame * Send given reponame to codespeed + --results-dir * Directory where test results are located --sort-by * Sort output (only "regression" criteria is supported) --subsection * Use results from given subsection @@ -90,12 +91,13 @@ sub sane_backticks { } my (@dirs, %dirnames, %dirabbrevs, %prefixes, @tests, - $codespeed, $sortby, $subsection, $reponame); + $codespeed, $sortby, $subsection, $reponame, $resultsdir); Getopt::Long::Configure qw/ require_order /; my $rc = GetOptions("codespeed" => \$codespeed, "reponame=s" => \$reponame, + "results-dir=s" => \$resultsdir, "sort-by=s" => \$sortby, "subsection=s" => \$subsection); usage() unless $rc; @@ -137,7 +139,9 @@ sub sane_backticks { @tests = glob "p????-*.sh"; } -my $resultsdir = "test-results"; +if (not $resultsdir) { + $resultsdir = "test-results"; +} if (! $subsection and exists $ENV{GIT_PERF_SUBSECTION} and diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh index 601d9f67dd..8ca6dd0297 100644 --- a/t/perf/perf-lib.sh +++ b/t/perf/perf-lib.sh @@ -45,7 +45,7 @@ export TEST_DIRECTORY TRASH_DIRECTORY GIT_BUILD_DIR GIT_TEST_CMP MODERN_GIT=$GIT_BUILD_DIR/bin-wrappers/git export MODERN_GIT -perf_results_dir=$TEST_OUTPUT_DIRECTORY/test-results +perf_results_dir=$TEST_RESULTS_DIR test -n "$GIT_PERF_SUBSECTION" && perf_results_dir="$perf_results_dir/$GIT_PERF_SUBSECTION" mkdir -p "$perf_results_dir" rm -f "$perf_results_dir"/$(basename "$0" .sh).subtests @@ -253,7 +253,7 @@ test_size () { # and does it after running everything) test_at_end_hook_ () { if test -z "$GIT_PERF_AGGREGATING_LATER"; then - ( cd "$TEST_DIRECTORY"/perf && ./aggregate.perl $(basename "$0") ) + ( cd "$TEST_DIRECTORY"/perf && ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $(basename "$0") ) fi } diff --git a/t/perf/run b/t/perf/run index c7b86104e1..03128d440a 100755 --- a/t/perf/run +++ b/t/perf/run @@ -188,10 +188,10 @@ run_subsection () { if test -z "$GIT_PERF_SEND_TO_CODESPEED" then - ./aggregate.perl $codespeed_opt "$@" + ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" $codespeed_opt "$@" else - json_res_file="test-results/$GIT_PERF_SUBSECTION/aggregate.json" - ./aggregate.perl --codespeed "$@" | tee "$json_res_file" + json_res_file=""$TEST_RESULTS_DIR"/$GIT_PERF_SUBSECTION/aggregate.json" + ./aggregate.perl --results-dir="$TEST_RESULTS_DIR" --codespeed "$@" | tee "$json_res_file" send_data_url="$GIT_PERF_SEND_TO_CODESPEED/result/add/json/" curl -v --request POST --data-urlencode "json=$(cat "$json_res_file")" "$send_data_url" fi @@ -203,10 +203,17 @@ get_var_from_env_or_config "GIT_PERF_SEND_TO_CODESPEED" "perf" "sendToCodespeed" cd "$(dirname $0)" . ../../GIT-BUILD-OPTIONS -mkdir -p test-results -get_subsections "perf" >test-results/run_subsections.names +if test -n "$TEST_OUTPUT_DIRECTORY" +then + TEST_RESULTS_DIR="$TEST_OUTPUT_DIRECTORY/test-results" +else + TEST_RESULTS_DIR=test-results +fi -if test $(wc -l "$TEST_RESULTS_DIR"/run_subsections.names + +if test $(wc -l <"$TEST_RESULTS_DIR"/run_subsections.names) -eq 0 then if test -n "$GIT_PERF_SUBSECTION" then @@ -222,10 +229,10 @@ then ) elif test -n "$GIT_PERF_SUBSECTION" then - egrep "^$GIT_PERF_SUBSECTION\$" test-results/run_subsections.names >/dev/null || + egrep "^$GIT_PERF_SUBSECTION\$" "$TEST_RESULTS_DIR"/run_subsections.names >/dev/null || die "subsection '$GIT_PERF_SUBSECTION' not found in '$GIT_PERF_CONFIG_FILE'" - egrep "^$GIT_PERF_SUBSECTION\$" test-results/run_subsections.names | while read -r subsec + egrep "^$GIT_PERF_SUBSECTION\$" "$TEST_RESULTS_DIR"/run_subsections.names | while read -r subsec do ( GIT_PERF_SUBSECTION="$subsec" @@ -243,5 +250,5 @@ else echo "======== Run for subsection '$GIT_PERF_SUBSECTION' ========" run_subsection "$@" ) - done X-Patchwork-Id: 12268267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FA5CC433ED for ; Wed, 19 May 2021 19:13:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2EE736108D for ; Wed, 19 May 2021 19:13:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231926AbhESTO6 (ORCPT ); Wed, 19 May 2021 15:14:58 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:37411 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229979AbhESTOw (ORCPT ); Wed, 19 May 2021 15:14:52 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 59BE25C0105 for ; Wed, 19 May 2021 15:13:32 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Wed, 19 May 2021 15:13:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:subject:message-id:references:mime-version:content-type :in-reply-to; s=fm1; bh=OM+CcW43VMgIM/sbQSc191cpo6QdgLsdz6cAA1l/ vYU=; b=YzNA6LiIIlmR7Qbo8Acj8Uoihf6eVkPhzG75fdlkn6rjfmzRL5m5K9Q0 Btd1lyifRxpAzS/1aGUyDvuukqTJKzX5tdW30vYlLmq8yawdQc9kzKPVBE3DX7Fs m3qZ92Y0DChF+qJp9d9JARcUmYfitHYcJsHa8eIEOCNu9uU6lcq+SRFNe6UfkcOg NDjndrrM4ixFyrKtZeNZN3V6m7E/4Q7k6YmLrmWno5SybySqW7BmZeSxzmkq3Wvp SJPSWALenZczIRmWiWomsPXC0yhxgVF9/YZAcajpBPje7L7VMdLSjym5fytl8fct L2uAw56iDqPb7QTUuzP9rHlXQNMjFw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=OM+CcW 43VMgIM/sbQSc191cpo6QdgLsdz6cAA1l/vYU=; b=CNqK+vnhcOdR6F+f77L6RY /6JMSZ1s9shDQCxF/39Bq6ce/8fmh3TFzazZt31AcUGHyjqjtIdrY2dsIO4DBbex cbnA1T/yjXVRHihfMiUD0Rsx6dUBvxJogIDfNXKC27nSKG2IMSCYMCJfl6dgKtPl QwssBhsO6M1g6VmYX3aUKtpfxucwniGFu+H5a7Dn8Yjbahy/98altlgqmsCPOBT1 jojRsFY+SLsg2zlwPy4UbYQIiaGApeYrAqNsK/t6YAGmQbjqB1YxVYoXfKlbFzZN mvReOpTbJQxfQ4PrpMAw1tQIo0Y9cN3fcWejY/Sg8b+y4xLWODW0HoUdf8SQIsbw == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdeiledgudefudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucfkphepjeejrddukeefrdefjedrudehieen ucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: from vm-mail.pks.im (x4db7259c.dyn.telefonica.de [77.183.37.156]) by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 19 May 2021 15:13:31 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 0b1b9e74 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Wed, 19 May 2021 19:13:28 +0000 (UTC) Date: Wed, 19 May 2021 21:13:27 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 2/8] p5400: add perf tests for git-receive-pack(1) Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org We'll the connectivity check logic for git-receive-pack(1) in the following commits to make it perform better. As a preparatory step, add some benchmarks such that we can measure these changes. Signed-off-by: Patrick Steinhardt --- t/perf/p5400-receive-pack.sh | 74 ++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100755 t/perf/p5400-receive-pack.sh diff --git a/t/perf/p5400-receive-pack.sh b/t/perf/p5400-receive-pack.sh new file mode 100755 index 0000000000..2b0c89d977 --- /dev/null +++ b/t/perf/p5400-receive-pack.sh @@ -0,0 +1,74 @@ +#!/bin/sh + +test_description="Tests performance of receive-pack" + +. ./perf-lib.sh + +test_perf_large_repo + +test_expect_success 'setup' ' + # Create a main branch such that we do not have to rely on any specific + # branch to exist in the perf repository. + git switch --force-create main && + + TARGET_REPO_CLONE=$(pwd)/target-clone.git && + git clone --bare --dissociate --branch main "$(pwd)" "$TARGET_REPO_CLONE" && + TARGET_REPO_REFS=$(pwd)/target-refs.git && + git clone --bare --dissociate --branch main "$(pwd)" "$TARGET_REPO_REFS" && + TARGET_REPO_EMPTY=$(pwd)/target-empty.git && + git init --bare "$TARGET_REPO_EMPTY" && + + # Set up a pre-receive hook such that no refs will ever be changed. + # This easily allows multiple perf runs, but still exercises + # server-side reference negotiation and checking for consistency. + mkdir hooks && + write_script hooks/pre-receive <<-EOF && + #!/bin/sh + echo "failed in pre-receive hook" + exit 1 + EOF + cat >config <<-EOF && + [core] + hooksPath=$(pwd)/hooks + EOF + GIT_CONFIG_GLOBAL="$(pwd)/config" && + export GIT_CONFIG_GLOBAL && + + # Create a reference for each commit in the target repository with + # extra-refs. While this may be an atypical setup, biggish repositories + # easily end up with hundreds of thousands of refs, and this is a good + # enough approximation. + git -C "$TARGET_REPO_REFS" log --all --format="tformat:create refs/commit/%h %H" | + git -C "$TARGET_REPO_REFS" update-ref --stdin && + + git switch --create updated && + test_commit --no-tag updated +' + +while read name repo +do + refs=("create updated:new") + + # If the target repository is the empty one, then the only thing we can + # do is to create a new branch. + if test "$repo" != "$TARGET_REPO_EMPTY" + then + refs+=("update updated:main" "reset main~:main" "delete :main") + fi + + while read desc ref + do + test_expect_success "setup $name $desc" " + test_must_fail git push --force '$repo' '$ref' \ + --receive-pack='tee pack | git receive-pack' 2>err && + grep 'failed in pre-receive hook' err + " + + test_perf "receive-pack $name $desc" " + git receive-pack '$repo' negotiation && + grep 'pre-receive hook declined' negotiation + " + done < <(printf "%s\n" "${refs[@]}") +done < <(printf "%s\n" "clone $TARGET_REPO_CLONE" "extrarefs $TARGET_REPO_REFS" "empty $TARGET_REPO_EMPTY") + +test_done From patchwork Wed May 19 19:13:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12268269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86059C433B4 for ; Wed, 19 May 2021 19:13:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6464A6135A for ; Wed, 19 May 2021 19:13:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231932AbhESTO7 (ORCPT ); Wed, 19 May 2021 15:14:59 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:46641 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230499AbhESTOx (ORCPT ); Wed, 19 May 2021 15:14:53 -0400 Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 12B9C5C0198 for ; Wed, 19 May 2021 15:13:33 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Wed, 19 May 2021 15:13:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:subject:message-id:references:mime-version:content-type :in-reply-to; s=fm1; bh=3w9UTEi5ughA1WPCjA7SpmfmOlIHVDVAYUWX2MEM hPE=; b=eA2zz1zT++Br51tUxbgsehwCbPzlfwsfFk69FckBkbmV0C5NJVdwCUFf TWET5GHkKpT80LP3oS8udf04KHEjdAR1qTqLhZoAI7JXuN+RzYnqfEhbdukGWhRJ WYx7kPlG45wB5FGCme+HkvQyX6ph5S0ZN+78uFyCMXXRVb2x8V7o+fzqjit8zJ5S RMyv/9lknY9xgsNlBmZUbo1VF14jjVUMWaOLYgQDp5aqERcONbMjYz1NTRqTql3V ElqUoFJvtQwJUVrSV8WOevah3BnxgIb/XBdB1RO4aVmu9u/hTs4Y086eGVTPFZlM 52iyTufFWiN083vvFFRPtUliFTM9pg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=3w9UTE i5ughA1WPCjA7SpmfmOlIHVDVAYUWX2MEMhPE=; b=sVN6byB311SfBulFy9UltB gYReynn65OIbrBTWBl6cB5F9HHZ1XrXUYf6kmc9h6fsUi4pSTFyEPc2E7owL3k1P oO7B1t1QXncnfI5JmA0M3AMqVpU9+401EjyQxwCxpeyOfgZ2wuaZalGTXHOmAiB2 Fo+hGyCuR3A7VEwPb5d3pB49ELKCnTPfs20kzIFGZMhjrot5GivE/G/8kfzBFetW 9b//GMbIhWbUkwenV3U7QddfmJUiDzV9ihYlT35aJy0+Xq59NBYbWfO030qq0pBX SucSMaHCpm2vl32BYiTESbNC2kZTvKABAdHTydIiGQnMdgj4aiDJjiXGDsw29T4Q == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdeiledgudefudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucfkphepjeejrddukeefrdefjedrudehieen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: from vm-mail.pks.im (x4db7259c.dyn.telefonica.de [77.183.37.156]) by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 19 May 2021 15:13:32 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 61a9e053 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Wed, 19 May 2021 19:13:32 +0000 (UTC) Date: Wed, 19 May 2021 21:13:31 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 3/8] tmp-objdir: expose function to retrieve path Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org It's currently impossible to tell the path of a temporary object directory for outside users of `struct tmp_objdir`. We'll soon need that information though so that we can reuse information from the quarantine environment in git-receive-pack(1). Provide a new function `tmp_objdir_path()` which returns the path of a temporary object directory to prepare for this. Signed-off-by: Patrick Steinhardt --- tmp-objdir.c | 7 +++++++ tmp-objdir.h | 5 +++++ 2 files changed, 12 insertions(+) diff --git a/tmp-objdir.c b/tmp-objdir.c index b8d880e362..6056917c63 100644 --- a/tmp-objdir.c +++ b/tmp-objdir.c @@ -288,6 +288,13 @@ const char **tmp_objdir_env(const struct tmp_objdir *t) return t->env.v; } +const char *tmp_objdir_path(const struct tmp_objdir *t) +{ + if (!t) + return NULL; + return t->path.buf; +} + void tmp_objdir_add_as_alternate(const struct tmp_objdir *t) { add_to_alternates_memory(t->path.buf); diff --git a/tmp-objdir.h b/tmp-objdir.h index b1e45b4c75..da3ccb98bc 100644 --- a/tmp-objdir.h +++ b/tmp-objdir.h @@ -51,4 +51,9 @@ int tmp_objdir_destroy(struct tmp_objdir *); */ void tmp_objdir_add_as_alternate(const struct tmp_objdir *); +/* + * Return the path of the temporary object directory. + */ +const char *tmp_objdir_path(const struct tmp_objdir *t); + #endif /* TMP_OBJDIR_H */ From patchwork Wed May 19 19:13:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12268271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 041DCC433B4 for ; Wed, 19 May 2021 19:13:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E3D356135F for ; Wed, 19 May 2021 19:13:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231970AbhESTPC (ORCPT ); Wed, 19 May 2021 15:15:02 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:39831 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231889AbhESTO6 (ORCPT ); Wed, 19 May 2021 15:14:58 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 072D95C0103 for ; Wed, 19 May 2021 15:13:38 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Wed, 19 May 2021 15:13:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:subject:message-id:references:mime-version:content-type :in-reply-to; s=fm1; bh=kri4+NReiWrG8mL6/s39X0QgR/ffmmdi2gzmlXUD dZQ=; b=cO7iK8ejSsatsW3orI0gXNXzQaHRcf6OuFoMaahOa3S5nAl7sDdZ18D7 fIofmwcFpwl8z7/sS1RVjDCZu5QExanozx9wTje2btUW4Dr+MAnMkE6QL/bkIG2C x4qFK0taJrUiF403KfPD5NNHTyVw/MwN2iKPpMEHjchBOb3rsQgKb26oxC0ul5Jo wf4GhCN7rBhh5NTwUoD5OK3e6Eo/iqiKJmOcZqxwQ/b69+CM1dPiP8uk4E/Hs1xX ZSoyoJsrT/Pr83Cy8rZaDO8Wm6Q2B4m6GDTeE5QGQ+npCyu231KJehD17anHhaTd 4QHBlGXWkw7nj83gy+qyVxDjXmGYDA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=kri4+N ReiWrG8mL6/s39X0QgR/ffmmdi2gzmlXUDdZQ=; b=J1VRxw0Gg1e02J1xIppeEY VmNCoH/75nb2snnxuJyDqUEfJyx2BE9OlLNh0NE/g91S2Xp45s2NiAA8tvDCOIwL Q5HfYZZMnEqtEOW/aPueJ4KfNMYA4n9k2382bj8KpFZ9qkpm++6q5M2de+zIcugU 0PF6wBdJn3RYhNX1PX3sPQSU9nPNpPQ11aAlhPuTjNAVfmQHhCgb4AzTCX/qi57u g7ox9HgVdpPKMxLZFHid2S8RJq8h3Z1eLzUjupKYFTIx0jtvHVeQKb81Uwm2MJ4U ca6cwL53l3mLwkYcN5+jcy8wNddGX5Yv49LFJcEgBQkC4XrTtVChE6fD1TnyjDAA == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdeiledgudefudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucfkphepjeejrddukeefrdefjedrudehieen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: from vm-mail.pks.im (x4db7259c.dyn.telefonica.de [77.183.37.156]) by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 19 May 2021 15:13:37 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 42328069 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Wed, 19 May 2021 19:13:36 +0000 (UTC) Date: Wed, 19 May 2021 21:13:35 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 4/8] packfile: have `for_each_file_in_pack_dir()` return error codes Message-ID: <64a1bcf2c11d180542e06f67c451f9f90019dea4.1621451532.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The function `for_each_file_in_pack_dir()` doesn't ever return error codes right now: any errors it hits are simply swallowed. A new user we're about to add would like to know what's going on though such that it can decide whether the call was successful or not. Refactor the function to return an error code, and adapt the signature of the callback function to also return an error code. This enables the callback to abort iteration and have its error code passed through. Existing callers are adapted to this change, but keep ignoring errors. Signed-off-by: Patrick Steinhardt --- midx.c | 22 +++++++++++++--------- packfile.c | 26 ++++++++++++++++---------- packfile.h | 10 +++++----- 3 files changed, 34 insertions(+), 24 deletions(-) diff --git a/midx.c b/midx.c index 21d6a05e88..f87a3d36ab 100644 --- a/midx.c +++ b/midx.c @@ -470,15 +470,15 @@ struct write_midx_context { int preferred_pack_idx; }; -static void add_pack_to_midx(const char *full_path, size_t full_path_len, - const char *file_name, void *data) +static int add_pack_to_midx(const char *full_path, size_t full_path_len, + const char *file_name, void *data) { struct write_midx_context *ctx = data; if (ends_with(file_name, ".idx")) { display_progress(ctx->progress, ++ctx->pack_paths_checked); if (ctx->m && midx_contains_pack(ctx->m, file_name)) - return; + return 0; ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); @@ -489,7 +489,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, if (!ctx->info[ctx->nr].p) { warning(_("failed to add packfile '%s'"), full_path); - return; + return 0; } if (open_pack_index(ctx->info[ctx->nr].p)) { @@ -497,7 +497,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, full_path); close_pack(ctx->info[ctx->nr].p); FREE_AND_NULL(ctx->info[ctx->nr].p); - return; + return 0; } ctx->info[ctx->nr].pack_name = xstrdup(file_name); @@ -505,6 +505,8 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, ctx->info[ctx->nr].expired = 0; ctx->nr++; } + + return 0; } struct pack_midx_entry { @@ -1110,19 +1112,21 @@ struct clear_midx_data { const char *ext; }; -static void clear_midx_file_ext(const char *full_path, size_t full_path_len, - const char *file_name, void *_data) +static int clear_midx_file_ext(const char *full_path, size_t full_path_len, + const char *file_name, void *_data) { struct clear_midx_data *data = _data; if (!(starts_with(file_name, "multi-pack-index-") && ends_with(file_name, data->ext))) - return; + return 0; if (data->keep && !strcmp(data->keep, file_name)) - return; + return 0; if (unlink(full_path)) die_errno(_("failed to remove %s"), full_path); + + return 0; } static void clear_midx_files_ext(struct repository *r, const char *ext, diff --git a/packfile.c b/packfile.c index b79cbc8cd4..f488a214cf 100644 --- a/packfile.c +++ b/packfile.c @@ -792,14 +792,15 @@ static void report_pack_garbage(struct string_list *list) report_helper(list, seen_bits, first, list->nr); } -void for_each_file_in_pack_dir(const char *objdir, - each_file_in_pack_dir_fn fn, - void *data) +int for_each_file_in_pack_dir(const char *objdir, + each_file_in_pack_dir_fn fn, + void *data) { struct strbuf path = STRBUF_INIT; size_t dirnamelen; DIR *dir; struct dirent *de; + int ret = 0; strbuf_addstr(&path, objdir); strbuf_addstr(&path, "/pack"); @@ -809,7 +810,7 @@ void for_each_file_in_pack_dir(const char *objdir, error_errno("unable to open object pack directory: %s", path.buf); strbuf_release(&path); - return; + return -1; } strbuf_addch(&path, '/'); dirnamelen = path.len; @@ -820,11 +821,14 @@ void for_each_file_in_pack_dir(const char *objdir, strbuf_setlen(&path, dirnamelen); strbuf_addstr(&path, de->d_name); - fn(path.buf, path.len, de->d_name, data); + ret = fn(path.buf, path.len, de->d_name, data); + if (ret) + break; } closedir(dir); strbuf_release(&path); + return ret; } struct prepare_pack_data { @@ -834,8 +838,8 @@ struct prepare_pack_data { struct multi_pack_index *m; }; -static void prepare_pack(const char *full_name, size_t full_name_len, - const char *file_name, void *_data) +static int prepare_pack(const char *full_name, size_t full_name_len, + const char *file_name, void *_data) { struct prepare_pack_data *data = (struct prepare_pack_data *)_data; struct packed_git *p; @@ -858,13 +862,13 @@ static void prepare_pack(const char *full_name, size_t full_name_len, } if (!report_garbage) - return; + return 0; if (!strcmp(file_name, "multi-pack-index")) - return; + return 0; if (starts_with(file_name, "multi-pack-index") && ends_with(file_name, ".rev")) - return; + return 0; if (ends_with(file_name, ".idx") || ends_with(file_name, ".rev") || ends_with(file_name, ".pack") || @@ -874,6 +878,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len, string_list_append(data->garbage, full_name); else report_garbage(PACKDIR_FILE_GARBAGE, full_name); + + return 0; } static void prepare_packed_git_one(struct repository *r, char *objdir, int local) diff --git a/packfile.h b/packfile.h index 3ae117a8ae..eac930cd27 100644 --- a/packfile.h +++ b/packfile.h @@ -39,11 +39,11 @@ const char *pack_basename(struct packed_git *p); struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path); -typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len, - const char *file_pach, void *data); -void for_each_file_in_pack_dir(const char *objdir, - each_file_in_pack_dir_fn fn, - void *data); +typedef int each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len, + const char *file_pach, void *data); +int for_each_file_in_pack_dir(const char *objdir, + each_file_in_pack_dir_fn fn, + void *data); /* A hook to report invalid files in pack directory */ #define PACKDIR_FILE_PACK 1 From patchwork Wed May 19 19:13:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12268273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA45DC43460 for ; Wed, 19 May 2021 19:13:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7EF4B6135A for ; Wed, 19 May 2021 19:13:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231889AbhESTPD (ORCPT ); Wed, 19 May 2021 15:15:03 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:45657 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231956AbhESTPB (ORCPT ); Wed, 19 May 2021 15:15:01 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id BB3555C00E4 for ; Wed, 19 May 2021 15:13:41 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Wed, 19 May 2021 15:13:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:subject:message-id:references:mime-version:content-type :in-reply-to; s=fm1; bh=8syEP58N1OpoHFUpudab6VLfAK4BVvwzE8lkgyNe anA=; b=MvfCSjDTRqQxmVxIWdHFDk+7Bka9VKrhvhz+4+xEurmeXDAWS4O9+kLH Wj12swrzyZS3wI+cZ0v40vA/Ka4eicKV8ZWEm6s+LLq5Qo7HSX8j6NZymXFwXooF DiG+DlkBTfy/aAkWBK7TW6SfgR0rFGf7yAhzvQ5MTDwzr7Ra7YRtXhCSlcCEL15s X9Tbkr4NOuWk0+qx3DUKE3x+VZ9qVVVhv2aC8PV17r4elss7jBNDivG9e8JQSGd7 gt3vdR1hIszglnj+vjppfIyDNb3ayaF4TfB/mablMhgR7lvpLop8jyqhGWVy+XcQ 1NMHmQf7OF9S3wcm3sdx8Pl4V9PdqQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=8syEP5 8N1OpoHFUpudab6VLfAK4BVvwzE8lkgyNeanA=; b=pbsL4Gq/YXyYKZEIXbYoQK a2941sUTF8Qq3Mn5Zd53kmfhZwkWvK0qZIPCeN/EqxbShKj5KY6WofBW5QqYGhCd B09GQFplmUAUagRy2owyW+rolyUpwsPWJ3838/mOYz+ziKp1jHCLeuPbJNrHVNFb VnVJBnb9va9/r7FZx8eEIQMCxnd7ZbTRldNG5nndtGvwSGNdkzQ2CXQ6Kl74mhfp 7xCrhM8xPqx85xbT1CZ9dyC6DPU6tv9c88VY56PiB008KNz2hU9HkDejIlRGjA+j QKDzD7I1ZMt9rNJlDVfJTt2g+ExU1yeFOePO4+LGWfmUV7Z0XSJx/9xkFuzGa6PQ == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdeiledgudefudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucfkphepjeejrddukeefrdefjedrudehieen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: from vm-mail.pks.im (x4db7259c.dyn.telefonica.de [77.183.37.156]) by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 19 May 2021 15:13:41 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id a6964163 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Wed, 19 May 2021 19:13:40 +0000 (UTC) Date: Wed, 19 May 2021 21:13:39 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 5/8] object-file: allow reading loose objects without reading their contents Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org We're about to add a new implementation of the connectivity check via the object quarantine directory, which will directly iterate over any loose object via their paths. For optimization purposes, we'll want to skip over any loose blobs, but right now there's no easy way to read a loose object without also slurping in all its contents. Fix this shortcoming by modifying `read_loose_object()` such that it can handle the case where no pointer for `contents` was passed in. If so, then we'll simply skip reading the loose object. Signed-off-by: Patrick Steinhardt --- object-file.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/object-file.c b/object-file.c index 884855b9df..ab695823fd 100644 --- a/object-file.c +++ b/object-file.c @@ -2552,7 +2552,8 @@ int read_loose_object(const char *path, git_zstream stream; char hdr[MAX_HEADER_LEN]; - *contents = NULL; + if (contents) + *contents = NULL; map = map_loose_object_1(the_repository, path, NULL, &mapsize); if (!map) { @@ -2572,6 +2573,12 @@ int read_loose_object(const char *path, goto out; } + if (!contents) { + git_inflate_end(&stream); + ret = 0; + goto out; + } + if (*type == OBJ_BLOB && *size > big_file_threshold) { if (check_stream_oid(&stream, hdr, *size, path, expected_oid) < 0) goto out; From patchwork Wed May 19 19:13:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12268275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36024C433ED for ; Wed, 19 May 2021 19:13:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 141BF6108D for ; Wed, 19 May 2021 19:13:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231910AbhESTPI (ORCPT ); Wed, 19 May 2021 15:15:08 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:39053 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231939AbhESTPF (ORCPT ); Wed, 19 May 2021 15:15:05 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 7C2B35C01CE for ; Wed, 19 May 2021 15:13:45 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Wed, 19 May 2021 15:13:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:subject:message-id:references:mime-version:content-type :in-reply-to; s=fm1; bh=XBE2DKNW8XYwAnCyazi/JQgjabzUqmqn1KFXWTbZ 7+0=; b=FJdV7OVIDWcPW65bZfCOY5e77sVMXq2zi8kbWswbCkBuRBwvR4YYhUJg e2aWBrE/m4k1acvBHjx+SVYx/AalSRVqHVcaxgsO6Wju6WDxSmKseVndzYHuqLEa FHUHMZiGQMHpR3PKS3uiLFI3vv692/m+7f+90ACmQ89MxvQH9eUbT4qMyXHmqU59 p5r6Xne9EaElR2d95mJTogU8q6hgXWsikONN0PljNTk96Wk+ZdOoI+43MODAXutF TzlBcDsJUXUiZ6nNqoJYsBVshZ0xzDV4pRU6jkFwzybuBzUoE+7rHs0MCnmZuaG+ v7Ucrnwlfj6xGkapWYWxfeAPmiFHIw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=XBE2DK NW8XYwAnCyazi/JQgjabzUqmqn1KFXWTbZ7+0=; b=BmS8PEjn1eaeguXJJL9WoJ 6wKHgpFNXFOKn2MB5YswPzP51JHGT04hPgOs5ijbRpBlcu92Y7vRyaXCBuP+VsPa WwGgfAl0Hsv6CSVT5Jl9MeJtVL/1qbmYbMGs/l5d+s6dVfdfRbrZ0cPJdYJ7WCnt MCInGxvYkPBk6rfzBdNRQKoyhEXKWMSmf5pJzj+GKTQPN+RghrYi2arpr3HAIOEF kpKPDWeijKL3brFa8ZWfK1yV0nQORAr+XH90ny44AGNgWKNQZcK5udfYgBmoZ9Sm jNQC872peDn/5sZhNz0MT7QmT+n0t7TBDjVU81VcaKW2AuTgnnoT5CjA6b6c6g0w == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdeiledgudefvdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucfkphepjeejrddukeefrdefjedrudehieen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: from vm-mail.pks.im (x4db7259c.dyn.telefonica.de [77.183.37.156]) by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 19 May 2021 15:13:44 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 36817fdd (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Wed, 19 May 2021 19:13:44 +0000 (UTC) Date: Wed, 19 May 2021 21:13:43 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 6/8] connected: implement connectivity check via temporary object dirs Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The connectivity checks are currently implemented via git-rev-list(1): we simply ignore any preexisting refs via `--not --all`, and pass all new refs which are to be checked via its stdin. While this works well enough for repositories which do not have a lot of references, it's clear that `--not --all` will linearly scale with the number of refs which do exist: for each reference, we'll walk through its commit as well as its five parent commits (defined via `SLOP`). Given that many major hosting sites which use a pull/merge request workflow keep refs to the request's HEAD, this effectively means that `--not --all` will do a nearly complete graph walk. We can compute the set of pushed objects much more easily though. We know exactly which objects were just received, which is the set of objects which are in git-receive-pack(1)'s object quarantine directory. While this may overshoot new objects because the client may have sent objects which we already know, it lets us avoid doing a graph walk of preexisting commits. The implementation of this is quite simple: we iterate through all loose and packed objects in the object directory, and for each object we check whether its immediate referenced objects exist. There is no need to do a walk beyond these direct links: if the receiving repository already has the directly referenced object, then it's not a new object in the first place and thus does not need a re-check. If the referenced object is a new one and we can load it, then we know that it will eventually be checked via the quarantined objects walk, too. Finally, after we have concluded that all quarantined objects are indeed fully connected, the only thing left to check is whether all reference updates reference an existing object. This new connectivity check will be wired up in a subsequent commit. Signed-off-by: Patrick Steinhardt --- connected.c | 192 ++++++++++++++++++++++++++++++++++++++++++++++++++++ connected.h | 19 ++++++ 2 files changed, 211 insertions(+) diff --git a/connected.c b/connected.c index b18299fdf0..ab0eb9f974 100644 --- a/connected.c +++ b/connected.c @@ -6,6 +6,13 @@ #include "transport.h" #include "packfile.h" #include "promisor-remote.h" +#include "object.h" +#include "tree-walk.h" +#include "commit.h" +#include "tag.h" +#include "progress.h" +#include "tmp-objdir.h" +#include "oidset.h" /* * If we feed all the commits we want to verify to this command @@ -151,3 +158,188 @@ int check_connected(oid_iterate_fn fn, void *cb_data, sigchain_pop(SIGPIPE); return finish_command(&rev_list) || err; } + +struct connected_payload { + struct repository *repo; + struct progress *progress; + struct oidset seen; + size_t checked_objects; + size_t missing_objects; +}; + +static void check_missing(struct connected_payload *payload, const struct object_id *oid) +{ + if (oidset_contains(&payload->seen, oid)) + return; + if (oid_object_info(payload->repo, oid, NULL) <= 0) + payload->missing_objects++; + oidset_insert(&payload->seen, oid); + display_progress(payload->progress, ++payload->checked_objects); +} + +static int check_quarantined_object(const struct object_id *oid, enum object_type type, + void *content, unsigned long size, void *data) +{ + struct connected_payload *payload = data; + struct object *object; + int eaten; + + object = parse_object_buffer(payload->repo, oid, type, size, content, &eaten); + if (!object) { + if (!eaten) + free(content); + error(_("could not parse quarantined object %s"), oid_to_hex(oid)); + return -1; + } + if (!eaten) + free(content); + + if (type == OBJ_TREE) { + struct tree *tree = (struct tree *)object; + struct tree_desc tree_desc; + struct name_entry entry; + + if (init_tree_desc_gently(&tree_desc, tree->buffer, tree->size)) + return -1; + while (tree_entry_gently(&tree_desc, &entry)) + check_missing(payload, &entry.oid); + + free_tree_buffer(tree); + } else if (type == OBJ_COMMIT) { + struct commit *commit = (struct commit *) object; + struct commit_list *parents; + + check_missing(payload, get_commit_tree_oid(commit)); + for (parents = commit->parents; parents; parents = parents->next) + check_missing(payload, &parents->item->object.oid); + } else if (type == OBJ_TAG) { + check_missing(payload, get_tagged_oid((struct tag *) object)); + } else { + error(_("unexpected object type")); + return -1; + } + + return 0; +} + +static int loose_object_iterator(const struct object_id *oid, + const char *path, void *_data) +{ + struct connected_payload *payload = _data; + enum object_type type; + unsigned long size; + void *content; + + if (read_loose_object(path, oid, &type, &size, NULL) < 0) + return -1; + + oidset_insert(&payload->seen, oid); + display_progress(payload->progress, ++payload->checked_objects); + + if (type == OBJ_BLOB) + return 0; + if (read_loose_object(path, oid, &type, &size, &content) < 0) + return -1; + + return check_quarantined_object(oid, type, content, size, payload); +} + +static int packed_object_iterator(const struct object_id *oid, + struct packed_git *pack, uint32_t pos, + void *_data) +{ + off_t off = nth_packed_object_offset(pack, pos); + struct connected_payload *payload = _data; + struct object_info oi = OBJECT_INFO_INIT; + enum object_type type; + unsigned long size; + void *content; + + oi.typep = &type; + if (packed_object_info(payload->repo, pack, off, &oi) < 0) + return -1; + + oidset_insert(&payload->seen, oid); + display_progress(payload->progress, ++payload->checked_objects); + + if (type == OBJ_BLOB) + return 0; + + oi.contentp = &content; + oi.sizep = &size; + + if (packed_object_info(payload->repo, pack, off, &oi) < 0) + return -1; + + return check_quarantined_object(oid, type, content, size, payload); +} + +static int pack_dir_iterator(const char *full_path, size_t full_path_len, + const char *file_path, void *data) +{ + if (ends_with(full_path, ".idx")) { + struct packed_git *pack = add_packed_git(full_path, full_path_len, 1); + if (!pack) { + error(_("unable to add quarantined pack")); + return -1; + } + + if (for_each_object_in_pack(pack, packed_object_iterator, data, + FOR_EACH_OBJECT_PACK_ORDER) < 0) { + error(_("unable to iterate packed objects")); + return -1; + } + } + + return 0; +} + +int check_connected_quarantine(struct repository *repo, + struct tmp_objdir *quarantine, + oid_iterate_fn fn, void *cb_data) +{ + struct connected_payload payload = { + .repo = repo, + .seen = OIDSET_INIT, + }; + struct object_id oid; + int ret; + + payload.progress = start_delayed_progress("Checking connectivity", 0); + + /* + * Iterate through all loose and packed objects part of the object + * quarantine and verify that for each of them, all directly referenced + * objects exist. + */ + if (for_each_loose_file_in_objdir(tmp_objdir_path(quarantine), + loose_object_iterator, NULL, + NULL, &payload) < 0) { + error(_("unable to check quarantined loose objects")); + ret = -1; + goto out; + } + + if (for_each_file_in_pack_dir(tmp_objdir_path(quarantine), + pack_dir_iterator, &payload) < 0) { + error(_("unable to check quarantined packed objects")); + ret = -1; + goto out; + } + + /* + * After we've verified that all quarantined objects are indeed + * connected, we need to verify that all new tips are contained in the + * repository, too. + */ + while (!fn(cb_data, &oid)) { + if (oid_object_info(repo, &oid, NULL) <= 0) + payload.missing_objects++; + display_progress(payload.progress, ++payload.checked_objects); + } + +out: + stop_progress(&payload.progress); + oidset_clear(&payload.seen); + return ret || payload.missing_objects != 0; +} diff --git a/connected.h b/connected.h index 8d5a6b3ad6..336b6f362e 100644 --- a/connected.h +++ b/connected.h @@ -3,6 +3,7 @@ struct object_id; struct transport; +struct tmp_objdir; /* * Take callback data, and return next object name in the buffer. @@ -62,4 +63,22 @@ struct check_connected_options { int check_connected(oid_iterate_fn fn, void *cb_data, struct check_connected_options *opt); +/* + * Make sure that all objects in the given quarantine directory are connected + * and that all OIDs returned by the given callback are backed by an existing + * object. The quarantine directory must have been made available to the + * repository via `add_to_alternates_memory()`. + * + * Given that objects are enumerated via the quarantine directory directly, + * this check should in general be more efficient than `check_connected()` + * while being more pedantic at the same time (e.g. quarantined objects which + * aren't referenced by anything but which do have missing links would get + * rejected). + * + * Return 0 if Ok, non zero otherwise (i.e. some missing objects) + */ +int check_connected_quarantine(struct repository *repo, + struct tmp_objdir *quarantine, + oid_iterate_fn fn, void *cb_data); + #endif /* CONNECTED_H */ From patchwork Wed May 19 19:13:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12268277 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3FE8C433B4 for ; Wed, 19 May 2021 19:13:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C7D576135A for ; Wed, 19 May 2021 19:13:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231939AbhESTPK (ORCPT ); Wed, 19 May 2021 15:15:10 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:51613 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230298AbhESTPJ (ORCPT ); Wed, 19 May 2021 15:15:09 -0400 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 43A925C018D for ; Wed, 19 May 2021 15:13:49 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Wed, 19 May 2021 15:13:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:subject:message-id:references:mime-version:content-type :in-reply-to; s=fm1; bh=oohEfvtXETt+4nTgvdIMbiItD4RoxGuP+HUtYoAI Sxc=; b=NDMgJSfhTDsfmAHJW1Hps/CFoyCxn/baf5X8FSr/Q1WdKcCayUebfIKX J5woE3qbL+fSz0UgTvWYMhmWct046HOIyYLqELbcPE7moh/RU/Hq/WNTzFsIk0GD S71DSFzG8Hfdpcc0P9d/HDGcMZ5DkUAKGQEUaq82reZvKV6BAYk8l5cF6s46K0fQ 0Y9oEFoaKd7CgLpkLB7jM6oyMAUz+nDzPlQ7rVvcGwexuaYZL45M/A5rlUnTJtHA M9uLbXcnNYUTLUtL1Nf2l901k8XDUqjPhrvpVkPtnyR7RustZk0k8dS0N2lcHRe8 IrQ+IHS84TpjAQVWQkbl81oNT+wXcQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=oohEfv tXETt+4nTgvdIMbiItD4RoxGuP+HUtYoAISxc=; b=h7muzR8oBZoiLUh8bKc9yO BmrP+BcqwYxl2S6EB00mpGapNpuSxSL483j/+Ngv5tfc/hdzLWNWvAi+75C8E5IA GIJfi9X3dagCv60V5xGialUyKgLQghwI8MFNAge9jG4+a/0gA+TmomyLgFm0BYF2 QQhoPTd+8LaJD2I22stzIMkOf6vEpQilyoV1uwgyK/pjYcrgU0Kd93jx85CfUmsZ hKHF1hLirjC6mI59K5jKGT4pPexOoXmwMgHMY/hAjDll6enwM/guyM/mklnSZlN5 lQb5ghv90dHwhulIztUU0B5gimZ3WruEltWtLUEuIjJMH7n5FUtizXlBNpbLl9kg == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdeiledgudefudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucfkphepjeejrddukeefrdefjedrudehieen ucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: from vm-mail.pks.im (x4db7259c.dyn.telefonica.de [77.183.37.156]) by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 19 May 2021 15:13:48 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 9bd5f0ea (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Wed, 19 May 2021 19:13:48 +0000 (UTC) Date: Wed, 19 May 2021 21:13:47 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 7/8] receive-pack: skip connectivity checks on delete-only commands Message-ID: <68c60aff0c77c562aba5613ccbb9ab33ad8e0e08.1621451532.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In the case where git-receive-pack(1) receives only commands which delete references, then per technical specification the client MUST NOT send a packfile. As a result, we know that no new objects have been received, which makes it a moot point to check whether all received objects are fully connected. Fix this by not doing a connectivity check in case there were no pushed objects. Given that git-rev-walk(1) with only negative references will not do any graph walk, no performance improvements are to be expected. Conceptionally, it is still the right thing to do though. The following tests were executed on linux.git and back up above expectation: Test v2.32.0-rc0 HEAD -------------------------------------------------------------------------------------------- 5400.3: receive-pack clone create 1.27(1.11+0.16) 1.26(1.12+0.14) -0.8% 5400.5: receive-pack clone update 1.27(1.13+0.13) 1.27(1.11+0.16) +0.0% 5400.7: receive-pack clone reset 0.13(0.11+0.02) 0.14(0.11+0.02) +7.7% 5400.9: receive-pack clone delete 0.02(0.01+0.01) 0.02(0.00+0.01) +0.0% 5400.11: receive-pack extrarefs create 33.01(18.80+14.43) 32.63(18.52+14.24) -1.2% 5400.13: receive-pack extrarefs update 33.13(18.85+14.50) 32.82(18.85+14.29) -0.9% 5400.15: receive-pack extrarefs reset 32.90(18.82+14.32) 32.70(18.76+14.20) -0.6% 5400.17: receive-pack extrarefs delete 9.13(4.35+4.77) 8.99(4.28+4.70) -1.5% 5400.19: receive-pack empty create 223.35(640.63+127.74) 226.96(655.16+131.93) +1.6% Signed-off-by: Patrick Steinhardt --- builtin/receive-pack.c | 49 ++++++++++++++++++++++++++---------------- 1 file changed, 30 insertions(+), 19 deletions(-) diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index a34742513a..b9263cec15 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -1918,11 +1918,8 @@ static void execute_commands(struct command *commands, struct shallow_info *si, const struct string_list *push_options) { - struct check_connected_options opt = CHECK_CONNECTED_INIT; struct command *cmd; struct iterate_data data; - struct async muxer; - int err_fd = 0; int run_proc_receive = 0; if (unpacker_error) { @@ -1931,25 +1928,39 @@ static void execute_commands(struct command *commands, return; } - if (use_sideband) { - memset(&muxer, 0, sizeof(muxer)); - muxer.proc = copy_to_sideband; - muxer.in = -1; - if (!start_async(&muxer)) - err_fd = muxer.in; - /* ...else, continue without relaying sideband */ - } - data.cmds = commands; data.si = si; - opt.err_fd = err_fd; - opt.progress = err_fd && !quiet; - opt.env = tmp_objdir_env(tmp_objdir); - if (check_connected(iterate_receive_command_list, &data, &opt)) - set_connectivity_errors(commands, si); - if (use_sideband) - finish_async(&muxer); + /* + * If received commands only consist of deletions, then the client MUST + * NOT send a packfile because there cannot be any new objects in the + * first place. As a result, we do not set up a quarantine environment + * because we know no new objects will be received. And that in turn + * means that we can skip connectivity checks here. + */ + if (tmp_objdir) { + struct check_connected_options opt = CHECK_CONNECTED_INIT; + struct async muxer; + int err_fd = 0; + + if (use_sideband) { + memset(&muxer, 0, sizeof(muxer)); + muxer.proc = copy_to_sideband; + muxer.in = -1; + if (!start_async(&muxer)) + err_fd = muxer.in; + /* ...else, continue without relaying sideband */ + } + + opt.err_fd = err_fd; + opt.progress = err_fd && !quiet; + opt.env = tmp_objdir_env(tmp_objdir); + if (check_connected(iterate_receive_command_list, &data, &opt)) + set_connectivity_errors(commands, si); + + if (use_sideband) + finish_async(&muxer); + } reject_updates_to_hidden(commands); From patchwork Wed May 19 19:13:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12268279 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D9D7C433B4 for ; Wed, 19 May 2021 19:13:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6CEF76108D for ; Wed, 19 May 2021 19:13:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231880AbhESTPP (ORCPT ); Wed, 19 May 2021 15:15:15 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:60011 "EHLO out1-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231972AbhESTPO (ORCPT ); Wed, 19 May 2021 15:15:14 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 0625D5C0040 for ; Wed, 19 May 2021 15:13:54 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Wed, 19 May 2021 15:13:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:subject:message-id:references:mime-version:content-type :in-reply-to; s=fm1; bh=QIdEtu1foRS4unYtCGZyYi595Z5wNCOF3DHhYN5j 2MM=; b=h9fNqZX8G16GSh9bx6tPPMhp7y30VPG2+qlLszc/xgI6wUOgp2iqBXlV plGWJb6lRSoRWfPvKVGoapwdbHut34Qq+MR1khkWxebkiYFCkf5JxstkSpom2Zuu Lq5KkST+yEAzBIuEveyzGfkrcxO63lKSbxIuFHKPSrbTqEkBqIutRtTl64H87Ber Dae4B7Fkxjn8p77Xi5+vocsyy1tazX1AdMtG3O+jSGwUvNnl1wetfuTQDyIVBTuA QzhedBNRLlpTI9dq/f91G87/sbAUhqTllepvtcrzgpOl/OhiKgh+FKg/iW6AKsVv XpScVYoIjX93j+Sl5lI90P/5fMJltw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=QIdEtu 1foRS4unYtCGZyYi595Z5wNCOF3DHhYN5j2MM=; b=cHEiQrVZUKtyp4WiIEGwFP Fa5Z9DtZc8vjNHNZKnfHUswcDjm3+48ZoxL/ZcvjgbZDD/igWNJhV6kjhF4vrIH3 xFN0SLcVLyhKyyb7XhLV4M8UGdHwFkK4XvLFgJHVmx+Diym63fnOEUFODMz8NTCB 6bRIwpUx/EmOdpzooIE6M9s2M9OzDIs40idmAV6/MfmlzKftR3/dUSwQIZw/JKKy e7bfrrtVY08bL8AN+z31iedeOz2TfifxnBlVqNBIOtD8yW8vxaXkock0RJ/xoFfW cQXKdSpwVFDLCjZGx80ggOlKdycCsDfM1sDSamVN96Kps4hPWwTjtntaCfhwgXLw == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdeiledgudefudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucfkphepjeejrddukeefrdefjedrudehieen ucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: from vm-mail.pks.im (x4db7259c.dyn.telefonica.de [77.183.37.156]) by mail.messagingengine.com (Postfix) with ESMTPA for ; Wed, 19 May 2021 15:13:53 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 6f06a1a9 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Wed, 19 May 2021 19:13:52 +0000 (UTC) Date: Wed, 19 May 2021 21:13:51 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 8/8] receive-pack: check connectivity via quarantined objects Message-ID: <010974639c66c8f15d4dc9e4feb69eed92b52d13.1621451532.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Convert git-receive-pack(1) to use the new connectivity check which uses the quarantine directory to enumerate new objects instead of using references. This change significantly decreases the time required to accept reference updates due to the fact that we don't need to do a graph walk anymore. The following measurements have been executed on linux.git: Test v2.32.0-rc0 HEAD -------------------------------------------------------------------------------------------- 5400.3: receive-pack clone create 1.27(1.11+0.16) 0.02(0.01+0.01) -98.4% 5400.5: receive-pack clone update 1.27(1.13+0.13) 0.02(0.02+0.00) -98.4% 5400.7: receive-pack clone reset 0.13(0.11+0.02) 0.02(0.01+0.01) -84.6% 5400.9: receive-pack clone delete 0.02(0.01+0.01) 0.03(0.02+0.01) +50.0% 5400.11: receive-pack extrarefs create 33.01(18.80+14.43) 9.00(4.30+4.65) -72.7% 5400.13: receive-pack extrarefs update 33.13(18.85+14.50) 9.01(4.28+4.67) -72.8% 5400.15: receive-pack extrarefs reset 32.90(18.82+14.32) 9.04(4.26+4.77) -72.5% 5400.17: receive-pack extrarefs delete 9.13(4.35+4.77) 8.94(4.29+4.64) -2.1% 5400.19: receive-pack empty create 223.35(640.63+127.74) 227.55(651.75+130.94) +1.9% Interestingly enough, the normal "clone" repository shows a more extreme improvement compared to the "extrarefs" repository, which has one reference per commit. While I haven't yet dived into the root cause, I expect that there is another location besides the connectivity check which does scale with the number of references. Notably, the connectivity check when pushing into an empty repository shows comparable performance with the previous implementation. This is because in both cases we're actually performing a complete graph walk to determine whether there are any unreachable commits. Signed-off-by: Patrick Steinhardt --- builtin/receive-pack.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index b9263cec15..23087acad0 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -1937,8 +1937,12 @@ static void execute_commands(struct command *commands, * first place. As a result, we do not set up a quarantine environment * because we know no new objects will be received. And that in turn * means that we can skip connectivity checks here. + * + * Using the quarantine directory to determine connectivity may not + * work with shallow clones, so let's let's play it safe for now and + * fall back to the old connectivity check. */ - if (tmp_objdir) { + if (tmp_objdir && (si->nr_ours || si->nr_theirs)) { struct check_connected_options opt = CHECK_CONNECTED_INIT; struct async muxer; int err_fd = 0; @@ -1960,6 +1964,10 @@ static void execute_commands(struct command *commands, if (use_sideband) finish_async(&muxer); + } else if (tmp_objdir) { + if (check_connected_quarantine(the_repository, tmp_objdir, + iterate_receive_command_list, &data)) + set_connectivity_errors(commands, si); } reject_updates_to_hidden(commands);