From patchwork Thu Apr 13 11:16:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13210157 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E31FC77B61 for ; Thu, 13 Apr 2023 11:16:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230399AbjDMLQY (ORCPT ); Thu, 13 Apr 2023 07:16:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50126 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230064AbjDMLQM (ORCPT ); Thu, 13 Apr 2023 07:16:12 -0400 Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2601BF for ; Thu, 13 Apr 2023 04:16:10 -0700 (PDT) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 4B4DC5C0143; Thu, 13 Apr 2023 07:16:10 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Thu, 13 Apr 2023 07:16:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681384570; x=1681470970; bh=gd 9KpwTyqqu1to0OfxFuskUgZclLgXDuX5CdVRAHRqQ=; b=aqjOL5LFXBZqAC1caO hI4tK+VuanOQxuvP/Vwv5gWdgaV0v2U6kLxm/sSvUTx8dPKz52VxO8oiSUjMLRaZ fnXbtlYo0cJ6ksQqYBsQbrcqaqxoUsHX9n0RFP/l4GrDAb5/9+Oxg55e4uOs9aB8 DVdo1uEpVaIPZjmIh7qBr1HP62hvt5Gwi0bVWGEBHYAmPklvJkSE5xAS1QXASlo3 rselXBV6g1JrJRHGMFc/epw8cwlGfk4eEj3kGZoN6thMi1e8al9/ZjGJ1edfYBIl kk01JFYDNgG98XYmpd1SQhEpgCaQ6SZbzilTc9D0lrYz1WqtQOVYS7yOs2dSffGW A0lw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681384570; x=1681470970; bh=gd9KpwTyqqu1t o0OfxFuskUgZclLgXDuX5CdVRAHRqQ=; b=ZDBHYtmjG4OmV8rHcbBy+wngNhGB6 DRDpKwtcy8e1KmkUp7wC7e2/GIOuE7z9YkDGfgJvt0LNKsFpJUGSIQl/vwv3AdzG oW7wuq8Lz/CVsuKnY3cvK2r83orGFvHLPDqw87/0u/WX2k/UunGEvAfkcbkmovbL G5WYD612jKACINJNkgPMk22liDHKJgoUCAXh21enTzMFXo/Y/3XGmEk/MipAeMQF gvatefArG7HtJUkb37IEh2/IhKfxa17Tf4rSbouwGOoxr3eH9+FiUnBvlSTfIZaZ +QJMHMAB5ex7wyOidyGlTGWXLot9LWrG3SgQlKW7DBe+u5YpyXITfqYtA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekkedgfeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 13 Apr 2023 07:16:08 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id 3ea7c0d1 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 13 Apr 2023 11:15:45 +0000 (UTC) Date: Thu, 13 Apr 2023 13:16:06 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v3 01/10] midx: fix segfault with no packs and invalid preferred pack Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When asked to write a multi-pack-index the user can specify a preferred pack that is used as a tie breaker when multiple packs contain the same objects. When this packfile cannot be found, we just pick the first pack that is getting tracked by the newly written multi-pack-index as a fallback. Picking the fallback can fail in the case where we're asked to write a multi-pack-index with no packfiles at all: picking the fallback value will cause a segfault as we blindly index into the array of packfiles, which would be empty. Fix this bug by resetting the preferred packfile index to `-1` before searching for the preferred pack. This fixes the segfault as we already check for whether the index is `> - 1`. If it is not, we simply don't pick a preferred packfile at all. Helped-by: Taylor Blau Signed-off-by: Patrick Steinhardt --- midx.c | 6 +++--- t/t5319-multi-pack-index.sh | 12 ++++++++++++ 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/midx.c b/midx.c index 47989f7ea7..67eb617591 100644 --- a/midx.c +++ b/midx.c @@ -1328,17 +1328,17 @@ static int write_midx_internal(const char *object_dir, } if (preferred_pack_name) { - int found = 0; + ctx.preferred_pack_idx = -1; + for (i = 0; i < ctx.nr; i++) { if (!cmp_idx_or_pack_name(preferred_pack_name, ctx.info[i].pack_name)) { ctx.preferred_pack_idx = i; - found = 1; break; } } - if (!found) + if (ctx.preferred_pack_idx == -1) warning(_("unknown preferred pack: '%s'"), preferred_pack_name); } else if (ctx.nr && diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 499d5d4c78..0883c7c6bd 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -183,6 +183,18 @@ test_expect_success 'write midx with --stdin-packs' ' compare_results_with_midx "mixed mode (one pack + extra)" +test_expect_success 'write with no objects and preferred pack' ' + test_when_finished "rm -rf empty" && + git init empty && + test_must_fail git -C empty multi-pack-index write \ + --stdin-packs --preferred-pack=does-not-exist err && + cat >expect <<-EOF && + warning: unknown preferred pack: ${SQ}does-not-exist${SQ} + error: no pack files to index. + EOF + test_cmp expect err +' + test_expect_success 'write progress off for redirected stderr' ' git multi-pack-index --object-dir=$objdir write 2>err && test_line_count = 0 err From patchwork Thu Apr 13 11:16:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13210159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 948B6C77B6C for ; Thu, 13 Apr 2023 11:16:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229632AbjDMLQ1 (ORCPT ); Thu, 13 Apr 2023 07:16:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50126 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230403AbjDMLQX (ORCPT ); Thu, 13 Apr 2023 07:16:23 -0400 Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A024E93E4 for ; Thu, 13 Apr 2023 04:16:14 -0700 (PDT) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 08F855C0138; Thu, 13 Apr 2023 07:16:14 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Thu, 13 Apr 2023 07:16:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681384574; x=1681470974; bh=gW PRXT63q7Y9UyTU1uylUeB4M2Uhh+kvmwTnBeCnWiI=; b=p1G42qI92NNtUH8ZAk ZKEnCxP2abEsl5JCjQIOXVNiwXQmSAFKeH8n9V8Kx8/PpOepv4VQ0lCQPfiGqbcM SLsMP9QRBmwAOommtPIANHyEC8ZZV/71eKQnurs33rpNP/Gngh+6xSNJxjPcyn3h Oaz5sQ3espP1bOv2NVYCrBv/e9+WpsNAzGBgwPHIAvokmzI6jYvFJoVsdtd6Vu1b ylcUoFohoIZRwIuo+xm3bVKgDwOkhBOjekQqiiYW1iM6b+KQZxFuR22oKK0GvN3t 1DOEkgcI+wkpZJ9q07xCEac/Ernd1W5Jg6wrntlVAwkGwy1PY9O7HA31vDnotjSF IEMw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681384574; x=1681470974; bh=gWPRXT63q7Y9U yTU1uylUeB4M2Uhh+kvmwTnBeCnWiI=; b=DH5IawQlTgnipDgeMX9+eQvV43+6D IwG9oGwY8jAXwWK3p2I/uvzaAo5mCATB445vp4E1X5lIbSFIjqcu5EyHVf3ydWDv FQBhvkYXqlskGq5POD50GRhZ5luPEVAENxF6Z6fmtfeia241xjj9LKUvl969YzmN +Z8aieN+yfqZ7dN9UU1MYgc+QHjf/Z/lxCWmlI/OVgTL9kwZVCVmc0yslH/BHXTL RXoH+6aIsjKTUhPQa5Mj6L5RLrzU96WJkvdVSqpaRZfG8+5IImhUG7oXoG2yR1Tn PBXFX1QdMh2iRk9whAY+m/1Fa/dW+l5GvAefWAWlsxSmaG7WciAa/VJQw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekkedgfeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 13 Apr 2023 07:16:12 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id 82dcf232 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 13 Apr 2023 11:15:50 +0000 (UTC) Date: Thu, 13 Apr 2023 13:16:11 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v3 02/10] repack: fix trying to use preferred pack in alternates Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When doing a geometric repack with multi-pack-indices, then we ask git-multi-pack-index(1) to use the largest packfile as the preferred pack. It can happen though that the largest packfile is not part of the main object database, but instead part of an alternate object database. The result is that git-multi-pack-index(1) will not be able to find the preferred pack and print a warning. It then falls back to use the first packfile that the multi-pack-index shall reference. Fix this bug by only considering packfiles as preferred pack that are local. This is the right thing to do given that a multi-pack-index should never reference packfiles borrowed from an alternate. While at it, rename the function `get_largest_active_packfile()` to `get_preferred_pack()` to better document its intent. Helped-by: Taylor Blau Signed-off-by: Patrick Steinhardt --- builtin/repack.c | 26 +++++++++++++++++++++----- t/t7703-repack-geometric.sh | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+), 5 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index 87f73c8923..687419776d 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -446,8 +446,10 @@ static void split_pack_geometry(struct pack_geometry *geometry, int factor) geometry->split = split; } -static struct packed_git *get_largest_active_pack(struct pack_geometry *geometry) +static struct packed_git *get_preferred_pack(struct pack_geometry *geometry) { + uint32_t i; + if (!geometry) { /* * No geometry means either an all-into-one repack (in which @@ -462,7 +464,21 @@ static struct packed_git *get_largest_active_pack(struct pack_geometry *geometry } if (geometry->split == geometry->pack_nr) return NULL; - return geometry->pack[geometry->pack_nr - 1]; + + /* + * The preferred pack is the largest pack above the split line. In + * other words, it is the largest pack that does not get rolled up in + * the geometric repack. + */ + for (i = geometry->pack_nr; i > geometry->split; i--) + /* + * A pack that is not local would never be included in a + * multi-pack index. We thus skip over any non-local packs. + */ + if (geometry->pack[i - 1]->pack_local) + return geometry->pack[i - 1]; + + return NULL; } static void clear_pack_geometry(struct pack_geometry *geometry) @@ -589,7 +605,7 @@ static int write_midx_included_packs(struct string_list *include, { struct child_process cmd = CHILD_PROCESS_INIT; struct string_list_item *item; - struct packed_git *largest = get_largest_active_pack(geometry); + struct packed_git *preferred = get_preferred_pack(geometry); FILE *in; int ret; @@ -610,9 +626,9 @@ static int write_midx_included_packs(struct string_list *include, if (write_bitmaps) strvec_push(&cmd.args, "--bitmap"); - if (largest) + if (preferred) strvec_pushf(&cmd.args, "--preferred-pack=%s", - pack_basename(largest)); + pack_basename(preferred)); if (refs_snapshot) strvec_pushf(&cmd.args, "--refs-snapshot=%s", refs_snapshot); diff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh index 8821fbd2dd..d6db362541 100755 --- a/t/t7703-repack-geometric.sh +++ b/t/t7703-repack-geometric.sh @@ -281,4 +281,36 @@ test_expect_success '--geometric with pack.packSizeLimit' ' ) ' +test_expect_success '--geometric --write-midx with packfiles in main and alternate ODB' ' + test_when_finished "rm -fr shared member" && + + # Create a shared repository that will serve as the alternate object + # database for the member linked to it. It has got some objects on its + # own that are packed into a single packfile. + git init shared && + test_commit -C shared common-object && + git -C shared repack -ad && + + # We create member so that its alternates file points to the shared + # repository. We then create a commit in it so that git-repack(1) has + # something to repack. + # of the shared object database. + git clone --shared shared member && + test_commit -C member unique-object && + git -C member repack --geometric=2 --write-midx 2>err && + test_must_be_empty err && + + # We should see that a new packfile was generated. + find shared/.git/objects/pack -type f -name "*.pack" >packs && + test_line_count = 1 packs && + + # We should also see a multi-pack-index. This multi-pack-index should + # never refer to any packfiles in the alternate object database. + test -f member/.git/objects/pack/multi-pack-index && + test-tool read-midx member/.git/objects >packs.midx && + grep "^pack-.*\.idx$" packs.midx | sort >actual && + basename member/.git/objects/pack/pack-*.idx >expect && + test_cmp expect actual +' + test_done From patchwork Thu Apr 13 11:16:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13210160 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 759B5C77B61 for ; Thu, 13 Apr 2023 11:16:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230032AbjDMLQj (ORCPT ); Thu, 13 Apr 2023 07:16:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231207AbjDMLQZ (ORCPT ); Thu, 13 Apr 2023 07:16:25 -0400 Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C7B39EE1 for ; Thu, 13 Apr 2023 04:16:19 -0700 (PDT) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 93D265C00F7; Thu, 13 Apr 2023 07:16:18 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Thu, 13 Apr 2023 07:16:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681384578; x=1681470978; bh=j9 ybEYk24bL/jiuMo1i6Zntu7VgoT8tZFMME5+blZ2E=; b=RwWfUuB6oEPIlNKzqN YsgcalYsZTuWx4hiClzdevFM3UAfqvOfCWArvzcSc90E5GMCmAWEr2zBZ8ro2Mkw gRGtf6GL1D1ski0ZRHnCs7O8I99f4iv6gEBTLlIS/Jk6Ipk7XYTy+G2W1DEsLf9J sXkOTFAMR4RqYN08fiPcPcrMw0p3D7DwGVDrILf1kOrynZL8WmRmCJyw6Rw1YYHH qlq9UrtaeAlN0I71XXcqeUJOEkwPufG1KydHCF/NqHcfro30GdxYtxiPAp69kjUp g/OkY0QtXcC/dwdg1ClMceGTCwYTmBSR7UomWTYt4ogpnDvkLT2YP1Uk/E2tzZPF hvKw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681384578; x=1681470978; bh=j9ybEYk24bL/j iuMo1i6Zntu7VgoT8tZFMME5+blZ2E=; b=QDeYgv8W293Dg/G3PdqySdYBm3IkJ aGZqQKa6kEFc9OIptruzFHcZSMa+vp1+zA2MqnYXS+A6yXTc1cLqStx0gedN4Hf2 qNyllRX/7uvu8AIQhAbP+t7Fs/mSrMKiZVfG5Nu9hPpWeWc/gBO4qLuri8wTDfUo EMv4E+o17tdvbl0xfoIJJ86My0p0Z3OIWXRKdT21cXHMWydfD97m5SeQl5PnVzH9 IgFMzU1aLiJGLgEbo2PxvTs1lUeNkA0HKPYjI52fOR4K/+zM+kFP8RPdvkRiQ3Ke M9jeNLYsx3iHc3DotjrI60MiSpNjNvBd/9/ZgHMVbr1rW7CCJXIK8n89A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekkedgfeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedunecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 13 Apr 2023 07:16:17 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id 0ac6559f (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 13 Apr 2023 11:15:54 +0000 (UTC) Date: Thu, 13 Apr 2023 13:16:15 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v3 03/10] repack: fix generating multi-pack-index with only non-local packs Message-ID: <8c3193268fd5938ebf85bd43a551d0c50d2e8336.1681384405.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When writing the multi-pack-index with geometric repacking we will add all packfiles to the index that are part of the geometric sequence. This can potentially also include packfiles borrowed from an alternate object directory. But given that a multi-pack-index can only ever include packs that are part of the main object database this does not make much sense whatsoever. In the edge case where all packfiles are contained in the alternate object database and the local repository has none itself this bug can cause us to invoke git-multi-pack-index(1) with only non-local packfiles that it ultimately cannot find. This causes it to return an error and thus causes the geometric repack to fail. Fix the code to skip non-local packfiles. Co-authored-by: Taylor Blau Signed-off-by: Taylor Blau Signed-off-by: Patrick Steinhardt --- builtin/repack.c | 11 +++++++++++ t/t7703-repack-geometric.sh | 23 +++++++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/builtin/repack.c b/builtin/repack.c index 687419776d..80fc860613 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -572,6 +572,17 @@ static void midx_included_packs(struct string_list *include, for (i = geometry->split; i < geometry->pack_nr; i++) { struct packed_git *p = geometry->pack[i]; + /* + * The multi-pack index never refers to packfiles part + * of an alternate object database, so we skip these. + * While git-multi-pack-index(1) would silently ignore + * them anyway, this allows us to skip executing the + * command completely when we have only non-local + * packfiles. + */ + if (!p->pack_local) + continue; + strbuf_addstr(&buf, pack_basename(p)); strbuf_strip_suffix(&buf, ".pack"); strbuf_addstr(&buf, ".idx"); diff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh index d6db362541..7027676977 100755 --- a/t/t7703-repack-geometric.sh +++ b/t/t7703-repack-geometric.sh @@ -313,4 +313,27 @@ test_expect_success '--geometric --write-midx with packfiles in main and alterna test_cmp expect actual ' +test_expect_success '--geometric --with-midx with no local objects' ' + test_when_finished "rm -fr shared member" && + + # Create a repository with a single packfile that acts as alternate + # object database. + git init shared && + test_commit -C shared "shared-objects" && + git -C shared repack -ad && + + # Create a second repository linked to the first one and perform a + # geometric repack on it. + git clone --shared shared member && + git -C member repack --geometric 2 --write-midx 2>err && + test_must_be_empty err && + + # Assert that we wrote neither a new packfile nor a multi-pack-index. + # We should not have a packfile because the single packfile in the + # alternate object database does not invalidate the geometric sequence. + # And we should not have a multi-pack-index because these only index + # local packfiles, and there are none. + test_dir_is_empty member/$packdir +' + test_done From patchwork Thu Apr 13 11:16:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13210161 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7DE8C77B6C for ; Thu, 13 Apr 2023 11:16:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231134AbjDMLQw (ORCPT ); Thu, 13 Apr 2023 07:16:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231138AbjDMLQh (ORCPT ); Thu, 13 Apr 2023 07:16:37 -0400 Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EEDE39EE5 for ; Thu, 13 Apr 2023 04:16:23 -0700 (PDT) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 326A85C00F7; Thu, 13 Apr 2023 07:16:23 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Thu, 13 Apr 2023 07:16:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681384583; x=1681470983; bh=Yo q3Ol0C/r5DInKrvIhJCc4iy6qTwlwOhLMm28kwb68=; b=GUTGs2XcZjR4KNEyGX 28o4wm7mYpnKOvNRe15++493BT2EbY3avOWG5zjkon2afYCVhVaoXWsHVvpfiWO1 J0ynB3l/KXY0IAr3hrjgxSAQS4gXMj40UTHz5KXSh87+BH769IVTF4ya7wWuLvTc iaQiGXdZVeS95D7fn2nMEsiOLrXrfClVvAcPoI7hRp5gzl2rSXKaXSkVUEQUpeag dXFX8J4KClYYuGw8OX2Ivag5vT8pkgfpV2UN3E8+Gke/nbYUXTs/SAsBilEXKfAS LBqTqlq4YwkkSi/7z2gSswpQQ1FLqt1b0LnHCmZ3QcAJ6mmU3/tJrvaCeH5HMNjy CHBQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681384583; x=1681470983; bh=Yoq3Ol0C/r5DI nKrvIhJCc4iy6qTwlwOhLMm28kwb68=; b=WUpTiQx7oX1smVLyq4nGOGBKIVQBw wLKPXbTMLCsi6O8LjsHIXz3s3Urp20+TNCchSE7dcSaVuWQPWI8qNRupjWS6/RI4 ZKQgPlJvGx8squbf3KkwWVjelZSs0NRcgyj6TnpVyIwiM40H/jElrrXpIWasrIpb kv4gsku4LzzQgtXZMItdrOnPR5mVtdGZ+ebwWscx0bhiScF5YUAAYz/Aie6s30o8 jRRu0MlFlILyq9w6yH+y10XrZ6IiRhutmNtLcbjkeTaGhdQnoOYQ8n9zYhqkq8Ut O379ut2aNqvX6L42TjFehZnEeI/5qJNei5sjst2jJL9pzYWb7toN0ORYw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekkedgfeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedvnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 13 Apr 2023 07:16:21 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id 3c0b2771 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 13 Apr 2023 11:15:58 +0000 (UTC) Date: Thu, 13 Apr 2023 13:16:19 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v3 04/10] pack-objects: split out `--stdin-packs` tests into separate file Message-ID: <8d47d753dc03979bd515f473a591906ba03c0c2b.1681384405.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The test suite for git-pack-objects(1) is quite huge, and we're about to add more tests that relate to the `--stdin-packs` option. Split out all tests related to this option into a standalone file so that it becomes easier to test the feature in isolation. Signed-off-by: Patrick Steinhardt --- t/t5300-pack-object.sh | 135 ------------------------------- t/t5331-pack-objects-stdin.sh | 145 ++++++++++++++++++++++++++++++++++ 2 files changed, 145 insertions(+), 135 deletions(-) create mode 100755 t/t5331-pack-objects-stdin.sh diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh index f8a0f309e2..d2ce236d61 100755 --- a/t/t5300-pack-object.sh +++ b/t/t5300-pack-object.sh @@ -589,141 +589,6 @@ test_expect_success 'prefetch objects' ' test_line_count = 1 donelines ' -test_expect_success 'setup for --stdin-packs tests' ' - git init stdin-packs && - ( - cd stdin-packs && - - test_commit A && - test_commit B && - test_commit C && - - for id in A B C - do - git pack-objects .git/objects/pack/pack-$id \ - --incremental --revs <<-EOF || exit 1 - refs/tags/$id - EOF - done && - - ls -la .git/objects/pack - ) -' - -test_expect_success '--stdin-packs with excluded packs' ' - ( - cd stdin-packs && - - PACK_A="$(basename .git/objects/pack/pack-A-*.pack)" && - PACK_B="$(basename .git/objects/pack/pack-B-*.pack)" && - PACK_C="$(basename .git/objects/pack/pack-C-*.pack)" && - - git pack-objects test --stdin-packs <<-EOF && - $PACK_A - ^$PACK_B - $PACK_C - EOF - - ( - git show-index <$(ls .git/objects/pack/pack-A-*.idx) && - git show-index <$(ls .git/objects/pack/pack-C-*.idx) - ) >expect.raw && - git show-index <$(ls test-*.idx) >actual.raw && - - cut -d" " -f2 expect && - cut -d" " -f2 actual && - test_cmp expect actual - ) -' - -test_expect_success '--stdin-packs is incompatible with --filter' ' - ( - cd stdin-packs && - test_must_fail git pack-objects --stdin-packs --stdout \ - --filter=blob:none err && - test_i18ngrep "cannot use --filter with --stdin-packs" err - ) -' - -test_expect_success '--stdin-packs is incompatible with --revs' ' - ( - cd stdin-packs && - test_must_fail git pack-objects --stdin-packs --revs out \ - err && - test_i18ngrep "cannot use internal rev list with --stdin-packs" err - ) -' - -test_expect_success '--stdin-packs with loose objects' ' - ( - cd stdin-packs && - - PACK_A="$(basename .git/objects/pack/pack-A-*.pack)" && - PACK_B="$(basename .git/objects/pack/pack-B-*.pack)" && - PACK_C="$(basename .git/objects/pack/pack-C-*.pack)" && - - test_commit D && # loose - - git pack-objects test2 --stdin-packs --unpacked <<-EOF && - $PACK_A - ^$PACK_B - $PACK_C - EOF - - ( - git show-index <$(ls .git/objects/pack/pack-A-*.idx) && - git show-index <$(ls .git/objects/pack/pack-C-*.idx) && - git rev-list --objects --no-object-names \ - refs/tags/C..refs/tags/D - - ) >expect.raw && - ls -la . && - git show-index <$(ls test2-*.idx) >actual.raw && - - cut -d" " -f2 expect && - cut -d" " -f2 actual && - test_cmp expect actual - ) -' - -test_expect_success '--stdin-packs with broken links' ' - ( - cd stdin-packs && - - # make an unreachable object with a bogus parent - git cat-file -p HEAD >commit && - sed "s/$(git rev-parse HEAD^)/$(test_oid zero)/" in && - - git pack-objects .git/objects/pack/pack-D expect.raw && - git show-index <$(ls test3-*.idx) >actual.raw && - - cut -d" " -f2 expect && - cut -d" " -f2 actual && - test_cmp expect actual - ) -' - test_expect_success 'negative window clamps to 0' ' git pack-objects --progress --window=-1 neg-window stderr && check_deltas stderr = 0 diff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh new file mode 100755 index 0000000000..d5eece5899 --- /dev/null +++ b/t/t5331-pack-objects-stdin.sh @@ -0,0 +1,145 @@ +#!/bin/sh + +test_description='pack-objects --stdin' +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +TEST_PASSES_SANITIZE_LEAK=true +. ./test-lib.sh + +test_expect_success 'setup for --stdin-packs tests' ' + git init stdin-packs && + ( + cd stdin-packs && + + test_commit A && + test_commit B && + test_commit C && + + for id in A B C + do + git pack-objects .git/objects/pack/pack-$id \ + --incremental --revs <<-EOF || exit 1 + refs/tags/$id + EOF + done && + + ls -la .git/objects/pack + ) +' + +test_expect_success '--stdin-packs with excluded packs' ' + ( + cd stdin-packs && + + PACK_A="$(basename .git/objects/pack/pack-A-*.pack)" && + PACK_B="$(basename .git/objects/pack/pack-B-*.pack)" && + PACK_C="$(basename .git/objects/pack/pack-C-*.pack)" && + + git pack-objects test --stdin-packs <<-EOF && + $PACK_A + ^$PACK_B + $PACK_C + EOF + + ( + git show-index <$(ls .git/objects/pack/pack-A-*.idx) && + git show-index <$(ls .git/objects/pack/pack-C-*.idx) + ) >expect.raw && + git show-index <$(ls test-*.idx) >actual.raw && + + cut -d" " -f2 expect && + cut -d" " -f2 actual && + test_cmp expect actual + ) +' + +test_expect_success '--stdin-packs is incompatible with --filter' ' + ( + cd stdin-packs && + test_must_fail git pack-objects --stdin-packs --stdout \ + --filter=blob:none err && + test_i18ngrep "cannot use --filter with --stdin-packs" err + ) +' + +test_expect_success '--stdin-packs is incompatible with --revs' ' + ( + cd stdin-packs && + test_must_fail git pack-objects --stdin-packs --revs out \ + err && + test_i18ngrep "cannot use internal rev list with --stdin-packs" err + ) +' + +test_expect_success '--stdin-packs with loose objects' ' + ( + cd stdin-packs && + + PACK_A="$(basename .git/objects/pack/pack-A-*.pack)" && + PACK_B="$(basename .git/objects/pack/pack-B-*.pack)" && + PACK_C="$(basename .git/objects/pack/pack-C-*.pack)" && + + test_commit D && # loose + + git pack-objects test2 --stdin-packs --unpacked <<-EOF && + $PACK_A + ^$PACK_B + $PACK_C + EOF + + ( + git show-index <$(ls .git/objects/pack/pack-A-*.idx) && + git show-index <$(ls .git/objects/pack/pack-C-*.idx) && + git rev-list --objects --no-object-names \ + refs/tags/C..refs/tags/D + + ) >expect.raw && + ls -la . && + git show-index <$(ls test2-*.idx) >actual.raw && + + cut -d" " -f2 expect && + cut -d" " -f2 actual && + test_cmp expect actual + ) +' + +test_expect_success '--stdin-packs with broken links' ' + ( + cd stdin-packs && + + # make an unreachable object with a bogus parent + git cat-file -p HEAD >commit && + sed "s/$(git rev-parse HEAD^)/$(test_oid zero)/" in && + + git pack-objects .git/objects/pack/pack-D expect.raw && + git show-index <$(ls test3-*.idx) >actual.raw && + + cut -d" " -f2 expect && + cut -d" " -f2 actual && + test_cmp expect actual + ) +' + +test_done From patchwork Thu Apr 13 11:16:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13210162 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCA79C77B6C for ; Thu, 13 Apr 2023 11:16:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231253AbjDMLQ6 (ORCPT ); Thu, 13 Apr 2023 07:16:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231207AbjDMLQm (ORCPT ); Thu, 13 Apr 2023 07:16:42 -0400 Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FE7BA265 for ; Thu, 13 Apr 2023 04:16:28 -0700 (PDT) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id C67415C00B9; Thu, 13 Apr 2023 07:16:27 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Thu, 13 Apr 2023 07:16:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681384587; x=1681470987; bh=YD +uzFYy86g02M+Kx5Mt4oxEeBv89essUbFzbMywJUg=; b=rW02CeqR2AaygisN15 WvvdqcgrXXhkOH+l7T9h22MaEro2z0TO+HkWg0zRLELKn1+tK1h+pkcYUHm0WvNw 5BtgLHXueBdJ5sRQXLrtw2MJWRjWOYS5cvvRg/Gr0Eeh5axjhGh1peQmngO5Eb0U YMjKg2Y0kfgVvw5LJmHFPBFmffyK/7rlmxKNVNcYoyJ9p2ICC8yPJfnsuUJA/YBq Kt4rsSN1p6HsQKFXpaZ8sN8qdsgaauhrqTW4Q4McJ1RKkfSi3ZTvdMjL4e8inRHr DvW77mTFZoYPOIfH+OctjrrnE/fwDv9l+qQKqpRkSqmpVeg5kEr0H+gOw1hHrTFA fL4w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681384587; x=1681470987; bh=YD+uzFYy86g02 M+Kx5Mt4oxEeBv89essUbFzbMywJUg=; b=IMH/6rsl9hFSDLPe7SgbpNQ35Yy4U vYG9rXouhu7RVWXFCdRoflnV7U5q1Vj9B0r6HbeiswyjE23fpQ7hpCe6lO/X72uE 5szsYe5yG2KHnCPA6u8aZxMMyymH0et05MLY8hcz2e/d/qxs+ypp7U8N1zP8RgoZ ZN9PmOeWJNpqhVuPxTXMbiugzHayEBiqV/JSLY50nCZxH2fkSVde3K6bNEFukarv eUWFC6J0RnV5JylqOWjbUcl6EiOMao+bj+kUjuE3X6I20HdMsNfZSq3YWHqccKjA LeO4nVYqSHc5dl+kDZGLzSmv+/ZBrPWOfO8mUEFCIDqiBArO7Hfeh5SrQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekkedgfeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 13 Apr 2023 07:16:26 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id 3c5cc697 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 13 Apr 2023 11:16:03 +0000 (UTC) Date: Thu, 13 Apr 2023 13:16:24 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v3 05/10] pack-objects: fix error when packing same pack twice Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When passed the same packfile twice via `--stdin-packs` we return an error that the packfile supposedly was not found. This is because when reading packs into the list of included or excluded packfiles, we will happily re-add packfiles even if they are part of the lists already. And while the list can now contain duplicates, we will only set the `util` pointer of the first list entry to the `packed_git` structure. We notice that at a later point when checking that all list entries have their `util` pointer set and die with an error. While this is kind of a nonsensical request, this scenario can be hit when doing geometric repacks. When a repository is connected to an alternate object directory and both have the exact same packfile then both would get added to the geometric sequence. And when we then decide to perform the repack, we will invoke git-pack-objects(1) with the same packfile twice. Fix this bug by removing any duplicates from both the included and excluded packs. Signed-off-by: Patrick Steinhardt --- builtin/pack-objects.c | 2 ++ t/t5331-pack-objects-stdin.sh | 27 +++++++++++++++++++++++++++ t/t7703-repack-geometric.sh | 25 +++++++++++++++++++++++++ 3 files changed, 54 insertions(+) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 7e9e20172a..7d0e864c35 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3348,7 +3348,9 @@ static void read_packs_list_from_stdin(void) } string_list_sort(&include_packs); + string_list_remove_duplicates(&include_packs, 0); string_list_sort(&exclude_packs); + string_list_remove_duplicates(&exclude_packs, 0); for (p = get_all_packs(the_repository); p; p = p->next) { const char *pack_name = pack_basename(p); diff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh index d5eece5899..def7640c91 100755 --- a/t/t5331-pack-objects-stdin.sh +++ b/t/t5331-pack-objects-stdin.sh @@ -7,6 +7,12 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME TEST_PASSES_SANITIZE_LEAK=true . ./test-lib.sh +packed_objects() { + git show-index <"$1" >tmp-object-list && + cut -d' ' -f2 tmp-object-list | sort && + rm tmp-object-list + } + test_expect_success 'setup for --stdin-packs tests' ' git init stdin-packs && ( @@ -142,4 +148,25 @@ test_expect_success '--stdin-packs with broken links' ' ) ' +test_expect_success 'pack-objects --stdin with duplicate packfile' ' + test_when_finished "rm -fr repo" && + + git init repo && + ( + cd repo && + test_commit "commit" && + git repack -ad && + + { + basename .git/objects/pack/pack-*.pack && + basename .git/objects/pack/pack-*.pack + } >packfiles && + + git pack-objects --stdin-packs generated-pack expect && + packed_objects generated-pack-*.idx >actual && + test_cmp expect actual + ) +' + test_done diff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh index 7027676977..d0823f2eb2 100755 --- a/t/t7703-repack-geometric.sh +++ b/t/t7703-repack-geometric.sh @@ -336,4 +336,29 @@ test_expect_success '--geometric --with-midx with no local objects' ' test_dir_is_empty member/$packdir ' +test_expect_success '--geometric with same pack in main and alternate ODB' ' + test_when_finished "rm -fr shared member" && + + # Create a repository with a single packfile that acts as alternate + # object database. + git init shared && + test_commit -C shared "shared-objects" && + git -C shared repack -ad && + + # We create the member repository as an exact copy so that it has the + # same packfile. + cp -r shared member && + test-tool path-utils real_path shared/.git/objects >member/.git/objects/info/alternates && + find shared/.git/objects -type f >expected-files && + + # Verify that we can repack objects as expected without observing any + # error. Having the same packfile in both ODBs used to cause an error + # in git-pack-objects(1). + git -C member repack --geometric 2 2>err && + test_must_be_empty err && + # Nothing should have changed. + find shared/.git/objects -type f >actual-files && + test_cmp expected-files actual-files +' + test_done From patchwork Thu Apr 13 11:16:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13210163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE3E8C77B6E for ; Thu, 13 Apr 2023 11:17:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230499AbjDMLRG (ORCPT ); Thu, 13 Apr 2023 07:17:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230257AbjDMLQw (ORCPT ); Thu, 13 Apr 2023 07:16:52 -0400 Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56ABBA24E for ; Thu, 13 Apr 2023 04:16:32 -0700 (PDT) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 809BD5C0076; Thu, 13 Apr 2023 07:16:31 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Thu, 13 Apr 2023 07:16:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681384591; x=1681470991; bh=9s qLGlAbOJqQ1Q/dr20+KitMYFnrWMdZfkZS7V6pQJ4=; b=Xw31XP3jIfe0wxYuno pvK7GjfUwx4t8aTPpI5levuQJb80RKH25xBhYvMCm8qtUgirvHCYadKXCebQGiPm l0WxEgJdm98PG1JgZpgF4f/zitHfzmsQXehdilq2vMoMqcx/LOGMHeBn0+7BaACw AN/Dnl3+hCb8RwWid28XL8E4oV28zY3HJOdnlGKmPfjLzTBDHYAAXjonLx7h8Rbe /wW/UHxOmiQ7plf/x1COBFWz1ABEfcAvp0MqjJxLqC7MkBGjOfWLD4Hda4jjf1Uw 2VEFH5vKzwD40vJ84rd4SgkG2IF8er1Wwp1u9U8KnUMDlYen122tQha+Xbyii1WG e56Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681384591; x=1681470991; bh=9sqLGlAbOJqQ1 Q/dr20+KitMYFnrWMdZfkZS7V6pQJ4=; b=a3freVCd7OKzRFiwHXFETe6borLQ+ aA7ZYhKb8SCZ8sBYYTrqJ+IlnAaF+IW1w/30D/g30H/z7PbfCyzCjY3PaJXXhDtI YNiyq1FGGgdzk7hP+rbcD7OJsK4+nADoCak961dOENZxFA4UzcaR8l2Z3rK2Fm9g ylmR2a6cdXq0b8xZcUoVnRwlvcBSyXEEwbwRVBpUmVouuy/ovktMsYjhaKM6P/pd oKhctRuRTIFAVW8w8alJdJKf8L2M5KBNuF5DXPlJiT7EhC9cgAaymTwS5A27IL3u VlIoTCBhLmUlX1rOS3/KKkhX5K8e1aJLRQuqgyxUFyuEhW069d4CF7QYA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekkedgfeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 13 Apr 2023 07:16:29 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id e97d63e3 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 13 Apr 2023 11:16:07 +0000 (UTC) Date: Thu, 13 Apr 2023 13:16:28 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v3 06/10] pack-objects: fix error when same packfile is included and excluded Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When passing the same packfile both as included and excluded via the `--stdin-packs` option, then we will return an error because the excluded packfile cannot be found. This is because we will only set the `util` pointer for the included packfile list if it was found, so that we later die when we notice that it's in fact not set for the excluded packfile list. Fix this bug by always setting the `util` pointer for both the included and excluded list entries. Signed-off-by: Patrick Steinhardt --- builtin/pack-objects.c | 8 +++----- t/t5331-pack-objects-stdin.sh | 20 ++++++++++++++++++++ 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 7d0e864c35..c5147d392f 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3355,11 +3355,9 @@ static void read_packs_list_from_stdin(void) for (p = get_all_packs(the_repository); p; p = p->next) { const char *pack_name = pack_basename(p); - item = string_list_lookup(&include_packs, pack_name); - if (!item) - item = string_list_lookup(&exclude_packs, pack_name); - - if (item) + if ((item = string_list_lookup(&include_packs, pack_name))) + item->util = p; + if ((item = string_list_lookup(&exclude_packs, pack_name))) item->util = p; } diff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh index def7640c91..1e23cc2e61 100755 --- a/t/t5331-pack-objects-stdin.sh +++ b/t/t5331-pack-objects-stdin.sh @@ -169,4 +169,24 @@ test_expect_success 'pack-objects --stdin with duplicate packfile' ' ) ' +test_expect_success 'pack-objects --stdin with same packfile excluded and included' ' + test_when_finished "rm -fr repo" && + + git init repo && + ( + cd repo && + test_commit "commit" && + git repack -ad && + + { + basename .git/objects/pack/pack-*.pack && + printf "^%s\n" "$(basename .git/objects/pack/pack-*.pack)" + } >packfiles && + + git pack-objects --stdin-packs generated-pack packed-objects && + test_must_be_empty packed-objects + ) +' + test_done From patchwork Thu Apr 13 11:16:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13210164 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3D40C77B61 for ; Thu, 13 Apr 2023 11:17:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231322AbjDMLRQ (ORCPT ); Thu, 13 Apr 2023 07:17:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231232AbjDMLQ5 (ORCPT ); Thu, 13 Apr 2023 07:16:57 -0400 Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 979D976AD for ; Thu, 13 Apr 2023 04:16:36 -0700 (PDT) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 0F8E45C00A7; Thu, 13 Apr 2023 07:16:36 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Thu, 13 Apr 2023 07:16:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681384596; x=1681470996; bh=m8 uCdIrVFWBb08sQaxzQA0rAW3FwHkL4jFTfYVo8NYU=; b=A6FSuem9NRkzfeayBz DY78Z78tLfWWJCHTZfoj+EfV3sE0eHv2Lu78RUiFwvF1AfhhYL2u8LbeVhUb42oo nyFMFqzPq1yru1pV7dLTd4+gmwC5Do+uvJ1USELiHvyNZUWSogM+bqTkxVHrZabV O5F9n1w7VqJAgzEqbb8MWwRsQZyQ0xmkaHsEkc7Exe/pn0bD0vvL1TlYQvOsBpuy MVqAkOlpvMaqzkJtYDqU9ac9y1m6Z+CWX4AWcfpMawD4Ly9rSlHe101cpEabKs3d U7T/qh7o/xWDrX0boPYA8NlzHmGjFic8u/0h/TUIE869NDEMxlwLFKwzlxWzaHpE 9xMg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681384596; x=1681470996; bh=m8uCdIrVFWBb0 8sQaxzQA0rAW3FwHkL4jFTfYVo8NYU=; b=SIA+k55OaEFBul7esZcugTOTpymJq Q7/OQTTaObh5G2AuylbJCMOm2+rO0YELp043TtEY2BSmySmO4cyLJIqTqvDa40xZ WdS+w0RUwT6qwALG3MullYLEwCJmfWEhk5PUdcMyNn/I2jvdAjljWai5DGUM/6ej a3MD8oCcmyUcbRWbRvTlDLtXMqlc1VtnOEsCtHrAHNmK33c9jUcluC+i/5st05ex Q+D21stI6jfoZ1VYIt16l8qKjjHQCGGVrLYtfbNIQOdxh+8oyC3pOgQBQ5Rq4LwO Q4wcI9b6KEKzR5kEzcBzxMm71BXY1zfRUl5xFxbijOB4bYzRsBTFsCO6A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekkedgfeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedunecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 13 Apr 2023 07:16:34 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id b692c1da (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 13 Apr 2023 11:16:11 +0000 (UTC) Date: Thu, 13 Apr 2023 13:16:32 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v3 07/10] pack-objects: extend test coverage of `--stdin-packs` with alternates Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org We don't have any tests that verify that git-pack-objects(1) works with `--stdin-packs` when combined with alternate object directories. Add some to make sure that the basic functionality works as expected. Signed-off-by: Patrick Steinhardt --- t/t5331-pack-objects-stdin.sh | 48 +++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh index 1e23cc2e61..45e24fa94a 100755 --- a/t/t5331-pack-objects-stdin.sh +++ b/t/t5331-pack-objects-stdin.sh @@ -189,4 +189,52 @@ test_expect_success 'pack-objects --stdin with same packfile excluded and includ ) ' +test_expect_success 'pack-objects --stdin with packfiles from alternate object database' ' + test_when_finished "rm -fr shared member" && + + # Set up a shared repository with a single packfile. + git init shared && + test_commit -C shared "shared-objects" && + git -C shared repack -ad && + basename shared/.git/objects/pack/pack-*.pack >packfile && + + # Set up a repository that is connected to the shared repository. This + # repository has no objects on its own, but we still expect to be able + # to pack objects from its alternate. + git clone --shared shared member && + git -C member pack-objects --stdin-packs generated-pack packfiles && + + { + packed_objects shared/.git/objects/pack/pack-*.idx && + packed_objects member/.git/objects/pack/pack-*.idx + } | sort >expected-objects && + + git -C member pack-objects --stdin-packs generated-pack actual-objects && + test_cmp expected-objects actual-objects +' + test_done From patchwork Thu Apr 13 11:16:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13210165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B36CC77B61 for ; Thu, 13 Apr 2023 11:17:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231344AbjDMLRT (ORCPT ); Thu, 13 Apr 2023 07:17:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230486AbjDMLQ7 (ORCPT ); Thu, 13 Apr 2023 07:16:59 -0400 Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 103139777 for ; Thu, 13 Apr 2023 04:16:41 -0700 (PDT) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 7084E5C0143; Thu, 13 Apr 2023 07:16:40 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Thu, 13 Apr 2023 07:16:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681384600; x=1681471000; bh=2w GOYkw5nzkZOfVF8Exuyo2UFYPMtUKFIpuZpl00aEY=; b=cg9XFQjpn0/OhxaPks r1wXUtQrzKVd62LF8HQQPz+3WLQxMtlCdpqbjj09jtP9n4sgEaJ3LctMF0CS23fI lBs2TH9cVihPWtJiWExD/vViH6VwIftP5Kqn3j3cOXM/Q5aGOq+DrFLid8QSCIK7 VaYq3J8NepilchCkzYtJrM8t1iM+UM5A9ElPARPBpUHb1TgAqD3/9aFmyICyLUDq Ir/Rb5Id5kxJShLiraYa8YNWkYEflVzWToJHT4NvJiWe9SSl9gUm86rbuKaKBnSp FPuJyDbnOBWWSXFIzlQYZJHkUxevgxJ5hj9BzZ7pKl72Nn9hsIAX8ilAgMzn9OK9 H2Lg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681384600; x=1681471000; bh=2wGOYkw5nzkZO fVF8Exuyo2UFYPMtUKFIpuZpl00aEY=; b=G8clR7zTYHCruGhwvHf5y1MyMUYqb qqRwOsuQPIVKJNib3LpUzVSvFyO5H41MWbe+zqZZnCUnlSSqE7AjdCIcFfvWb9EL MJ4prytinfZirmwt+b4zEbrekuoaFmKxWLEUQOMjDz5pfuBk3kB4uUncGOclU8Z3 4JW9bCQoR4xMrLifwDkA8ZGuchwIpDR5utp2abUXHojVdVN5hiUo4Z7A8hKp2Xih rIDk0KSaSC5tfL+8S7EsWPzwM/P8dnw9y9cgDjicOYXgfhhscO/95OAy1l1XIWVt l8GAaIuB9Jl+Gf9srrVUv+dR+doo+D8/4ITjiy4BZsUczQCmCcVG9/S6A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekkedgfeekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedunecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 13 Apr 2023 07:16:39 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id ac0cb7ca (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 13 Apr 2023 11:16:16 +0000 (UTC) Date: Thu, 13 Apr 2023 13:16:37 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v3 08/10] t/helper: allow chmtime to print verbosely without modifying mtime Message-ID: <18bac902896d6955a359093c4bb072166209ffc4.1681384405.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The `test-tool chmtime` helper allows us to both read and modify the modification time of files. But while it is possible to only read the mtimes of a file via `--get`, it is not possible to read the mtimes and report them together with their respective file paths via the `--verbose` flag without also modifying the mtime at the same time. Fix this so that it is possible to call `test-tool chmtime --verbose ...` without modifying any mtimes. Signed-off-by: Patrick Steinhardt --- t/helper/test-chmtime.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/helper/test-chmtime.c b/t/helper/test-chmtime.c index dc28890a18..0e5538833a 100644 --- a/t/helper/test-chmtime.c +++ b/t/helper/test-chmtime.c @@ -94,7 +94,7 @@ int cmd__chmtime(int argc, const char **argv) if (timespec_arg(argv[i], &set_time, &set_eq)) { ++i; } else { - if (get == 0) { + if (get == 0 && verbose == 0) { fprintf(stderr, "Not a base-10 integer: %s\n", argv[i] + 1); goto usage; } From patchwork Thu Apr 13 11:16:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13210166 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CB46C77B61 for ; Thu, 13 Apr 2023 11:17:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231364AbjDMLR1 (ORCPT ); Thu, 13 Apr 2023 07:17:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230064AbjDMLRF (ORCPT ); Thu, 13 Apr 2023 07:17:05 -0400 Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 938D69EC4 for ; Thu, 13 Apr 2023 04:16:45 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 0A3255C0106; Thu, 13 Apr 2023 07:16:45 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Thu, 13 Apr 2023 07:16:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681384605; x=1681471005; bh=DA is+3XDiFN27zOmt3TK1xaki9YbfH4QNk9jA8rOj0k=; b=SLurKIAnCHLf8uY+fi z2x+V9GkSMRrB4y5HW+VVXIOOzE67zIuTdOOz6zl7WW0F066Z4J4eZcDxdMgV4Av m2a3XMej8czAhKssXT4Coppzh40BTRdA7JE1k+Ta2v+C8CZAhY0j2siHXt8sDHs7 McTE89mUBRVsE4QRdqXSpqyR91H3071p1cK6ZrlueIRAqcmuOMMLEbeg5C/RC2fV MrMot81qpsB6IeMDCJ/zlIwwWZEkqTzW+2N7mZXRQBbH8NmcfaqoyF/6wvozH0sc qhzM+umh16RDJI69jTzuzppLPiEekUUBJQLuJPV3HhoAraf8gbEVD9dELkR2/KpP vJOQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681384605; x=1681471005; bh=DAis+3XDiFN27 zOmt3TK1xaki9YbfH4QNk9jA8rOj0k=; b=A/1wW/kuwLibBwyl+vHc0BtqxRozC 0tKo/JPAH5nqJr4xYu7T2vHGalXj80AXG+UM3kRpKnwZDhSEjxaa8PvPFa4dUIMZ frem+C94FNKXXNHAhnfH8++KVvuf2o7k6nKkXMWhPKBWIpL3fGWleVgNhGr8kMMT 1y45JcYHmtQ0y9eo09EM6LgYmmCCQKX22XeAC36JsLbOLANZOyT9vraGZY6JO1l1 YWPzeOne/cpEF6c4hj7VrFqkaxTuhl3CFGiVBs9rUMT4z5jdAQlN4D4x9LG6Dyi0 u44se2ph4OFOWTexI4aYpTWePjD9+rA8rGRdgrHr/nKYy6PDHpspRvtpQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekkedgfeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedunecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 13 Apr 2023 07:16:43 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id 012e6f1b (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 13 Apr 2023 11:16:21 +0000 (UTC) Date: Thu, 13 Apr 2023 13:16:42 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v3 09/10] repack: honor `-l` when calculating pack geometry Message-ID: <285695deafa5a4a49f774dc484782dd8e4fd4997.1681384405.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When the user passes `-l` to git-repack(1), then they essentially ask us to only repack objects part of the local object database while ignoring any packfiles part of an alternate object database. And we in fact honor this bit when doing a geometric repack as the resulting packfile will only ever contain local objects. What we're missing though is that we don't take locality of packfiles into account when computing whether the geometric sequence is intact or not. So even though we would only ever roll up local packfiles anyway, we could end up trying to repack because of non-local packfiles. This does not make much sense, and in the worst case it can cause us to try and do the geometric repack over and over again because we're never able to restore the geometric sequence. Fix this bug by honoring whether the user has passed `-l`. If so, we skip adding any non-local packfiles to the pack geometry. Signed-off-by: Patrick Steinhardt --- builtin/repack.c | 13 +++++++-- t/t7703-repack-geometric.sh | 58 +++++++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+), 2 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index 80fc860613..5768aeb1f3 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -323,7 +323,8 @@ static int geometry_cmp(const void *va, const void *vb) } static void init_pack_geometry(struct pack_geometry **geometry_p, - struct string_list *existing_kept_packs) + struct string_list *existing_kept_packs, + const struct pack_objects_args *args) { struct packed_git *p; struct pack_geometry *geometry; @@ -333,6 +334,14 @@ static void init_pack_geometry(struct pack_geometry **geometry_p, geometry = *geometry_p; for (p = get_all_packs(the_repository); p; p = p->next) { + if (args->local && !p->pack_local) + /* + * When asked to only repack local packfiles we skip + * over any packfiles that are borrowed from alternate + * object directories. + */ + continue; + if (!pack_kept_objects) { /* * Any pack that has its pack_keep bit set will appear @@ -900,7 +909,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (geometric_factor) { if (pack_everything) die(_("options '%s' and '%s' cannot be used together"), "--geometric", "-A/-a"); - init_pack_geometry(&geometry, &existing_kept_packs); + init_pack_geometry(&geometry, &existing_kept_packs, &po_args); split_pack_geometry(geometry, geometric_factor); } diff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh index d0823f2eb2..c440956ad5 100755 --- a/t/t7703-repack-geometric.sh +++ b/t/t7703-repack-geometric.sh @@ -10,6 +10,12 @@ objdir=.git/objects packdir=$objdir/pack midx=$objdir/pack/multi-pack-index +packed_objects() { + git show-index <"$1" >tmp-object-list && + cut -d' ' -f2 tmp-object-list && + rm tmp-object-list + } + test_expect_success '--geometric with no packs' ' git init geometric && test_when_finished "rm -fr geometric" && @@ -361,4 +367,56 @@ test_expect_success '--geometric with same pack in main and alternate ODB' ' test_cmp expected-files actual-files ' +test_expect_success '--geometric -l with non-intact geometric sequence across ODBs' ' + test_when_finished "rm -fr shared member" && + + git init shared && + test_commit_bulk -C shared --start=1 1 && + + git clone --shared shared member && + test_commit_bulk -C member --start=2 1 && + + # Verify that our assumptions actually hold: both generated packfiles + # should have three objects and should be non-equal. + packed_objects shared/.git/objects/pack/pack-*.idx >packed-objects && + test_line_count = 3 packed-objects && + packed_objects member/.git/objects/pack/pack-*.idx >packed-objetcs && + test_line_count = 3 packed-objects && + test "$(basename member/.git/objects/pack/pack-*.pack)" != "$(basename shared/.git/objects/pack/pack-*.pack)" && + + # Perform the geometric repack. With `-l`, we should only see the local + # packfile and thus arrive at the conclusion that the geometric + # sequence is intact. We thus expect no changes. + # + # Note that we are tweaking mtimes of the packfiles so that we can + # verify they did not change. This is done in order to detect the case + # where we do repack objects, but the resulting packfile is the same. + test-tool chmtime --verbose =0 member/.git/objects/pack/* >expected-member-packs && + git -C member repack --geometric=2 -l -d && + test-tool chmtime --verbose member/.git/objects/pack/* >actual-member-packs && + test_cmp expected-member-packs actual-member-packs && + + { + packed_objects shared/.git/objects/pack/pack-*.idx && + packed_objects member/.git/objects/pack/pack-*.idx + } | sort >expected-objects && + + # On the other hand, when doing a non-local geometric repack we should + # see both packfiles and thus repack them. We expect that the shared + # object database was not changed. + test-tool chmtime --verbose =0 shared/.git/objects/pack/* >expected-shared-packs && + git -C member repack --geometric=2 -d && + test-tool chmtime --verbose shared/.git/objects/pack/* >actual-shared-packs && + test_cmp expected-shared-packs actual-shared-packs && + + # Furthermore, we expect that the member repository now has a single + # packfile that contains the combined shared and non-shared objects. + ls member/.git/objects/pack/pack-*.idx >actual && + test_line_count = 1 actual && + packed_objects member/.git/objects/pack/pack-*.idx >actual-objects && + test_line_count = 6 actual-objects && + sort actual-objects.sorted && + test_cmp expected-objects actual-objects.sorted +' + test_done From patchwork Thu Apr 13 11:16:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13210167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78315C77B61 for ; Thu, 13 Apr 2023 11:17:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231370AbjDMLRb (ORCPT ); Thu, 13 Apr 2023 07:17:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51074 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229853AbjDMLRJ (ORCPT ); Thu, 13 Apr 2023 07:17:09 -0400 Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80D797D83 for ; Thu, 13 Apr 2023 04:16:50 -0700 (PDT) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 9CD0C5C00B3; Thu, 13 Apr 2023 07:16:49 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Thu, 13 Apr 2023 07:16:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1681384609; x=1681471009; bh=dW TeKZX6b1nxaCGNtlKWBn4EL7y6fkExTN2HM7HHgzA=; b=fIcR348dMoeedfJGRZ iK94sYO2o0cNGP2t6y5go2Avlw7msBEvZfs+HEN9aOdZdulVHly6cg9ehnmpA0aq 9/zBfdcNj06OmY5KLyqlKdXaIhN8h4ICRQ1+HI23WE5Vk9V2rlvjKlatQRuv1DAY /BFL4NucFmVIB/a3xbhpNITCT8ZiFqMSQpY5GrTJAG8wUa4N2PlRSYov6RwL47Re qkF/tRf5rMm8nL53WfF/dfCAYqtsq/VevKzjb8arfdbqyEtZx/oTsfADi8W5WzZa 4mhFrPU8OL28DJQZBUm3CH9enje0pVClIjGWReUUuJrIelphX6HkWh6NWImm76z0 M2cg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1681384609; x=1681471009; bh=dWTeKZX6b1nxa CGNtlKWBn4EL7y6fkExTN2HM7HHgzA=; b=UbUdPISzq+B5yaU6CzsCFxoV53RcV 7v2tz4khrZjfvMjadB+ZW8EckKlzL2Ymx/FRgm0yzjj9MrOLVWUPkJuck30eu/Qd v9IPSD0mX8Cp/3OwyqH8ROP7kb72yJlule/IvVhtjK+NfIz3rYutxpPoGWMHrMGg SPvX+/V7bpabJM44bp0uWq6zbIiqAkLfxLdp6Em8gQAMQt/TzN0QCAvwS7ORRueg 1wnJGX42UBL0gsXY7yw9Ag7kV3C31J/cN3NhZRwvoL0lBWlimORoU2mAWVQ3tP11 BIJqUTaHC8uTHvBOlu8+VrKByCDvoke9fIGApO5OWwrwIEkWlR/3r3nMg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdekkedgfeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvvefukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeeukedtvedtffevleejtefgheehieegke eluddvfeefgeehgfeltddtheejleffteenucevlhhushhtvghrufhiiigvpedvnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 13 Apr 2023 07:16:48 -0400 (EDT) Received: by pks.im (OpenSMTPD) with ESMTPSA id 238be82d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 13 Apr 2023 11:16:25 +0000 (UTC) Date: Thu, 13 Apr 2023 13:16:46 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Taylor Blau , peff@peff.net, dstolee@microsoft.com Subject: [PATCH v3 10/10] repack: disable writing bitmaps when doing a local repack Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In order to write a bitmap, we need to have full coverage of all objects that are about to be packed. In the traditional non-multi-pack-index world this meant we need to do a full repack of all objects into a single packfile. But in the new multi-pack-index world we can get away with writing bitmaps when we have multiple packfiles as long as the multi-pack-index covers all objects. This is not always the case though. When asked to perform a repack of local objects, only, then we cannot guarantee to have full coverage of all objects regardless of whether we do a full repack or a repack with a multi-pack-index. The end result is that writing the bitmap will fail in both worlds: $ git multi-pack-index write --stdin-packs --bitmap --- builtin/repack.c | 12 ++++++++++++ object-file.c | 6 ++++++ object-store.h | 1 + t/t7700-repack.sh | 17 +++++++++++++++++ t/t7703-repack-geometric.sh | 27 +++++++++++++++++++++++++++ 5 files changed, 63 insertions(+) diff --git a/builtin/repack.c b/builtin/repack.c index 5768aeb1f3..e2abcea2ee 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -887,6 +887,18 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (write_bitmaps && !(pack_everything & ALL_INTO_ONE) && !write_midx) die(_(incremental_bitmap_conflict_error)); + if (write_bitmaps && po_args.local && has_alt_odb(the_repository)) { + /* + * When asked to do a local repack, but we have + * packfiles that are inherited from an alternate, then + * we cannot guarantee that the multi-pack-index would + * have full coverage of all objects. We thus disable + * writing bitmaps in that case. + */ + warning(_("disabling bitmap writing, as some objects are not being packed")); + write_bitmaps = 0; + } + if (write_midx && write_bitmaps) { struct strbuf path = STRBUF_INIT; diff --git a/object-file.c b/object-file.c index 8fab8dbe80..3ad11c683b 100644 --- a/object-file.c +++ b/object-file.c @@ -946,6 +946,12 @@ void prepare_alt_odb(struct repository *r) r->objects->loaded_alternates = 1; } +int has_alt_odb(struct repository *r) +{ + prepare_alt_odb(r); + return !!r->objects->odb->next; +} + /* Returns 1 if we have successfully freshened the file, 0 otherwise. */ static int freshen_file(const char *fn) { diff --git a/object-store.h b/object-store.h index 82201ec3e7..574f16140f 100644 --- a/object-store.h +++ b/object-store.h @@ -56,6 +56,7 @@ KHASH_INIT(odb_path_map, const char * /* key: odb_path */, struct object_directory *, 1, fspathhash, fspatheq) void prepare_alt_odb(struct repository *r); +int has_alt_odb(struct repository *r); char *compute_alternate_path(const char *path, struct strbuf *err); struct object_directory *find_odb(struct repository *r, const char *obj_dir); typedef int alt_odb_fn(struct object_directory *, void *); diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh index 4aabe98139..93c5763e7d 100755 --- a/t/t7700-repack.sh +++ b/t/t7700-repack.sh @@ -106,6 +106,23 @@ test_expect_success SYMLINKS '--local keeps packs when alternate is objectdir ' test_cmp expect actual ' +test_expect_success '--local disables writing bitmaps when connected to alternate ODB' ' + test_when_finished "rm -rf shared member" && + + git init shared && + git clone --shared shared member && + ( + cd member && + test_commit "object" && + git repack -Adl --write-bitmap-index 2>err && + cat >expect <<-EOF && + warning: disabling bitmap writing, as some objects are not being packed + EOF + test_cmp expect err && + test ! -f .git/objects/pack-*.bitmap + ) +' + test_expect_success 'packed obs in alt ODB are repacked even when local repo is packless' ' mkdir alt_objects/pack && mv .git/objects/pack/* alt_objects/pack && diff --git a/t/t7703-repack-geometric.sh b/t/t7703-repack-geometric.sh index c440956ad5..33d7977fca 100755 --- a/t/t7703-repack-geometric.sh +++ b/t/t7703-repack-geometric.sh @@ -419,4 +419,31 @@ test_expect_success '--geometric -l with non-intact geometric sequence across OD test_cmp expected-objects actual-objects.sorted ' +test_expect_success '--geometric -l disables writing bitmaps with non-local packfiles' ' + test_when_finished "rm -fr shared member" && + + git init shared && + test_commit_bulk -C shared --start=1 1 && + + git clone --shared shared member && + test_commit_bulk -C member --start=2 1 && + + # When performing a geometric repack with `-l` while connecting to an + # alternate object database that has a packfile we do not have full + # coverage of objects. As a result, we expect that writing the bitmap + # will be disabled. + git -C member repack -l --geometric=2 --write-midx --write-bitmap-index 2>err && + cat >expect <<-EOF && + warning: disabling bitmap writing, as some objects are not being packed + EOF + test_cmp expect err && + test ! -f member/.git/objects/pack/multi-pack-index-*.bitmap && + + # On the other hand, when we repack without `-l`, we should see that + # the bitmap gets created. + git -C member repack --geometric=2 --write-midx --write-bitmap-index 2>err && + test_must_be_empty err && + test -f member/.git/objects/pack/multi-pack-index-*.bitmap +' + test_done