From patchwork Mon Mar 25 10:02:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601860 Received: from fhigh5-smtp.messagingengine.com (fhigh5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1ECD0194C96 for ; Mon, 25 Mar 2024 10:02:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360963; cv=none; b=fdI+lZGAOcQB4HfsjJpWA0JEyuHNM1jRRGuR89F3T1zQs0zNidmtVA2QUd31RS7HT76V+XNFXuKH1XHNsBlOraP3s3VNpERjZ6ABXDK4ZBIhtJZFXTfJqyI3tcxCUnaGBboI1l+0Nx5sAIxabksdbr9KjjPdGrZiPN4neRFoqqI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360963; c=relaxed/simple; bh=dZ4pqQZymqvXpsigrG8ikRjngF/vRXNi+Mqo7plMoT4=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=iKziFN5TiLTi02p4oZbKkcj19YICasT1iNt+6NrGhpyEABI2atBEVlOqLiVz9CfR0aWBJ8Yhd+y7d1x9slD6n791z1VvFri3Wbeq4QQuxA5heIbL8175cde8pOteGtbvpZYQxrP7ZZ7KwwiaBSVA0PjY/TGpojaz4ZSgT8v2IIo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=Mpljk1hA; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=ty6U9P25; arc=none smtp.client-ip=103.168.172.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="Mpljk1hA"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="ty6U9P25" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 3C44011400A9; Mon, 25 Mar 2024 06:02:41 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 25 Mar 2024 06:02:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711360961; x=1711447361; bh=UmzKqMBWIm 0m4AKIDI9HwXPX3O32Gm8aUhqiTlzRfrw=; b=Mpljk1hAAgWXmLFgUGr2zD2dn1 a3FvnNgbqicMe6+SgQYYbJyQYD2cvl9E5u6hLFFZO0ZlKt+ijYo5xa/VELW2Nwa+ 5Wi7VwHmNBsIvYD6UloJQwjvOwKIX9Sb2eBcMNPq7oV0yQ6nx2+NfrCmbGavvyJW hg8yoQPrdWhbcSsrZT05lfKvQ+k/jKN+d81GmotBIknkvIeZu5to32CaCigxWmP1 eTZBlm0Nm+PQO7fy5yDP4NiI8Aa0yX78unJ8DgcZEINLYPuT0Qk7lMeVaN50wgh3 cdIOBYXFQ48qYKlpjyC13jOtfl6BMunnSzjjeRyt+Qu0BXPr9vKRYk7icRQA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711360961; x=1711447361; bh=UmzKqMBWIm0m4AKIDI9HwXPX3O32 Gm8aUhqiTlzRfrw=; b=ty6U9P25Esj9oIdbM0dGz6PsxOxc+Xy8OWEXb68zrPmp V/OQzWMvNw/c3W2zH+7aUZD6wz8wnN68/afk02Y8216qQbvCOWDkRIlckLMeNGjO Lgb1H6TKJJ6NVJR8hs4FlIvgUovmh3iiUdW19wx1hyTjUhiDmjc9jgWH6HB4t3RL tHav9vxJXfSwZ0cdzTuSJTKNwY7C5eOPwbH9WqLzMEkR0iq7HxWOCL20R8MRx/rm BTkCrih7J6j0ILYvc+UJbyKSTI5aQWVBK/hMPQX/QxLTFwHnbk1cr144hGVtQpV9 HjwMLbxCw70bCSAAfxY9Fp+6HzBTpcSxVj6n6B0Y2w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:02:39 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 7d6d44fd (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:02:30 +0000 (UTC) Date: Mon, 25 Mar 2024 11:02:38 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 01/15] reftable/stack: fix error handling in `reftable_stack_init_addition()` Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In `reftable_stack_init_addition()` we call `stack_uptodate()` after having created the lockfile to check whether the stack was modified concurrently, which is indicated by a positive return code from the latter function. If so, we return a `REFTABLE_LOCK_ERROR` to the caller and abort the addition. The error handling has an off-by-one though because we check whether the error code is `> 1` instead of `> 0`. Thus, instead of returning the locking error, we would return a positive value. One of the callers of `reftable_stack_init_addition()` works around this bug by repeating the error code check without the off-by-one. But other callers are subtly broken by this bug. Fix this by checking for `err > 0` instead. This has the consequence that `reftable_stack_init_addition()` won't ever return a positive error code anymore, but will instead return `REFTABLE_LOCK_ERROR` now. Thus, we can drop the check for a positive error code in `stack_try_add()` now. Signed-off-by: Patrick Steinhardt --- reftable/stack.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/reftable/stack.c b/reftable/stack.c index 1ecf1b9751..92d9a7facb 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -590,8 +590,7 @@ static int reftable_stack_init_addition(struct reftable_addition *add, err = stack_uptodate(st); if (err < 0) goto done; - - if (err > 1) { + if (err > 0) { err = REFTABLE_LOCK_ERROR; goto done; } @@ -713,10 +712,6 @@ static int stack_try_add(struct reftable_stack *st, int err = reftable_stack_init_addition(&add, st); if (err < 0) goto done; - if (err > 0) { - err = REFTABLE_LOCK_ERROR; - goto done; - } err = reftable_addition_add(&add, write_table, arg); if (err < 0) From patchwork Mon Mar 25 10:02:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601857 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 848EB2575B for ; Mon, 25 Mar 2024 10:02:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360969; cv=none; b=FhorjmUMW9djVqTHpMheCQl9lk0aAkZzlxbKwbg3h3tOPgDxxIZ0NtgjnDFvmGi4GqiZpAxQzGq0OEoXdfa/MJSt4XPz/u9xtCXbjFTB8MBnjt36jNnFyjqrt4kPaEz3GTzJNQXFYJaGyY09oalysr0EOZQBdLwCzdCpP6QZ9yk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360969; c=relaxed/simple; bh=wJa8/EuvsqWwVY5PTcAoyHYWSiGUa11ae3aILn6cz1Q=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ETfYK8c9G6NaJA0/l+pV4F2u350Eq7XIcIhvqK7eTFtl+y8CmsvSraRCm+Xpi1P58h1BW3K/9Ctv/3s2/LRagIyNkLvaH+3gjvwBAmVrp2AiqP12Yf2pOItJy3PYfC2a5eND+42tBuONNCqZW6KKMYrV0cvccuQOg5ar7uhtv2A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=WvmN01eT; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=n+V0srz2; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="WvmN01eT"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="n+V0srz2" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.nyi.internal (Postfix) with ESMTP id 904961380088; Mon, 25 Mar 2024 06:02:45 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Mon, 25 Mar 2024 06:02:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711360965; x=1711447365; bh=2CYwPq2RDh d7vOyvaKjHP4wwleEdYhnfYVMO6Ohd9Tk=; b=WvmN01eTzxGSL3C9sdHRWNAhh+ WPeJ0qqqovmKrwgu/mBay0xzVhIXe9C16CwivNjT1StVHFPNmlPLWjZ8bIjO1xbA mnpFkvzWle8W03HAi66G95aqVbdMd6KvIBjpF/L7D0mjuBewk1qwrJDngsU1FM20 3oyrFs5x981XXzHC8OZxPtI9qypzPzbR1N0ax3wJ2qxk2nOVbYgT4z0zyzooxtpO X9AXvGf5fqhGIpIUmVKGBh/Sdq9y4GI8A1CK8O23O0Z05jayhRyf0k/rpAqU0cnD pKes9jXFI6WxyjXQGsWvlZla6zrwZ2XRDZXPdV912hNJ8idusYIbfMVyNbNA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711360965; x=1711447365; bh=2CYwPq2RDhd7vOyvaKjHP4wwleEd YhnfYVMO6Ohd9Tk=; b=n+V0srz2OAbcki7PTnqA4XYTRgm+rebS3NRuN/Qefkgr 0sdionBSJ5oUupuKl+jfFs12kahMG3cG1Fo1utpnW5KD/rH4vN9Xfgs8VIcnhO4A LkydWG+XLdAUDzQUSUt/GEbrWZRVvvSvXCNaVrrSafiKqjuH0XzM+JgCjBKJa/ba yFUPdl8mc8Z0lxowBptQTYt4VkuwOlSMZu1QBo7j9uNKJzLsniQ4KBouNxEEoUUP TYtsC9Kc1pbfTiUiKiXUO+rNdi3C3B1g+KIvmYXEaDlyl96BXg115ljPZ9ec53NX gqIAnzG6JVCsbXeanvVMyefENPFxQ2712AY5nq+M3A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:02:44 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 963c625c (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:02:34 +0000 (UTC) Date: Mon, 25 Mar 2024 11:02:42 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 02/15] reftable/error: discern locked/outdated errors Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We currently throw two different errors into a similar-but-different error code: - Errors when trying to lock the reftable stack. - Errors when trying to write to the reftable stack which has been modified concurrently. This results in unclear error handling and user-visible error messages. Create a new `REFTABLE_OUTDATED_ERROR` so that those error conditions can be clearly told apart from each other. Adjust users of the old `REFTABLE_LOCK_ERROR` to use the new error code as required. Signed-off-by: Patrick Steinhardt --- reftable/error.c | 4 +++- reftable/reftable-error.h | 5 ++++- reftable/stack.c | 6 +++--- reftable/stack_test.c | 2 +- 4 files changed, 11 insertions(+), 6 deletions(-) diff --git a/reftable/error.c b/reftable/error.c index 0d1766735e..cfb7a0fda4 100644 --- a/reftable/error.c +++ b/reftable/error.c @@ -22,7 +22,7 @@ const char *reftable_error_str(int err) case REFTABLE_NOT_EXIST_ERROR: return "file does not exist"; case REFTABLE_LOCK_ERROR: - return "data is outdated"; + return "data is locked"; case REFTABLE_API_ERROR: return "misuse of the reftable API"; case REFTABLE_ZLIB_ERROR: @@ -35,6 +35,8 @@ const char *reftable_error_str(int err) return "invalid refname"; case REFTABLE_ENTRY_TOO_BIG_ERROR: return "entry too large"; + case REFTABLE_OUTDATED_ERROR: + return "data concurrently modified"; case -1: return "general error"; default: diff --git a/reftable/reftable-error.h b/reftable/reftable-error.h index 4c457aaaf8..e9b07c9f36 100644 --- a/reftable/reftable-error.h +++ b/reftable/reftable-error.h @@ -25,7 +25,7 @@ enum reftable_error { */ REFTABLE_NOT_EXIST_ERROR = -4, - /* Trying to write out-of-date data. */ + /* Trying to access locked data. */ REFTABLE_LOCK_ERROR = -5, /* Misuse of the API: @@ -57,6 +57,9 @@ enum reftable_error { /* Entry does not fit. This can happen when writing outsize reflog messages. */ REFTABLE_ENTRY_TOO_BIG_ERROR = -11, + + /* Trying to write out-of-date data. */ + REFTABLE_OUTDATED_ERROR = -12, }; /* convert the numeric error code to a string. The string should not be diff --git a/reftable/stack.c b/reftable/stack.c index 92d9a7facb..eaa8bb9c99 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -529,9 +529,9 @@ int reftable_stack_add(struct reftable_stack *st, { int err = stack_try_add(st, write, arg); if (err < 0) { - if (err == REFTABLE_LOCK_ERROR) { + if (err == REFTABLE_OUTDATED_ERROR) { /* Ignore error return, we want to propagate - REFTABLE_LOCK_ERROR. + REFTABLE_OUTDATED_ERROR. */ reftable_stack_reload(st); } @@ -591,7 +591,7 @@ static int reftable_stack_init_addition(struct reftable_addition *add, if (err < 0) goto done; if (err > 0) { - err = REFTABLE_LOCK_ERROR; + err = REFTABLE_OUTDATED_ERROR; goto done; } diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 7336757cf5..2c3540d9e6 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -232,7 +232,7 @@ static void test_reftable_stack_uptodate(void) EXPECT_ERR(err); err = reftable_stack_add(st2, &write_test_ref, &ref2); - EXPECT(err == REFTABLE_LOCK_ERROR); + EXPECT(err == REFTABLE_OUTDATED_ERROR); err = reftable_stack_reload(st2); EXPECT_ERR(err); From patchwork Mon Mar 25 10:02:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601863 Received: from fhigh5-smtp.messagingengine.com (fhigh5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F371A128368 for ; Mon, 25 Mar 2024 10:02:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360972; cv=none; b=QqpoqIDAYOYGwkfZw1hjpyuP0n45kXLcRGVcEjuGvx+BittksZBOwe+xiDCuNPIgaJuZxGip71TcdMiMZBVbxfT7pY73VBSxzeNfo+RG/I0evkjiA6NFOS8KXhHkLanhN241qQDcJEAMg0ZepM6yL1nItoFL4xCmnllc4A8EZ3s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360972; c=relaxed/simple; bh=O8xSshl6Dlex+qsYKwcudTfprt5u3o0SlS88O4FiZJQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=jPrWx55RYbLgZjizWwhTXkrY0vap1ZtS8xxVlm568EvV9dr75rj6vjz5nW9ic2wILhaJzrUDQVMxtLCJgDZijsnBEFe6bG8rPh8NR/tcYpif7zkz7/BzoQYoQE88WyHZtHzqmOhRRl9VQkmzfvXn22ngCkchU/aS+xTuP+3oqis= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=dVExGaUT; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=ER7QyGsG; arc=none smtp.client-ip=103.168.172.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="dVExGaUT"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="ER7QyGsG" Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 18EFF11400A1; Mon, 25 Mar 2024 06:02:49 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Mon, 25 Mar 2024 06:02:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711360969; x=1711447369; bh=Ld9UXFv9km x1DA7JdSByfpyq3lf1+vB5G6uz4TlxKaU=; b=dVExGaUTDolU+bo69M7NnKu9oo 75WX/1RH11Z/SJqZQ6mPNNQlG3d5xrcKtJWZr2Ageu2tiFDo9fUZQDysA9Qxe8bt nKhmvLDJvnA0r2bdoEF4vfYkH0cccntfqBIiDGQn/qVG00lGVq9RetNa8vq2Q/cX 9oWA+bPvjKv8UzWj/ns8GCQjJEareF1nKjaGAreDAyyZZ8KP0u8xplMyxS8731Yq 3QtHlUzTEPvGLerP2q8YMIXyV2QuAP2iPjjyFueodsOj7pv3iCnbureqFXICRQaZ FSk+/uqr+SHgWQaDfYAs+JvqSVylK5n6PaPEaJH3Ap8C/xO5dkYFfYe6Kpvg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711360969; x=1711447369; bh=Ld9UXFv9kmx1DA7JdSByfpyq3lf1 +vB5G6uz4TlxKaU=; b=ER7QyGsGd5p6n5YBn1wKeDxOAocDoVMfRsi8jAT7eHuv YMga2ijYK8OSoj9tQhOPOruUSayBW8oqgieKFJ7791TjIgm0Zvs7b5J+12kz4sWx Ht8sxZtyeG4n1dOLHFY1cUpIAR0GBjbhqU84xPH1JRjYGf1d+beAqH86o3bLNkq/ XAFbD4tyyIBzf1LCLEz2olz0qS/a1plA/e61QvvS9fcxZR+N6N6aUroqVvZJ4PL1 /NPN8NktA/Bznh/L/xW9CXQzLb2uctdYr0j/xj+mwr/lVE/62/FkZQunaLHq+SNU z539ioftK9IhJEdOgoLttBZ7rXpx3NW2pxHP4x+2RQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:02:47 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id fd42cb8a (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:02:38 +0000 (UTC) Date: Mon, 25 Mar 2024 11:02:46 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 03/15] reftable/stack: use error codes when locking fails during compaction Message-ID: <95dda44672a253927d625685a3eb911463318f3a.1711360631.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Compaction of a reftable stack may fail gracefully when there is a concurrent process that writes to the reftable stack and which has thus locked either the "tables.list" file or one of the tables. This is expected and can be handled gracefully by some of the callers which invoke compaction. Thus, to indicate this situation to our callers, we return a positive return code from `stack_compact_range()` and bubble it up to the caller. This kind of error handling is somewhat awkward though as many callers in the call chain never even think of handling positive return values. Thus, the result is either that such errors are swallowed by accident, or that we abort operations with an unhelpful error message. Make the code more robust by always using negative error codes when compaction fails, with `REFTABLE_LOCK_ERROR` for the described benign error case. Note that only a single callsite knew to handle positive error codes gracefully in the first place. Subsequent commits will touch up some of the other sites to handle those errors better. Signed-off-by: Patrick Steinhardt --- reftable/stack.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/reftable/stack.c b/reftable/stack.c index eaa8bb9c99..79856b6565 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -973,7 +973,15 @@ static int stack_write_compact(struct reftable_stack *st, return err; } -/* < 0: error. 0 == OK, > 0 attempt failed; could retry. */ +/* + * Compact all tables in the range `[first, last)` into a single new table. + * + * This function returns `0` on success or a code `< 0` on failure. When the + * stack or any of the tables in the specified range are already locked then + * this function returns `REFTABLE_LOCK_ERROR`. This is a benign error that + * callers can either ignore, or they may choose to retry compaction after some + * amount of time. + */ static int stack_compact_range(struct reftable_stack *st, size_t first, size_t last, struct reftable_log_expiry_config *expiry) @@ -1003,7 +1011,7 @@ static int stack_compact_range(struct reftable_stack *st, LOCK_NO_DEREF); if (err < 0) { if (errno == EEXIST) - err = 1; + err = REFTABLE_LOCK_ERROR; else err = REFTABLE_IO_ERROR; goto done; @@ -1025,7 +1033,7 @@ static int stack_compact_range(struct reftable_stack *st, table_name.buf, LOCK_NO_DEREF); if (err < 0) { if (errno == EEXIST) - err = 1; + err = REFTABLE_LOCK_ERROR; else err = REFTABLE_IO_ERROR; goto done; @@ -1075,7 +1083,7 @@ static int stack_compact_range(struct reftable_stack *st, LOCK_NO_DEREF); if (err < 0) { if (errno == EEXIST) - err = 1; + err = REFTABLE_LOCK_ERROR; else err = REFTABLE_IO_ERROR; goto done; @@ -1187,7 +1195,7 @@ static int stack_compact_range_stats(struct reftable_stack *st, struct reftable_log_expiry_config *config) { int err = stack_compact_range(st, first, last, config); - if (err > 0) + if (err == REFTABLE_LOCK_ERROR) st->stats.failures++; return err; } From patchwork Mon Mar 25 10:02:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601855 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 42CB512B15A for ; Mon, 25 Mar 2024 10:02:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360976; cv=none; b=qx/zP1hYOA/kSV+oO9EB316IdCojS328+KmA9uhT7ORPb5+kPgm8B5Eu0Qxh7FmbM1GuHbBMJL0Hj9SY2v/2W/3+FQr4F3PujdpiYxuEA8lb2bLKOITqknIfvEJk9D+m747lKavUn0SzLWLQmyNIGRPlFghkqq0qeFH6WGUn5mw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360976; c=relaxed/simple; bh=X9eQR7WCzUWSMvEVkZRfr4N1EBXBCevP6CPJlG8+z8I=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=pkzJ1PRWUnFLX5xn9pqtITK60cssQziBBxIVVZYkYRYIokStXjkL9kX5LJAGiAxJTLTqya1w6v8LPpFogtb9G4Mv0q5gIapyVGMVK/UOwoHZYOMtDsu+fQ0IwYQe9S0DlkAqZ/jay4HeiFNutAAJnsDFQCUvVmaVWT4CPsH3f+M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=fgbru2yO; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=a2i4Mz+l; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="fgbru2yO"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="a2i4Mz+l" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfout.nyi.internal (Postfix) with ESMTP id 5D3AA13800AE; Mon, 25 Mar 2024 06:02:53 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 25 Mar 2024 06:02:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711360973; x=1711447373; bh=ekHJpNU+cX YjrbGprUqXiOVVzkgWAI1ADnfoltC118A=; b=fgbru2yOQKYVEhk7pPadekup/D KbfvToYdK27sX9boQ3vEQo5ESv8vuP3fjvKmE3/0YT2eKWg9xrDK28vwqwrBpcmi TDjoWpxa3o1eUb0psdgjIQLtCL52U6Ndy7PD9W+7qT4RNSxypYT8JIA6pewz7Il0 OAVqP5U4yPxhmY+U5FvEk1NVkZGyz4I40lXNHiA00xukCO09kAEBCucfpm2iCEYw k1r83bLDbGF+T3zoX+DL/dio4/M0gUct8beFim4ZmLDYjtiTJUBs2onsLS/d0s1o E6XKEitnLQZulGaCgso4D13pBpgnU8dNJxqYJm0pho0pPqVVKePIrL4VipSw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711360973; x=1711447373; bh=ekHJpNU+cXYjrbGprUqXiOVVzkgW AI1ADnfoltC118A=; b=a2i4Mz+lo6a68PnGRdd2w+KBmo79H+86N5JlSb4+O0qC Z3JAZe6cAWLGfCGwXDAbYcYqWjCjX/IOLo9Mr87pMtHgaL3UpxFqAa1B3mlFe9b0 6AFA4d2kIa+vNP+SxF+NXOo6WojuhYyXXDexg8AsqDOs+B7PMKt+RYWLm9rYhtwl LnEH5aQ0JJPOCDT20Vkoj15HAMOXDicHE1UQwXrouilkeKhDuW+q98eC97O0ZY/e kTQSYuJQVKqJFlAeYHF8A87Yfu6PZRF2Hue/uepua+IY4+O/6x5ZOlG/Z79PKmUg Xw60m9xgFv6Lw/3U7W8FWEpScXufkgHHWsVoarQfGw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:02:52 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id a16e4749 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:02:42 +0000 (UTC) Date: Mon, 25 Mar 2024 11:02:50 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 04/15] reftable/stack: gracefully handle failed auto-compaction due to locks Message-ID: <50a3c37f92a28876c0db24e515826485c863ecc3.1711360631.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Whenever we commit a new table to the reftable stack we will end up invoking auto-compaction of the stack to keep the total number of tables at bay. This auto-compaction may fail though in case at least one of the tables which we are about to compact is locked. This is indicated by the compaction function returning `REFTABLE_LOCK_ERROR`. We do not handle this case though, and thus bubble that return value up the calling chain, which will ultimately cause a failure. Fix this bug by ignoring `REFTABLE_LOCK_ERROR`. Signed-off-by: Patrick Steinhardt --- reftable/stack.c | 13 +++++++++++- reftable/stack_test.c | 43 ++++++++++++++++++++++++++++++++++++++ t/t0610-reftable-basics.sh | 20 ++++++++++++++++++ 3 files changed, 75 insertions(+), 1 deletion(-) diff --git a/reftable/stack.c b/reftable/stack.c index 79856b6565..dde50b61d6 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -680,8 +680,19 @@ int reftable_addition_commit(struct reftable_addition *add) if (err) goto done; - if (!add->stack->disable_auto_compact) + if (!add->stack->disable_auto_compact) { + /* + * Auto-compact the stack to keep the number of tables in + * control. It is possible that a concurrent writer is already + * trying to compact parts of the stack, which would lead to a + * `REFTABLE_LOCK_ERROR` because parts of the stack are locked + * already. This is a benign error though, so we ignore it. + */ err = reftable_stack_auto_compact(add->stack); + if (err < 0 && err != REFTABLE_LOCK_ERROR) + goto done; + err = 0; + } done: reftable_addition_close(add); diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 2c3540d9e6..822e681028 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -343,6 +343,48 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void) clear_dir(dir); } +static void test_reftable_stack_auto_compaction_fails_gracefully(void) +{ + struct reftable_ref_record ref = { + .refname = "refs/heads/master", + .update_index = 1, + .value_type = REFTABLE_REF_VAL1, + .value.val1 = {0x01}, + }; + struct reftable_write_options cfg = {0}; + struct reftable_stack *st; + struct strbuf table_path = STRBUF_INIT; + char *dir = get_tmp_dir(__LINE__); + int err; + + err = reftable_new_stack(&st, dir, cfg); + EXPECT_ERR(err); + + err = reftable_stack_add(st, write_test_ref, &ref); + EXPECT_ERR(err); + EXPECT(st->merged->stack_len == 1); + EXPECT(st->stats.attempts == 0); + EXPECT(st->stats.failures == 0); + + /* + * Lock the newly written table such that it cannot be compacted. + * Adding a new table to the stack should not be impacted by this, even + * though auto-compaction will now fail. + */ + strbuf_addf(&table_path, "%s/%s.lock", dir, st->readers[0]->name); + write_file_buf(table_path.buf, "", 0); + + ref.update_index = 2; + err = reftable_stack_add(st, write_test_ref, &ref); + EXPECT_ERR(err); + EXPECT(st->merged->stack_len == 2); + EXPECT(st->stats.attempts == 1); + EXPECT(st->stats.failures == 1); + + reftable_stack_destroy(st); + clear_dir(dir); +} + static void test_reftable_stack_validate_refname(void) { struct reftable_write_options cfg = { 0 }; @@ -1085,6 +1127,7 @@ int stack_test_main(int argc, const char *argv[]) RUN_TEST(test_reftable_stack_tombstone); RUN_TEST(test_reftable_stack_transaction_api); RUN_TEST(test_reftable_stack_transaction_api_performs_auto_compaction); + RUN_TEST(test_reftable_stack_auto_compaction_fails_gracefully); RUN_TEST(test_reftable_stack_update_index_check); RUN_TEST(test_reftable_stack_uptodate); RUN_TEST(test_reftable_stack_validate_refname); diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index 686781192e..5f2f9baa9b 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -340,6 +340,26 @@ test_expect_success 'ref transaction: empty transaction in empty repo' ' EOF ' +test_expect_success 'ref transaction: fails gracefully when auto compaction fails' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + + test_commit A && + for i in $(test_seq 10) + do + git branch branch-$i && + for table in .git/reftable/*.ref + do + touch "$table.lock" || exit 1 + done || + exit 1 + done && + test_line_count = 13 .git/reftable/tables.list + ) +' + test_expect_success 'pack-refs: compacts tables' ' test_when_finished "rm -rf repo" && git init repo && From patchwork Mon Mar 25 10:02:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601858 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 96D0912B150 for ; Mon, 25 Mar 2024 10:02:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360980; cv=none; b=E2LmCVqogwWsisjZDZFuMOox7MwFPQLnaEW/KKLn2jRnignSRPKkhxstZ/pS5BqSy62MVJuZtWTI1O7LYPLXotJeV8BDK1pMSVXH1WyqDOQB86o/Z3eO0Q8ti7pUUj5GCB2HFcIKS5B+HtEpMVaY5JRi6wTUilkVVjC63EX4Qj0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360980; c=relaxed/simple; bh=Dag3uSfsvMAMrnQlKlshyBZUiai7vJ9CiQCu2qRrnI0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=EO+XBzN2t8GHcefHu3jdzgAEdBoOMty/VRpfgRwiKYzQTkvihTM+2Ap5HlC93VpijqSkgcANmR0jupeWGRrq8V5XDJTYpwleBp2QAqQu+fj9FCJKw4ENkQYBYb2wIdSnqI5+QJ7J1ZXNyaLTE6twPdypLB2YK1PRZiiABhlt9Mo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=X0KeWTtY; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=VqKJbyUn; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="X0KeWTtY"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="VqKJbyUn" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.nyi.internal (Postfix) with ESMTP id 9D20D13800B7; Mon, 25 Mar 2024 06:02:57 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Mon, 25 Mar 2024 06:02:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711360977; x=1711447377; bh=SngaOgt+ge Xdxk4pXVxHwT3nPVn888kE9+8fIPLEHbA=; b=X0KeWTtYI+gGELD5H+G5X2udzy a17W8mh0UTDj9QH8gpce2mIs46XhjVptAWhJq4iMHkFLioTCq3GEyJppeca5UZ3r EZu2TT3jKU3zD4WUY4AeJT/yLoC4g2dYfx49b8QCiLnQnXIPyfw/8bOq+T0vFl7r QbpD6ZwSr0uX0ars65JIIQLBkng7Z3bYDJ09yXSdFU0f+xmwePultABWDT6aWahn vvuqgwNR8/cYWBa/SLmCY/mDeu88cEQAVTWzxKa+ddlIur57ev4HrQD7QWKrV2NH UCorvc+zKwrp4wpt+J4PaD38CS33v5p4fl7/3vdWZlq4Zcxh7VX/9D2BUEpg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711360977; x=1711447377; bh=SngaOgt+geXdxk4pXVxHwT3nPVn8 88kE9+8fIPLEHbA=; b=VqKJbyUnZtz8o5E3o/thKIDcUuD9yxldeAGi539WN+ID 6UHhbo2lZ+ty7BwdOl7I2Zdzg13w6K15QLsO8/xY2zlI3veI0iQGrVg3zrBiU3aH OhaVoFPvhpJevyk/VEcsrWilSUnSC48x4PTY5iLK3BB692nVIldDXeow82meKc8m xu+a0qU7Cnh0Bd5cc3IMATQPp8MLm1qPItQX4XCQJy8HQGp5VsO8XZctxdryZzrG KhKNKfUb8IT9s6j1/3NBApF32g8sMGPABN0S+b8FAm9Bc25/c+dldGi9wleKEOKX wlW3IoaoKgNOdR/jPPLOdNnEX3PCpX+hqBfZKFxPhQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:02:56 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 73df58f9 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:02:46 +0000 (UTC) Date: Mon, 25 Mar 2024 11:02:54 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 05/15] refs/reftable: print errors on compaction failure Message-ID: <6615f25b0890c934a31902bed024eaa9146ec641.1711360631.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When git-pack-refs(1) fails in the reftable backend we end up printing no error message at all, leaving the caller puzzled as to why compaction has failed. Fix this. Signed-off-by: Patrick Steinhardt --- refs/reftable-backend.c | 6 +++++- t/t0610-reftable-basics.sh | 12 ++++++++++++ 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index e206d5a073..694dc4845f 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -1204,8 +1204,12 @@ static int reftable_be_pack_refs(struct ref_store *ref_store, stack = refs->main_stack; ret = reftable_stack_compact_all(stack, NULL); - if (ret) + if (ret < 0) { + ret = error(_("unable to compact stack: %s"), + reftable_error_str(ret)); goto out; + } + ret = reftable_stack_clean(stack); if (ret) goto out; diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index 5f2f9baa9b..a53d1dc493 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -375,6 +375,18 @@ test_expect_success 'pack-refs: compacts tables' ' test_line_count = 1 repo/.git/reftable/tables.list ' +test_expect_success 'pack-refs: compaction raises locking errors' ' + test_when_finished "rm -rf repo" && + git init repo && + test_commit -C repo A && + touch repo/.git/reftable/tables.list.lock && + cat >expect <<-EOF && + error: unable to compact stack: data is locked + EOF + test_must_fail git -C repo pack-refs 2>err && + test_cmp expect err +' + test_expect_success 'pack-refs: prunes stale tables' ' test_when_finished "rm -rf repo" && git init repo && From patchwork Mon Mar 25 10:02:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601859 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C989C4CB37 for ; Mon, 25 Mar 2024 10:03:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360984; cv=none; b=JakbgiyksobUgTF1zkSw5vPzG6sDKDb8/PjcRmgLCoxGDpYPMFwkjXw/UXy0McJi4Ny2OxTkOdq0qkT7oa2InZlqrWQcgyXPnegig5IPOKCsXHcU6e3YNYe7vz+JZ0DK0P6U4Pfm/CVnqbof51r9xWPCgr9lHFA66y+4kXDctlk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360984; c=relaxed/simple; bh=pFCXwAfAt3miG2crjRX/AJahxbhde3+nWa/0T92++vw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ryT5NglxPkxgWb/MlGvFl/VP63+Qzsv1I5oM9YkxxG18Mz5L86FXv4Z4aX0Cq0Edu/raWbpp84jXxBM5U9C96DUCdaoYhB+13pKXqcIkMLc46mOll5IHkxGGfEuW6KiBkES88PBZh0kKoWzO/bF1uJNXjqa7U+IR5st+BnHY53Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=lPb93p3O; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=IVBiviVH; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="lPb93p3O"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="IVBiviVH" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfout.nyi.internal (Postfix) with ESMTP id D8A1413800AD; Mon, 25 Mar 2024 06:03:01 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 25 Mar 2024 06:03:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711360981; x=1711447381; bh=YmTp9GjzMl qtRVUd/3jFfpraHis1beNHwRqud3gsvEQ=; b=lPb93p3Oj45tWNVGmQBU9jdoYP buW7+G8L7ddLSa8dJU/dYw3JWsel9vA2Tehwb0SoW611ZrvfyN0s4ENk1zvw3cAi k2e3u/40HjmJnogmZp8EaUZWVIb+EvGoTG7QKjtpmugngzOKqzwur9+ka6n4nKzw Y0K20mXzbeGKh78vkEv84LqcnkMNaa1a/VUIjAcRci2mS68BWweNYfqKndZ8/x6c YsmnQknJLM42WBzSsZv4NHFYfqrzsqPXl8p76akf37SFR4R4yU5BoJZm5pcb4fbM y+c002wpd0H+zFUR1g1nZ0trsPqfG92HdlHkWdaJsrVIb3OTT60TiL8dXCKg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711360981; x=1711447381; bh=YmTp9GjzMlqtRVUd/3jFfpraHis1 beNHwRqud3gsvEQ=; b=IVBiviVH85fS9sFt153t+dOXIEJlMLFbIJ9uOqfm3U2R SL44BP5z8uk/jRagPzMi/qx38pmVlSdRfS7LCkHPExZNaw+qxFXCHlssLsOycmkb mZdsGNo1mK7ihhid2QnM4QV0cmlEIGV88B99uTMgtUjxk1v/IZvtcgwN6WVLr+9r wQvLy3omfeN68g7eoNfdbw1EtWLY84NiS8twlROSPwhBQwbr2+MdpSgLRoNUsZ5w Du3kJtIfCktHLizf451r9K7wPvTh+oJTX6B+q7c9aK8RPwI6Uj3yZiPmZsrRgPB/ xSwWSac5Grogx0Du+fz8JnTrLvAF5FnEna/0YD3GAA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:03:00 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 8e7db634 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:02:51 +0000 (UTC) Date: Mon, 25 Mar 2024 11:02:59 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 06/15] t/helper: drop pack-refs wrapper Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The test helper provides a "ref-store pack-refs" wrapper that more or less directly invokes `refs_pack_refs()`. This helper is only used in a single test with the "PACK_REFS_PRUNE" and "PACK_REFS_ALL" flags. Both of these flags can directly be accessed via git-pack-refs(1) though via the `--all` and `--prune` flags, which makes the helper superfluous. Refactor the test to use git-pack-refs(1) instead of the test helper. Drop the now-unused test helper command. Signed-off-by: Patrick Steinhardt --- t/helper/test-ref-store.c | 20 -------------------- t/t0601-reffiles-pack-refs.sh | 13 +++++++++---- 2 files changed, 9 insertions(+), 24 deletions(-) diff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c index 7a0f6cac53..82bbf6e2e6 100644 --- a/t/helper/test-ref-store.c +++ b/t/helper/test-ref-store.c @@ -112,25 +112,6 @@ static const char **get_store(const char **argv, struct ref_store **refs) return argv + 1; } -static struct flag_definition pack_flags[] = { FLAG_DEF(PACK_REFS_PRUNE), - FLAG_DEF(PACK_REFS_ALL), - { NULL, 0 } }; - -static int cmd_pack_refs(struct ref_store *refs, const char **argv) -{ - unsigned int flags = arg_flags(*argv++, "flags", pack_flags); - static struct ref_exclusions exclusions = REF_EXCLUSIONS_INIT; - static struct string_list included_refs = STRING_LIST_INIT_NODUP; - struct pack_refs_opts pack_opts = { .flags = flags, - .exclusions = &exclusions, - .includes = &included_refs }; - - if (pack_opts.flags & PACK_REFS_ALL) - string_list_append(pack_opts.includes, "*"); - - return refs_pack_refs(refs, &pack_opts); -} - static int cmd_create_symref(struct ref_store *refs, const char **argv) { const char *refname = notnull(*argv++, "refname"); @@ -326,7 +307,6 @@ struct command { }; static struct command commands[] = { - { "pack-refs", cmd_pack_refs }, { "create-symref", cmd_create_symref }, { "delete-refs", cmd_delete_refs }, { "rename-ref", cmd_rename_ref }, diff --git a/t/t0601-reffiles-pack-refs.sh b/t/t0601-reffiles-pack-refs.sh index c309d2bae8..b1cf587347 100755 --- a/t/t0601-reffiles-pack-refs.sh +++ b/t/t0601-reffiles-pack-refs.sh @@ -32,11 +32,16 @@ test_expect_success 'prepare a trivial repository' ' HEAD=$(git rev-parse --verify HEAD) ' -test_expect_success 'pack_refs(PACK_REFS_ALL | PACK_REFS_PRUNE)' ' - N=`find .git/refs -type f | wc -l` && +test_expect_success 'pack-refs --prune --all' ' + test_path_is_missing .git/packed-refs && + git pack-refs --no-prune --all && + test_path_is_file .git/packed-refs && + N=$(find .git/refs -type f | wc -l) && test "$N" != 0 && - test-tool ref-store main pack-refs PACK_REFS_PRUNE,PACK_REFS_ALL && - N=`find .git/refs -type f` && + + git pack-refs --prune --all && + test_path_is_file .git/packed-refs && + N=$(find .git/refs -type f) && test -z "$N" ' From patchwork Mon Mar 25 10:03:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601862 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1CD7012C550 for ; Mon, 25 Mar 2024 10:03:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360988; cv=none; b=WoEmXwln2VqZuerEp/g4+ztUtddKUypBF/FPabtNV0D8MlRUwmjxSHoh8VYX8/zyecNIGfRX+K9t50KLwOvDvaXzBEt2VobJfQ6pvpd+q2xVEp3hvbxvevIMMm3mhNliauAkLMU3WejIbgS/fHBWfj+gUaYFpBpTdju78F4/hJI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360988; c=relaxed/simple; bh=SvdDExjxp+VEjTH5CIdiVa7INZvzBvPFcRsRpWqKY9M=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=KLMRAMjTBQ/2H83wkVP9HjB/Go1FLSLMdRoZH6gYsP+Awa3804F3G4bJxPc+VrWXpZGvWU228F989YoDzgmA33uC1n/ZYuLy28YMu2/7pZ8UtWw/7Jqv2zEfJQAFt2FrNPpEQlPlIdqDhVb1/7PYCzDH5JdxBUfYgbT7CfE1uLw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=BKmcrYte; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=VS46aGRy; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="BKmcrYte"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="VS46aGRy" Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfout.nyi.internal (Postfix) with ESMTP id 28A4F1380083; Mon, 25 Mar 2024 06:03:06 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 25 Mar 2024 06:03:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711360986; x=1711447386; bh=I6toofywJk eF7jswopkRGwq5moffPwnyKcvdwRibSN4=; b=BKmcrYte+1sFT5T8WJhKmnxgZo kHM9+6YBMeCgIfNH6SfyScp1xvP/zGoWgqR1/JugJ2MmvMR2kOcCIkPaBimG3LwT RbT6m+m1UkI9a74UFi4RiibvMIUK4Z2NedAmFIDnpKKv2D/tp1hCmsoOwQNAfo+r I78mAV8thdkT+2dBNYOR6ilOw39OJ92HucGqDAuwATkFHyGLH1D+Vdv8dgs7PBV7 M8N9ZmOIPmxAn2pRNlKYGagUN6Kl1fyXcMkhUXiSqRhRXw8x0ZDcuF5j3rOW2+Ui 3DPkeHmgpOJFo9kVoReQTnGgg58UgPM9CXgdWTiVy+I1LBtGUZZ8bK+YkBOw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711360986; x=1711447386; bh=I6toofywJkeF7jswopkRGwq5moff PwnyKcvdwRibSN4=; b=VS46aGRyCuNIos8ikflu0H6VZgl4KupMP/2yO0Aizkv8 dZydaKaX95ycQrLSMVfGFUWjTvFc0Kgq8RcBeXwl7oMN32pu439+HvvVyKOVt6m2 luwfdagDktOf9dPi4lVLGiDl5aS3H6SJ6rL3TVXWZqwjYrEsGSKxfbNnPInNgKjt HPys18xZcnEN8ld8oMIWBC1ZlidLJA2CjGwszsT7vDQSRlbxBUfA8v3t3o4VO9Hu O2XYgrxubRjJUs7cNsZxFBpWxCm0PMstcxQwhODenWWDxAdS+shNHoRAbbTrpJKC K3wqceMiDSQZ9oFC37u3bcnHui0BoY65MAOUgyhWbw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:03:05 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 085286da (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:02:55 +0000 (UTC) Date: Mon, 25 Mar 2024 11:03:03 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 07/15] refs: move `struct pack_refs_opts` to where it's used Message-ID: <809094ffec47db9c2f95850106448ad3b2e83430.1711360631.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The declaration of `struct pack_refs_opts` is in a seemingly random place. Move it so that it's located right next to its flags and functions that use it. Signed-off-by: Patrick Steinhardt --- refs.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/refs.h b/refs.h index 298caf6c61..139ce7113b 100644 --- a/refs.h +++ b/refs.h @@ -66,12 +66,6 @@ const char *ref_storage_format_to_name(unsigned int ref_storage_format); #define RESOLVE_REF_NO_RECURSE 0x02 #define RESOLVE_REF_ALLOW_BAD_NAME 0x04 -struct pack_refs_opts { - unsigned int flags; - struct ref_exclusions *exclusions; - struct string_list *includes; -}; - const char *refs_resolve_ref_unsafe(struct ref_store *refs, const char *refname, int resolve_flags, @@ -433,6 +427,12 @@ void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, #define PACK_REFS_PRUNE 0x0001 #define PACK_REFS_ALL 0x0002 +struct pack_refs_opts { + unsigned int flags; + struct ref_exclusions *exclusions; + struct string_list *includes; +}; + /* * Write a packed-refs file for the current repository. * flags: Combination of the above PACK_REFS_* flags. From patchwork Mon Mar 25 10:03:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601861 Received: from fhigh5-smtp.messagingengine.com (fhigh5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5906C12C7E0 for ; Mon, 25 Mar 2024 10:03:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360992; cv=none; b=ohI2JWgl6DVtrxUaMM2RMYJxPIwlZEnw4gzE12s96MapeJBklYSsHFCP7337fXSwLmWW1fSckPuWECp4AhYCVauboCam6lHViVXosFQTs6BMaiqZOoixCeuGHtBY/cSq1NoyU8iXEylbpUE5F1lDD+4WF+GHtnP3qzIWYDuZx3A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360992; c=relaxed/simple; bh=Fg3t/vhWZAcdw/zuFXTXNuFa7/6p7zxNjsFTlej+6Zk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=oVip7214fCJxK+Q++ZgbS7mTT0ycYglKaqc7urI7OOkmhwT6k5Tm0tll9NZShPRPZuuJRKppsG4CgB8Fasvr17cUDbzbhbDb0cktd3sQTfvN1Rl6hfHjC8hnWTpfAbl2Vxebb5iEZ5tf0hsXra+AJx1UMdDT9/eU0fp19oiw000= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=WuaUBNnP; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=cXkfrO14; arc=none smtp.client-ip=103.168.172.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="WuaUBNnP"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="cXkfrO14" Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 6A1F811400B2; Mon, 25 Mar 2024 06:03:10 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 25 Mar 2024 06:03:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711360990; x=1711447390; bh=hQRJt4A8Jg 8iCyo62bjEcvEhyFpux5k/kE5uZ3/od7o=; b=WuaUBNnPC+mGaQ9PIht4DWAeWF /uggaU6K7JFYUe2Z+E6g+PhRd6d7Rb5egRn1DkdgylUZJZqrPQpDGDbm07TiaYHZ BZqlMHm92NaLDwDGQUjONziKVwV4Hn9rRdSNI7HK9pzKxX4pXkv0hlIulUksJNqM wRaOFy1al0BLSnC2up7tIrZgBTI9k8Kg3FCSJU8AjnQBceJwurSmzgS0OdsnhtuS r4d7IhXrab2/COKOcZ5BEJkxmJAY5tW54ClpQz0xrxX3bg+9TXI2GTvnGDB4LJIl MH9Q7MCEEgHk28E6w8kAD9lmGT0/WYDFCAOujUyFoNMqC9Ym0ydQgqkwTe5g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711360990; x=1711447390; bh=hQRJt4A8Jg8iCyo62bjEcvEhyFpu x5k/kE5uZ3/od7o=; b=cXkfrO14cW+aGwYulofrPlJLwHcEdbNNB4SVq9CA4hSU K8ZphVup4jpzyzs0KzBm0UHxmFsGr3dIoVjtHP2rLUcKWry1aomgdTc39zFZZ8os eLnOOoWVlGiYTBVGICRaQIK0P6zNphzf7PAQKNlmHoGGouqSczKbPTwbfpteK4+i h0832s+XzUb9uWbDJw9Y9r0YbQ4Gul8n5PF5z/YKsfLb05mhwXxigr8cxOGQsl2V PidzLsxXUB9cUW/U61Rl2EjNRcGgarNTMpG2nsMKnS1xropM4x8YgGJjuiZzYLUs dCCEe+Y4IET11GmXSQ2RJscKbK2C8hAGlZUjAiA/bg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:03:09 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 8886eb9f (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:02:59 +0000 (UTC) Date: Mon, 25 Mar 2024 11:03:07 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 08/15] refs: remove `PACK_REFS_ALL` flag Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The intent of the `PACK_REFS_ALL` flag is to ask the backend to compact all refs instead of only a subset of them. Thus, this flag gets passed down to `refs_pack_refs()` via `struct pack_refs_opts::flags`. But starting with 4fe42f326e (pack-refs: teach pack-refs --include option, 2023-05-12), the flag's semantics have changed. Instead of being handled by the respective backends, this flag is now getting handled by the callers of `refs_pack_refs()` which will add a single glob ("*") to the list of refs-to-be-packed. Thus, the flag serves no purpose to the ref backends anymore. Remove the flag and replace it with a local variable. Signed-off-by: Patrick Steinhardt --- builtin/pack-refs.c | 5 +++-- refs.h | 2 -- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/builtin/pack-refs.c b/builtin/pack-refs.c index bcf383cac9..97921beef2 100644 --- a/builtin/pack-refs.c +++ b/builtin/pack-refs.c @@ -21,9 +21,10 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix) .flags = flags }; static struct string_list option_excluded_refs = STRING_LIST_INIT_NODUP; struct string_list_item *item; + int pack_all = 0; struct option opts[] = { - OPT_BIT(0, "all", &pack_refs_opts.flags, N_("pack everything"), PACK_REFS_ALL), + OPT_BOOL(0, "all", &pack_all, N_("pack everything")), OPT_BIT(0, "prune", &pack_refs_opts.flags, N_("prune loose refs (default)"), PACK_REFS_PRUNE), OPT_STRING_LIST(0, "include", pack_refs_opts.includes, N_("pattern"), N_("references to include")), @@ -38,7 +39,7 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix) for_each_string_list_item(item, &option_excluded_refs) add_ref_exclusion(pack_refs_opts.exclusions, item->string); - if (pack_refs_opts.flags & PACK_REFS_ALL) + if (pack_all) string_list_append(pack_refs_opts.includes, "*"); if (!pack_refs_opts.includes->nr) diff --git a/refs.h b/refs.h index 139ce7113b..8c8994cb29 100644 --- a/refs.h +++ b/refs.h @@ -422,10 +422,8 @@ void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, /* * Flags for controlling behaviour of pack_refs() * PACK_REFS_PRUNE: Prune loose refs after packing - * PACK_REFS_ALL: Pack _all_ refs, not just tags and already packed refs */ #define PACK_REFS_PRUNE 0x0001 -#define PACK_REFS_ALL 0x0002 struct pack_refs_opts { unsigned int flags; From patchwork Mon Mar 25 10:03:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601864 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F5A6187655 for ; Mon, 25 Mar 2024 10:03:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360997; cv=none; b=LEyLB7WA2AXMyYfze3a3Jlj/7mk/OPGOXAIrGAHOBRVHIUtVyw/fN5WqDcglLCrws1IxkFFjuCotAS8LqDPnw1t68EyaGJrRRenKro3Ruq/Zr32v/h0pyOj2bEVJAwBD6bvHWE2MfMKhNqZggF4v/0j5ptQMHRRynEDRWmltXVE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711360997; c=relaxed/simple; bh=EyB1g/RyLPvRfz3pJrLyb/TQiChG/MvdswabBjBjk9Q=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=LfM86O6zBsv3HgNaKAkCg14lWEmB5y9F6+FlmBaNtdbUZRjjt49Y/wAfEafsH8n4oB0bG1T/SI1hjLtAKAPCdIxcKy35orV2FS6MdBvUsq052tHuNHwvm6J7bMh+TZX6rXvvVbXaxDCZ7pItFEMokJpQl9Ed3iA9XmrAvHTzdFU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=aS7HZGta; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=qUgBIqST; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="aS7HZGta"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="qUgBIqST" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfout.nyi.internal (Postfix) with ESMTP id 9F4501380083; Mon, 25 Mar 2024 06:03:14 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 25 Mar 2024 06:03:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711360994; x=1711447394; bh=WSxWqR9iW/ y2ufdK6Jjuh5lyTEAWhg4uBQT6I4Ex9Ts=; b=aS7HZGtaFVwOwSLp+mQk3A6rjm g+8GUbGZ6kdXRMr/99kPZIZYc81aUAYIe+TFZqr01t3D/fa8bk1igkSWHg5ecsaZ c11GGEVpQFZpl5JFoT6a6wwhR9V9Qdj54iKrjQRR1bsqt52Vje7lesgUteSHQORV Fmo9jXVKgdUDtnXpAO0/V4mt0BBQfYeLIT9mPNmZJzXf/4GuAzqktOEip+66cyZ3 t5HBMDD1u9vsNdNJI43B8bv2qvwSXvziewFPv9LDZ4wM7fsbr7v50zBBJvXUUKSm K1g4e5RGg++nbdqOnJXauaxEU50UObiFEvCNPjaYs3hUI6uCVDZvd+JEKyjw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711360994; x=1711447394; bh=WSxWqR9iW/y2ufdK6Jjuh5lyTEAW hg4uBQT6I4Ex9Ts=; b=qUgBIqSTbThleFpmeWLHL0xjf/3iJbnfjYGEpeoHHXV+ zSBJihyRb/bmAETm9n/jWohh78rQ57KFlGHdqx+kHPCilpLuJfHIQrkmsx9GRHBG KwQjtAFBdrcBsI3ihLKEIDMqW5dckxgN0CBBCsNdsAwHZyNhmMZBWOaX4zV/ob6h SEvtjcoIl588egIeXp1Tp4+a+BxumnqeHrkTzw4xKwig3KqmwxX+uKfbC5aMwrF+ jIfmFGj2gwEPBX+9cy4kgXl8MXCNS2AeGYSbC/0g12yCHW/QARN3Efj5Bv3dSzTB CDePeiPi5isOeXz3FpqXF+yoa2TTsATpA3Mpy6g4mg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:03:13 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 6cdce92e (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:03:03 +0000 (UTC) Date: Mon, 25 Mar 2024 11:03:11 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 09/15] refs/reftable: expose auto compaction via new flag Message-ID: <5d7af236d4a7fd9d133f400a3b814b03d772aacc.1711360631.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Under normal circumstances, the "reftable" backend will automatically perform compaction after appending to the stack. It is thus not necessary and may even be considered wasteful to run git-pack-refs(1) in "reftable"-backed repositories as it will cause the backend to compact all tables into a single one. We do exactly that though when running `git maintenance run --auto` or `git gc --auto`, which gets spawned by Git after running some specific commands. The `--auto` mode is typically only executing optimizations as needed. To do so, we already use several heuristics for the various different data structures in Git to determine whether to optimize them or not. We do not use any heuristics for refs though and instead always optimize them. Introduce a new `PACK_REFS_AUTO` flag that can be passed to the backend. When not handled by the backend we will continue to behave the exact same as we do right now, that is we optimize refs unconditionally. This is done for the "files" backend for now to retain current behaviour, even though we may eventually also want to introduce heuristics here. For the "reftable" backend though we already do have auto-compaction, so we can easily reuse that logic to implement the new auto-packing flag. Note that under normal circumstances, this should always end up being a no-op. After all, we already invoke the code for every single addition to the stack. But there are special cases where it can still be helpful to execute the auto-compaction code explicitly: - Concurrent writers may cause compaction to not run due to locks. - Callers may decide to disable compaction altogether and then pack refs at a later point due to various reasons. - Other implementations of the reftable format may do compaction differently or even not at all. Signed-off-by: Patrick Steinhardt --- refs.h | 6 +++++- refs/reftable-backend.c | 5 ++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/refs.h b/refs.h index 8c8994cb29..d278775e08 100644 --- a/refs.h +++ b/refs.h @@ -422,8 +422,12 @@ void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, /* * Flags for controlling behaviour of pack_refs() * PACK_REFS_PRUNE: Prune loose refs after packing + * PACK_REFS_AUTO: Pack refs on a best effort basis. The heuristics and end + * result are decided by the ref backend. Backends may ignore + * this flag and fall back to a normal repack. */ -#define PACK_REFS_PRUNE 0x0001 +#define PACK_REFS_PRUNE (1 << 0) +#define PACK_REFS_AUTO (1 << 1) struct pack_refs_opts { unsigned int flags; diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 694dc4845f..0bed6d2ab4 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -1203,7 +1203,10 @@ static int reftable_be_pack_refs(struct ref_store *ref_store, if (!stack) stack = refs->main_stack; - ret = reftable_stack_compact_all(stack, NULL); + if (opts->flags & PACK_REFS_AUTO) + ret = reftable_stack_auto_compact(stack); + else + ret = reftable_stack_compact_all(stack, NULL); if (ret < 0) { ret = error(_("unable to compact stack: %s"), reftable_error_str(ret)); From patchwork Mon Mar 25 10:03:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601870 Received: from fhigh5-smtp.messagingengine.com (fhigh5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6DC2157A5C for ; Mon, 25 Mar 2024 10:03:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711361001; cv=none; b=ARlYSFRhp4MHdmOl4VuMKYjsJyE1MZJNKKkd2HpIbNRCLuukyF8MG9YFUH9hlsNSMQnZiy4F5yV+gNzAqWEhEJ96lA7zc5nMcgBQrQFdSyKcZn+9YvPosDdyJS3YxDzHAXCeeRKX/pxSwlI7DspV9vNNoMzFHwx8aMoMmh8g+CI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711361001; c=relaxed/simple; bh=pGC5JOMMsDDm9NrEPuDh4xYSlLvvdRT7azYwwaXS46Y=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=DIG40JHc1Q2yS2Qnm/INME/yve7eGtdrRCwijTYwQlEA2UyWkK20WuZpbCFXXzuy1UTZTV5LIUC7rFioE5YK75qGsRye4nUUiL7Z/L1ma/JKc2/kiW5zmTlsfB7Iy2MsNubSnUE6q7Ri4lpBlZoX0fanNCT+3G0ScwoKDAQFDwY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=S0S8um1z; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=xtOulHBq; arc=none smtp.client-ip=103.168.172.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="S0S8um1z"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="xtOulHBq" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.nyi.internal (Postfix) with ESMTP id EAC4411400B2; Mon, 25 Mar 2024 06:03:18 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Mon, 25 Mar 2024 06:03:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711360998; x=1711447398; bh=BpxXWxcMvb Tp852bBm+Wfny8qNXppmKMMorTPRhfikQ=; b=S0S8um1z3MFg/7Zfxkz9HCQQ6C yu97GPcA4euOaf1izCtV9qRj6CRJG/pt6cw0m6gGUYC7Ni5SzVPRWLnPtJbz+8tv J7mwFkdkxZ37v6hummEZ8iIwbAYKdT8WxEG/QCP3m48Rm4jDPZqP7JLWpmgvnGJX tLjLcX948CFbOLB4mm5ieiJ1v7vlnqoFhsnecmh4+XVhgAigKi0xTvOINy1ss0VS 6Pq0uSOzKQVQBeCG0OuXnuK5KGULvdgaZS7RuRuBhC/CRL5Y8GeAdE5ZOX5h06Z4 rv911iDjNqLCVdMhkANfrOdGoLA96MVzkAqNuTILfYJKtFRL7j6kn/SFG+sg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711360998; x=1711447398; bh=BpxXWxcMvbTp852bBm+Wfny8qNXp pmKMMorTPRhfikQ=; b=xtOulHBqMEGj7HoHWvuntZ3ZBYFTt3DLQVsu4haG9ylK xpmUDn4KG9D0ma70HMGl/Y4/vDs/Xt5gfYxF1MCCcskptHYFcw9/7uwjDv3fxrzZ bl5q08By+s8WGGWWiLqmiyn18cE2bqqRXcBVuVLFZfzP1rmQKGgASaZA8c4Zmi29 soL6a0ShwsT+GEllY/G0jLFf4EBYOJsgG9sB0uuF4tJm0WJAZK7Vb/wQgL9Adqyo 86DnBMcHbQzmwt5HwoRmE55Xp3Q3uJKpAksrAvaq9+ZD/VQF+UkIae+AdiTZ9CXB JA6Z14F10knut4FL+pNTp2xzL58qrJtlyQbFFPSkCA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:03:17 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id a2e2de95 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:03:07 +0000 (UTC) Date: Mon, 25 Mar 2024 11:03:15 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 10/15] builtin/pack-refs: release allocated memory Message-ID: <23c6c20e4ee62bea22bb35c86baad0119985f426.1711360631.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Some of the command line options in `cmd_pack_refs()` require us to allocate memory. This memory is never released and thus leaking, but we paper over this leak by declaring the respective variables as `static` function-level variables, which is somewhat awkward. Refactor the code to release the allocated memory and drop the `static` declaration. While at it, remove the useless `flags` variable. Signed-off-by: Patrick Steinhardt --- builtin/pack-refs.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/builtin/pack-refs.c b/builtin/pack-refs.c index 97921beef2..ea2baeec76 100644 --- a/builtin/pack-refs.c +++ b/builtin/pack-refs.c @@ -13,15 +13,17 @@ static char const * const pack_refs_usage[] = { int cmd_pack_refs(int argc, const char **argv, const char *prefix) { - unsigned int flags = PACK_REFS_PRUNE; - static struct ref_exclusions excludes = REF_EXCLUSIONS_INIT; - static struct string_list included_refs = STRING_LIST_INIT_NODUP; - struct pack_refs_opts pack_refs_opts = { .exclusions = &excludes, - .includes = &included_refs, - .flags = flags }; - static struct string_list option_excluded_refs = STRING_LIST_INIT_NODUP; + struct ref_exclusions excludes = REF_EXCLUSIONS_INIT; + struct string_list included_refs = STRING_LIST_INIT_NODUP; + struct pack_refs_opts pack_refs_opts = { + .exclusions = &excludes, + .includes = &included_refs, + .flags = PACK_REFS_PRUNE, + }; + struct string_list option_excluded_refs = STRING_LIST_INIT_NODUP; struct string_list_item *item; int pack_all = 0; + int ret; struct option opts[] = { OPT_BOOL(0, "all", &pack_all, N_("pack everything")), @@ -45,5 +47,10 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix) if (!pack_refs_opts.includes->nr) string_list_append(pack_refs_opts.includes, "refs/tags/*"); - return refs_pack_refs(get_main_ref_store(the_repository), &pack_refs_opts); + ret = refs_pack_refs(get_main_ref_store(the_repository), &pack_refs_opts); + + clear_ref_exclusions(&excludes); + string_list_clear(&included_refs, 0); + string_list_clear(&option_excluded_refs, 0); + return ret; } From patchwork Mon Mar 25 10:03:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601865 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 072F7157A65 for ; Mon, 25 Mar 2024 10:03:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711361005; cv=none; b=eAoLTQm4XRsYbjJJe8RZsWkNOapRb/sKc/6c4QGFAmz1UDHjL16VW3jzfho7L453h63nkC4C5Sahwo8rQbBB4zzcl35GjX1TwEIjft8f5D2+vSxJgUaOewc6P6cjpOFEmGGCaHaWAj7/KuVaRk2kXESsZhDBmEZjbtGqJwMAuys= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711361005; c=relaxed/simple; bh=gXrTSO17S441FzVQQVFduhzayVPW5LYYv3IVBx4ofaI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=TQHtVbBSI5pw8GaNiy2HUi3tB1HE4RiKWMEp9Ic3DWeMrwE6gb5xBK6ucOBs2iorjUAFYfXoUHDKKpyP2DqNqJtKzbKXQRSZrY8WT4N2ZwZu3IB3L3zJETD2j2dC6ChfvRCodUgE41kK3XSnFvehRQrqzDeD1opA3bCThz56d6g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=VUdVkIsr; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=GgOH1v5m; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="VUdVkIsr"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="GgOH1v5m" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.nyi.internal (Postfix) with ESMTP id 2AE3E1380083; Mon, 25 Mar 2024 06:03:23 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Mon, 25 Mar 2024 06:03:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711361003; x=1711447403; bh=nOHTcM7vJ5 r6Vre8Ag4Y2AEW5xyggp4/uXmSQcYUkoE=; b=VUdVkIsrSqwfCCGe4ld8mjVpUK BuCF6wiJQnhylYEsO00g8vkhiNTStrzdCHi+gjij6YXEW4qobYWbX1xJ+U5yFQaa IfWv3Hh90jvL73ulcc15Rso3QuiJLyQP+HGLx6IxQi5UDs+0XC9nrgoz4IZB7ZcM MeoXrLOKDeHSivzMG0x74pVohmG3MotsJxGK9jtFGcIvE5wSLZaiMozhC47sgw/y MuirzmekVI0GE2QC6H4wkXu72P+xivupITSQOOT0Ut4Y+cvKpDBy+Uy1FMjTLnGR AnaMJs5E1NMmmqd0PwUX0sjtPpUZiNkOjnWE46ZmgzyAr3/1Pq1E25k7+Wqw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711361003; x=1711447403; bh=nOHTcM7vJ5r6Vre8Ag4Y2AEW5xyg gp4/uXmSQcYUkoE=; b=GgOH1v5mDnmmj6CjJ2fIWoNB/h34hNheriv8c2GPbxMJ 0KUPMiwEvV7SYACWLXjdOUESikPaQiEogq0NtPMYLsMAJ08eqZlvhaEAf1xj4jlh TXDR7cHgboil3eq9OT9D2GJ9+aKbNWE6zB4vpAXHAeR4tW7JIBN+XkuJih6M7Uda tnOn2mmaKvHJbgHFgQrmdRE88M/ZUL0KMjz8pdAuQ9VYf7PE4Q4Nm3ZMNNqR67qH sXbrEikJ+WTcM2sLvrkjVBt3LA3f9off9ad31r0w7BaY5hmiJjzzMCXT9HEjsax5 JySMnVI5n439j6+/TFIOzRbI4DQd43ouX7wIw+zqDw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepfeenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:03:22 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 06eed119 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:03:12 +0000 (UTC) Date: Mon, 25 Mar 2024 11:03:20 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 11/15] builtin/pack-refs: introduce new "--auto" flag Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Calling git-pack-refs(1) will unconditionally cause it to pack all requested refs regardless of the current state of the ref database. For example: - With the "files" backend we will end up rewriting the complete "packed-refs" file even if only a single ref would require compaction. - With the "reftable" backend we will end up always compacting all tables into a single table. This behaviour can be completely unnecessary depending on the backend and is thus wasteful. With the introduction of the `PACK_REFS_AUTO` flag in the preceding commit we can improve this and let the backends decide for themselves whether to pack refs in the first place. Expose this functionality via a new "--auto" flag in git-pack-refs(1), which mirrors the same flag in both git-gc(1) and git-maintenance(1). Signed-off-by: Patrick Steinhardt --- Documentation/git-pack-refs.txt | 15 ++++++++++++++- builtin/pack-refs.c | 3 ++- t/t0601-reffiles-pack-refs.sh | 7 +++++++ t/t0610-reftable-basics.sh | 34 +++++++++++++++++++++++++++++++++ 4 files changed, 57 insertions(+), 2 deletions(-) diff --git a/Documentation/git-pack-refs.txt b/Documentation/git-pack-refs.txt index 284956acb3..2dcabaf74c 100644 --- a/Documentation/git-pack-refs.txt +++ b/Documentation/git-pack-refs.txt @@ -8,7 +8,7 @@ git-pack-refs - Pack heads and tags for efficient repository access SYNOPSIS -------- [verse] -'git pack-refs' [--all] [--no-prune] [--include ] [--exclude ] +'git pack-refs' [--all] [--no-prune] [--auto] [--include ] [--exclude ] DESCRIPTION ----------- @@ -60,6 +60,19 @@ with many branches of historical interests. The command usually removes loose refs under `$GIT_DIR/refs` hierarchy after packing them. This option tells it not to. +--auto:: + +Pack refs as needed depending on the current state of the ref database. The +behavior depends on the ref format used by the repository and may change in the +future. ++ + - "files": No special handling for `--auto` has been implemented. ++ + - "reftable": Tables are compacted such that they form a geometric + sequence. For two tables N and N+1, where N+1 is newer, this + maintains the property that N is at least twice as big as N+1. Only + tables that violate this property are compacted. + --include :: Pack refs based on a `glob(7)` pattern. Repetitions of this option diff --git a/builtin/pack-refs.c b/builtin/pack-refs.c index ea2baeec76..db40825666 100644 --- a/builtin/pack-refs.c +++ b/builtin/pack-refs.c @@ -7,7 +7,7 @@ #include "revision.h" static char const * const pack_refs_usage[] = { - N_("git pack-refs [--all] [--no-prune] [--include ] [--exclude ]"), + N_("git pack-refs [--all] [--no-prune] [--auto] [--include ] [--exclude ]"), NULL }; @@ -28,6 +28,7 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix) struct option opts[] = { OPT_BOOL(0, "all", &pack_all, N_("pack everything")), OPT_BIT(0, "prune", &pack_refs_opts.flags, N_("prune loose refs (default)"), PACK_REFS_PRUNE), + OPT_BIT(0, "auto", &pack_refs_opts.flags, N_("auto-pack refs as needed"), PACK_REFS_AUTO), OPT_STRING_LIST(0, "include", pack_refs_opts.includes, N_("pattern"), N_("references to include")), OPT_STRING_LIST(0, "exclude", &option_excluded_refs, N_("pattern"), diff --git a/t/t0601-reffiles-pack-refs.sh b/t/t0601-reffiles-pack-refs.sh index b1cf587347..219a495451 100755 --- a/t/t0601-reffiles-pack-refs.sh +++ b/t/t0601-reffiles-pack-refs.sh @@ -164,6 +164,13 @@ test_expect_success 'test --exclude takes precedence over --include' ' git pack-refs --include "refs/heads/pack*" --exclude "refs/heads/pack*" && test -f .git/refs/heads/dont_pack5' +test_expect_success '--auto packs and prunes refs as usual' ' + git branch auto && + test_path_is_file .git/refs/heads/auto && + git pack-refs --auto --all && + test_path_is_missing .git/refs/heads/auto +' + test_expect_success 'see if up-to-date packed refs are preserved' ' git branch q && git pack-refs --all --prune && diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index a53d1dc493..6de7529575 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -387,6 +387,40 @@ test_expect_success 'pack-refs: compaction raises locking errors' ' test_cmp expect err ' +test_expect_success 'pack-refs: auto compaction' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + + test_commit A && + + # The tables should have been auto-compacted, and thus auto + # compaction should not have to do anything. + ls -1 .git/reftable >tables-expect && + test_line_count = 4 tables-expect && + git pack-refs --auto && + ls -1 .git/reftable >tables-actual && + test_cmp tables-expect tables-actual && + + # Lock all tables write some refs. Auto-compaction will be + # unable to compact tables and thus fails gracefully, leaving + # the stack in a sub-optimal state. + ls .git/reftable/*.ref | + while read table + do + touch "$table.lock" || exit 1 + done && + git branch B && + git branch C && + rm .git/reftable/*.lock && + test_line_count = 5 .git/reftable/tables.list && + + git pack-refs --auto && + test_line_count = 1 .git/reftable/tables.list + ) +' + test_expect_success 'pack-refs: prunes stale tables' ' test_when_finished "rm -rf repo" && git init repo && From patchwork Mon Mar 25 10:03:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601866 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0599318146A for ; Mon, 25 Mar 2024 10:03:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711361009; cv=none; b=BjFF3gSiPK17HdScl56mlBtC0TsfwBcX3qfIfUM5UdxO3cq4cVZESscAnZo1jtj8ubPWvBo8Okkrl3qHj1PY+IG11TAhh5+nVhWGIdNFgW4HE/SvmpJ9o7iX+bODojxMre7caLWnx9KgJsnnkvXBzJ65owJRfCWTYVtDqMnIClU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711361009; c=relaxed/simple; bh=pQoREen2kvlD/nI8e2OFfxMWKqRQL+IDn2OQcypuvck=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=DYvzUCuMMIEZYKOvBJCDSmkSoEaYgIs+pTm+704Vqn1DP6pKc+TagylqxVBoHzRv8G2In9VH2AgcuggqhESUacRcpYkz2J9++3wOOuxiH9IGemwsFjos0O58ianH6B1cs/IMRwCfcCpXwK6AYbcqQyfSZUJFVOIa92tnCuusAvs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=IAdGY1vq; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=oIgUGUh+; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="IAdGY1vq"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="oIgUGUh+" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfout.nyi.internal (Postfix) with ESMTP id 5861F13800AD; Mon, 25 Mar 2024 06:03:27 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 25 Mar 2024 06:03:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711361007; x=1711447407; bh=6ByXzITrTv ztJ1YR40SpTZ0kfsGPNzO+iVqLYGasWU0=; b=IAdGY1vq2i1Lb760ZrOr/PvC4r 7rF6gG8sLqSFJmrHMfCd3hpwztOCWScpA42DEyqFuNad37VqmSLutQ91Swnox6wV GhIA+2ejfWTPMsvNBwbDckGhj8GbZAef1fm5kT4XT/La1q4L2+GzW5LVShxs3B53 5Y8rHlUwHdzlCrehYnCfj3ttQDNePPVVw1rshks3C1iHQSeKPfaFCg8s2IR3WbRg X4isedz+FktEo8jBRbUFph3hgtssINW/VmvuX0iqSXbwLbtV4PJtSsZysSgqyEiE i1PKP0t5ppL/g4z2JTJnc3YRMTFsFaFyUy/AFQak2knPm0huf/2dYGQDIUPw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711361007; x=1711447407; bh=6ByXzITrTvztJ1YR40SpTZ0kfsGP NzO+iVqLYGasWU0=; b=oIgUGUh+VAkOyGwNpyMw/KP1pxSGxCU9j3odNFeZv+Nb hOu+M/hHmSpG9YuJKhT9nr5QwS51wjBOUCU4TK8ETE2TW71IJfqg42aQkqx69hPw AgDd+ybVA0dDWnGIQ04YmnPxFSefNMJss5/FKKwhXX4b5uke6ZwW9hxzfqRUCSRZ +uR43osFGddoGKDc4Bcje1LTbbyWUh38S5h8YLf1w1l+CnpuRUqwLxFVhaOZWyaB OSOkN7MDIAvTExiL54aiZychR3dJlFJ6aLkL9cIV2pTNcKmCg7X37yaMgNAfHGNb 3iP0zzv7YRlfyzHCjFsefi1c2Qp4u4aLKayYYHyx0g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledgudduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:03:26 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 111a1590 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:03:16 +0000 (UTC) Date: Mon, 25 Mar 2024 11:03:24 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 12/15] builtin/gc: move `struct maintenance_run_opts` Message-ID: <94cb036345230691f802f4afae4e801f74f90ab2.1711360631.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We're about to start using `struct maintenance_run_opts` in `maintenance_task_pack_refs()`. Move its definition up to prepare for this. Signed-off-by: Patrick Steinhardt --- builtin/gc.c | 53 ++++++++++++++++++++++++++-------------------------- 1 file changed, 26 insertions(+), 27 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index cb80ced6cb..e0029c88f9 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -180,7 +180,32 @@ static void gc_config(void) git_config(git_default_config, NULL); } -struct maintenance_run_opts; +enum schedule_priority { + SCHEDULE_NONE = 0, + SCHEDULE_WEEKLY = 1, + SCHEDULE_DAILY = 2, + SCHEDULE_HOURLY = 3, +}; + +static enum schedule_priority parse_schedule(const char *value) +{ + if (!value) + return SCHEDULE_NONE; + if (!strcasecmp(value, "hourly")) + return SCHEDULE_HOURLY; + if (!strcasecmp(value, "daily")) + return SCHEDULE_DAILY; + if (!strcasecmp(value, "weekly")) + return SCHEDULE_WEEKLY; + return SCHEDULE_NONE; +} + +struct maintenance_run_opts { + int auto_flag; + int quiet; + enum schedule_priority schedule; +}; + static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts) { struct child_process cmd = CHILD_PROCESS_INIT; @@ -773,26 +798,6 @@ static const char *const builtin_maintenance_run_usage[] = { NULL }; -enum schedule_priority { - SCHEDULE_NONE = 0, - SCHEDULE_WEEKLY = 1, - SCHEDULE_DAILY = 2, - SCHEDULE_HOURLY = 3, -}; - -static enum schedule_priority parse_schedule(const char *value) -{ - if (!value) - return SCHEDULE_NONE; - if (!strcasecmp(value, "hourly")) - return SCHEDULE_HOURLY; - if (!strcasecmp(value, "daily")) - return SCHEDULE_DAILY; - if (!strcasecmp(value, "weekly")) - return SCHEDULE_WEEKLY; - return SCHEDULE_NONE; -} - static int maintenance_opt_schedule(const struct option *opt, const char *arg, int unset) { @@ -809,12 +814,6 @@ static int maintenance_opt_schedule(const struct option *opt, const char *arg, return 0; } -struct maintenance_run_opts { - int auto_flag; - int quiet; - enum schedule_priority schedule; -}; - /* Remember to update object flag allocation in object.h */ #define SEEN (1u<<0) From patchwork Mon Mar 25 10:03:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601867 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3935718786A for ; Mon, 25 Mar 2024 10:03:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711361013; cv=none; b=nTt1Qh/JH1sL2MS7ke6Xjl/x+VoWAElVj+5hzGRIS2iUaRIfw17w6MeUgoSrM2VLBmD6wAaQKMCAHAu2QESyUMOedqPB69eUxRpxhFtjKzmhwpz0kVlOmB/p/EVUT3+ZUHzQ0ahfi/Ty2FZ2ZiueYAH+iHSVbwo0mHvwZKCMQQA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711361013; c=relaxed/simple; bh=OYiRtAflB5iadWfTLCmpZHC8Ib7HMdltVJvtsAodO6w=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=pT8rxKcsenTsFj1hgigkzdITwCQ+AeVr7UNijw2V0jQbP7ivfl9kYDYGL9HfOWnsVQEGPyyp/X/fYuo3lAijCx4LBwO4VL5yHNGPw9g34zVrJMEO0hbQFOY/J9Ml20m1ugt1Ua0NnPXC4oyvUq2aSP52h4ua2/6Xdoi+NO9tz/w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=doZZ5pk3; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Th2kucyT; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="doZZ5pk3"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Th2kucyT" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.nyi.internal (Postfix) with ESMTP id 83ECE13800AD; Mon, 25 Mar 2024 06:03:31 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Mon, 25 Mar 2024 06:03:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711361011; x=1711447411; bh=NxvyvfX12a EIroMEWyoMiEmdt8+96uXWqX5LCxzg2FI=; b=doZZ5pk3Z80Dn4ch5Q7mOcb+rk 6sWfOiJ+uiqCVBjfSc1eNmag0+GPum4bpm2XeCQcM3WlATMLcJuialit3chJ8s3M K5L2ZCSiepdZ3Gk08kRV6IzOmf/WEbyTTixxllmhqbziTMST8X9oClMd/zA8Gdmq hIszDVv7AkaHpNfCb9z19/RGEpU08J3XQ/VCbawl0+unKjW1ZZgp9NYEYwryV9l2 iqZNg5B+fiuFDoVdPDd+4E7wxXjxkVPXw3Q9CofRya8PQ3lNam42DrsxBEnzNdU4 0Qr+GS9Jc/vAxHZv2TkAwSgYB0ea+q1F4dg2sqJfexWPcqtJoSb80kkrBqZg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711361011; x=1711447411; bh=NxvyvfX12aEIroMEWyoMiEmdt8+9 6uXWqX5LCxzg2FI=; b=Th2kucyT7+CAL8P2x7ZYKu/D3MTlIkk915HPNyvp2L/P P76F15d+l22qvU/PW+8DKvaPi+Mn+TvjZowuJqSYmKJjaeFceSN+85TCosY0+YSr /Vj+T+XD69T2874ESJPeJGEPKXk34dASWdafTFT4OFVDL6CTukbHwGtmAG2CYY61 I+onqnxJrT2fXdfnY/iiDwZWKYB4z9SgsQuDj8tPrPmM9i41PQzkNVu0UVWg9r63 FeIg3RO2/PBfyqQH22YYtpAgrfRQIxLbALSTRlM5wq71+zIYr4/qKQ+w1JA+gayG SsIqaDozhde2fVPY74apfU8sdxCcSCnviTSdnclOgg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepgeenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:03:30 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id dac07008 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:03:20 +0000 (UTC) Date: Mon, 25 Mar 2024 11:03:28 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 13/15] t6500: extract objects with "17" prefix Message-ID: <9657c67b3b9136be27e469da40ef0520bea6598d.1711360631.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The ".git/obects/17/" shard is somewhat special because it is used by git-gc(1) to estimate how many objects there are by extrapolating the number of objects in that shard, only. In t6500 we thus have a hard coded set of data that, when written to the object database, result in blobs starting with that prefix. We are about to need such "17"-prefixed objects in another test suite. Extract them into "t/oid-info/hash-info" so that they can be reused by other tests. Signed-off-by: Patrick Steinhardt --- t/oid-info/hash-info | 12 ++++++++++++ t/t6500-gc.sh | 30 +++++++----------------------- 2 files changed, 19 insertions(+), 23 deletions(-) diff --git a/t/oid-info/hash-info b/t/oid-info/hash-info index d0736dd1a0..b8a5bcb187 100644 --- a/t/oid-info/hash-info +++ b/t/oid-info/hash-info @@ -15,3 +15,15 @@ empty_blob sha256:473a0f4c3be8a93681a267e3b1e9a7dcda1185436fe141f7749120a3037218 empty_tree sha1:4b825dc642cb6eb9a060e54bf8d69288fbee4904 empty_tree sha256:6ef19b41225c5369f1c104d45d8d85efa9b057b53b14b4b9b939dd74decc5321 + +blob17_1 sha1:263 +blob17_1 sha256:34 + +blob17_2 sha1:410 +blob17_2 sha256:174 + +blob17_3 sha1:523 +blob17_3 sha256:313 + +blob17_4 sha1:790 +blob17_4 sha256:481 diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh index 18fe1c25e6..43d40175f8 100755 --- a/t/t6500-gc.sh +++ b/t/t6500-gc.sh @@ -11,23 +11,7 @@ test_expect_success 'setup' ' # behavior, make sure we always pack everything to one pack by # default git config gc.bigPackThreshold 2g && - - # These are simply values which, when hashed as a blob with a newline, - # produce a hash where the first byte is 0x17 in their respective - # algorithms. - test_oid_cache <<-EOF - obj1 sha1:263 - obj1 sha256:34 - - obj2 sha1:410 - obj2 sha256:174 - - obj3 sha1:523 - obj3 sha256:313 - - obj4 sha1:790 - obj4 sha256:481 - EOF + test_oid_init ' test_expect_success 'gc empty repository' ' @@ -114,8 +98,8 @@ test_expect_success 'pre-auto-gc hook can stop auto gc' ' # We need to create two object whose sha1s start with 17 # since this is what git gc counts. As it happens, these # two blobs will do so. - test_commit "$(test_oid obj1)" && - test_commit "$(test_oid obj2)" && + test_commit "$(test_oid blob17_1)" && + test_commit "$(test_oid blob17_2)" && git gc --auto >../out.actual 2>../err.actual ) && @@ -146,13 +130,13 @@ test_expect_success 'auto gc with too many loose objects does not attempt to cre # We need to create two object whose sha1s start with 17 # since this is what git gc counts. As it happens, these # two blobs will do so. - test_commit "$(test_oid obj1)" && - test_commit "$(test_oid obj2)" && + test_commit "$(test_oid blob17_1)" && + test_commit "$(test_oid blob17_2)" && # Our first gc will create a pack; our second will create a second pack git gc --auto && ls .git/objects/pack/pack-*.pack | sort >existing_packs && - test_commit "$(test_oid obj3)" && - test_commit "$(test_oid obj4)" && + test_commit "$(test_oid blob17_3)" && + test_commit "$(test_oid blob17_4)" && git gc --auto 2>err && test_grep ! "^warning:" err && From patchwork Mon Mar 25 10:03:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601869 Received: from fhigh5-smtp.messagingengine.com (fhigh5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A3C71188A74 for ; Mon, 25 Mar 2024 10:03:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711361018; cv=none; b=HAXZtqNIJzauxSnY3ugYg3IRE5EICuy2odAbcDBn3YiiJFaWAlDkEu0B/1rsFVJhKDb6oDmWRTqSNDQbzzoMeNL82lY3n+cUOd5KF7TR/r4CtiMNMgheZC0KJii+wXDyg+/vhTAv1LyvWWmh4jXiIWmoAB9+3bmh2mAkmuWon0I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711361018; c=relaxed/simple; bh=87gPnVVFnXN1kD1yeiWaJAJ03Yo1/vMjiRXN8phy1w4=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=N33WQMBDuGSL3f/uDsBV6gdCXIk7INvfc8YDeTrwsA0TkUbGsJz45lnT8mLJ7NsI/+yCpGa+OboAUQJGnsppPfeuEiX5PuCrj/fm6IAHzz7a4eZR608cQlO7tBKreQULRCvviFtQfkDMhWZUnoFKOBBAWgpD+gjvUpN4oIfSPVY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=mDn7XkP9; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=kNndxH3Q; arc=none smtp.client-ip=103.168.172.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="mDn7XkP9"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="kNndxH3Q" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfhigh.nyi.internal (Postfix) with ESMTP id CA60711400DF; Mon, 25 Mar 2024 06:03:35 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 25 Mar 2024 06:03:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711361015; x=1711447415; bh=RfZSjyCfYm h61+5ftsHOhbT46ZZJtn+8Pp4RVBK441o=; b=mDn7XkP9N+yCKBXNRf7UYqyoU1 xfN73yzKvd2FVKDEZQ6hhYpact+hTOga22dab+d+uyo//lG71yXfi9yqrZ1pMN3L vgA/lgDFQ7OOwGj5GAX9+AhSe8LDlWZsSwrmTpnrcYcFiNH8hoR3DvInOczcXIwg F9WHwflmbKImqydmVSqu9NTv8dZkH2iHMo6q+4zbUkg6FFRJNceYcDR6dd2YC00X +WMLVDeDWpM7toaTAe89zwVQIzXn6qKLohX6cxrqalpm6HImwV7IFM5qacYlt8Pb y0j2aLxcyl1QtkyQNMQxpgIEzMbjjKeETNERCUzO1jjRPW7SZJHc1z1oisig== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711361015; x=1711447415; bh=RfZSjyCfYmh61+5ftsHOhbT46ZZJ tn+8Pp4RVBK441o=; b=kNndxH3QYCrd5ZA5ooxMWw4SjVgSNMfleMd6BqseHz3C c7O4IfRJb17MUF93VAKUQM4b0zhM0KnpfNAafNNK7W4OyEDYN9VpsU6QImEQE8Uy Fnd2tQML/wdKROQJ5NY3dL3+HdWn4sm42+kxNfsGuWo6heYJIyiczJF0KgwBa+gR +FFGr0+lStGZbddvbcgOuYym2138E/ZlSEflz1POkRI/a1WKwSPr++e4Zdd7sesP qNkuRqK5vpRZO3fMTPgEKEEHy2qsmjjfUpDh33ytJ+jfLuSpR+2FWpfSyLtk2het pSwqBs6Uk+xkNTgzwtEA1c0AUy0HTmyFtC+c1r1U8A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:03:34 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id a9d4e16a (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:03:24 +0000 (UTC) Date: Mon, 25 Mar 2024 11:03:32 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 14/15] builtin/gc: forward git-gc(1)'s `--auto` flag when packing refs Message-ID: <3eaff8289be5032cfd425272f63a6a02ae432eb1.1711360631.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Forward the `--auto` flag to git-pack-refs(1) when it has been invoked with this flag itself. This does not change anything for the "files" backend, which will continue to eagerly pack refs. But it does ensure that the "reftable" backend only compacts refs as required. This change does not impact git-maintenance(1) because this command will in fact never run the pack-refs task when run with `--auto`. This issue will be addressed in a subsequent commit. Signed-off-by: Patrick Steinhardt --- builtin/gc.c | 21 ++++++++++++--------- t/t0610-reftable-basics.sh | 19 ++++++++++++++++--- 2 files changed, 28 insertions(+), 12 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index e0029c88f9..bf1f2a621a 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -212,6 +212,9 @@ static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts * cmd.git_cmd = 1; strvec_pushl(&cmd.args, "pack-refs", "--all", "--prune", NULL); + if (opts->auto_flag) + strvec_push(&cmd.args, "--auto"); + return run_command(&cmd); } @@ -572,7 +575,7 @@ static int report_last_gc_error(void) return ret; } -static void gc_before_repack(void) +static void gc_before_repack(struct maintenance_run_opts *opts) { /* * We may be called twice, as both the pre- and @@ -583,7 +586,7 @@ static void gc_before_repack(void) if (done++) return; - if (pack_refs && maintenance_task_pack_refs(NULL)) + if (pack_refs && maintenance_task_pack_refs(opts)) die(FAILED_RUN, "pack-refs"); if (prune_reflogs) { @@ -599,7 +602,6 @@ static void gc_before_repack(void) int cmd_gc(int argc, const char **argv, const char *prefix) { int aggressive = 0; - int auto_gc = 0; int quiet = 0; int force = 0; const char *name; @@ -608,6 +610,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) int keep_largest_pack = -1; timestamp_t dummy; struct child_process rerere_cmd = CHILD_PROCESS_INIT; + struct maintenance_run_opts opts = {0}; struct option builtin_gc_options[] = { OPT__QUIET(&quiet, N_("suppress progress reporting")), @@ -618,7 +621,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) OPT_MAGNITUDE(0, "max-cruft-size", &max_cruft_size, N_("with --cruft, limit the size of new cruft packs")), OPT_BOOL(0, "aggressive", &aggressive, N_("be more thorough (increased runtime)")), - OPT_BOOL_F(0, "auto", &auto_gc, N_("enable auto-gc mode"), + OPT_BOOL_F(0, "auto", &opts.auto_flag, N_("enable auto-gc mode"), PARSE_OPT_NOCOMPLETE), OPT_BOOL_F(0, "force", &force, N_("force running gc even if there may be another gc running"), @@ -663,7 +666,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) if (quiet) strvec_push(&repack, "-q"); - if (auto_gc) { + if (opts.auto_flag) { /* * Auto-gc should be least intrusive as possible. */ @@ -688,7 +691,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) if (lock_repo_for_gc(force, &pid)) return 0; - gc_before_repack(); /* dies on failure */ + gc_before_repack(&opts); /* dies on failure */ delete_tempfile(&pidfile); /* @@ -713,7 +716,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) name = lock_repo_for_gc(force, &pid); if (name) { - if (auto_gc) + if (opts.auto_flag) return 0; /* be quiet on --auto */ die(_("gc is already running on machine '%s' pid %"PRIuMAX" (use --force if not)"), name, (uintmax_t)pid); @@ -728,7 +731,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) atexit(process_log_file_at_exit); } - gc_before_repack(); + gc_before_repack(&opts); if (!repository_format_precious_objects) { struct child_process repack_cmd = CHILD_PROCESS_INIT; @@ -783,7 +786,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) !quiet && !daemonized ? COMMIT_GRAPH_WRITE_PROGRESS : 0, NULL); - if (auto_gc && too_many_loose_objects()) + if (opts.auto_flag && too_many_loose_objects()) warning(_("There are too many unreachable loose objects; " "run 'git prune' to remove them.")); diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index 6de7529575..b37d8bf3b1 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -387,7 +387,9 @@ test_expect_success 'pack-refs: compaction raises locking errors' ' test_cmp expect err ' -test_expect_success 'pack-refs: auto compaction' ' +for command in pack-refs gc +do +test_expect_success "$command: auto compaction" ' test_when_finished "rm -rf repo" && git init repo && ( @@ -395,14 +397,24 @@ test_expect_success 'pack-refs: auto compaction' ' test_commit A && + # We need a bit of setup to ensure that git-gc(1) actually + # triggers, and that it does not write anything to the refdb. + git config gc.auto 1 && + git config gc.autoDetach 0 && + git config gc.reflogExpire never && + git config gc.reflogExpireUnreachable never && + test_oid blob17_1 | git hash-object -w --stdin && + # The tables should have been auto-compacted, and thus auto # compaction should not have to do anything. ls -1 .git/reftable >tables-expect && test_line_count = 4 tables-expect && - git pack-refs --auto && + git $command --auto && ls -1 .git/reftable >tables-actual && test_cmp tables-expect tables-actual && + test_oid blob17_2 | git hash-object -w --stdin && + # Lock all tables write some refs. Auto-compaction will be # unable to compact tables and thus fails gracefully, leaving # the stack in a sub-optimal state. @@ -416,10 +428,11 @@ test_expect_success 'pack-refs: auto compaction' ' rm .git/reftable/*.lock && test_line_count = 5 .git/reftable/tables.list && - git pack-refs --auto && + git $command --auto && test_line_count = 1 .git/reftable/tables.list ) ' +done test_expect_success 'pack-refs: prunes stale tables' ' test_when_finished "rm -rf repo" && From patchwork Mon Mar 25 10:03:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13601868 Received: from fhigh5-smtp.messagingengine.com (fhigh5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1FF1189240 for ; Mon, 25 Mar 2024 10:03:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711361021; cv=none; b=Xo6LzIpoUts4F78M/fz7SW6tJdAzgq4v688OQp0JQuq5/5rPZUxPuhX6O5cx6ifz5qgL0pB+GJkO4fhf32MKZlL+fXEe5RsNWvfs4Fd8yO2h8VOXghKcK9XaFH16raHTNHzHrbIebewWRwOaSAAk/abkkAmDD9xYTOkT6lxttZ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711361021; c=relaxed/simple; bh=Nfg+GC6HqQ41R8yP/YI/uU/xN/pcLYMGvNsqivs/UbE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=uVx1IEt0lxPginHyGzAeWvyL9vKUplwSg+EvkpB7r4GGv9aMeKYfmQxCIhDBF3GaHoh5sYWdV2fP8+Rjur/hqnlNtLuPMyFjsBEYNohwFh3AII7vg9rfkOCRsJzwmzi9LaRIE3WVW+08/r66+qVix7q14Y/rs4cFGylcMXev9sk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=Oz7d6feP; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Hvd5y0kg; arc=none smtp.client-ip=103.168.172.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="Oz7d6feP"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Hvd5y0kg" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfhigh.nyi.internal (Postfix) with ESMTP id F189B11400A1; Mon, 25 Mar 2024 06:03:38 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 25 Mar 2024 06:03:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1711361018; x=1711447418; bh=Sb0DuT8OEA +8riykvDTyEjwvGB9dWgk8HcZQQnJyLAQ=; b=Oz7d6fePwZ3u4ff9SxcHD5ZbDv UpxGPKWhC8kafy+HYRPbldWY1pzZ2xvUvuQiVnk6klkdz6aBdOIq86GM1HlQt6Xu VutAehb7J6slmX3/dmKC9NqxIk9cOJDrPn8VpS2nOb0xh5CdlsnK6iumeCAuPd1g 8gLJwa4IPXMf3LHCzcX1M+XKbzPY6vMZzO7UUMSjMDhYSmKSth/k9uj8I4um/wQo 84rHMHv3l75J9LsMaiNZzvpBhZYozS+tx2le+uDStijkIniGFZcBqp09zJqog7mD m1LWW27tHVzOLl15Cai7btfrm/4s4o3kHIfrIMMKO1NqN1iP6IRUvgT3WyRA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1711361018; x=1711447418; bh=Sb0DuT8OEA+8riykvDTyEjwvGB9d Wgk8HcZQQnJyLAQ=; b=Hvd5y0kgFG4aODEL5YEJZyjNGlpnpTP4qJzjqeHVnOVQ JmhSqplehi9X4d6QQ2JxtNRy9cIN7mj3VT3WhrLbKzRm+LbOENzkNRuIwkEU/Vhp f66myoWkrbLOdQa2Y48Z2Z8o1sUyEwCKnQaZwS114029QLuX3IYtCdlzCA+wGLpN k1Lhwtldg/y2dX+vCdp7x4pDUi2+c0nC48Cq5NpzokLkzH2YhG6/0nxz/bAq3Kjq LcE+kFUECemXmMmr9NpXiPa0HOdG1w9TGgp4G9hjMxZd1SIK3BhLsUJmAysfkHKQ UWF8MdUS8g0zxp5zdRJIdwsx8ncoYUlKeO9GWYh1Og== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledruddtledguddtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 25 Mar 2024 06:03:37 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 365ad41e (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 25 Mar 2024 10:03:28 +0000 (UTC) Date: Mon, 25 Mar 2024 11:03:36 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee , Karthik Nayak Subject: [PATCH v2 15/15] builtin/gc: pack refs when using `git maintenance run --auto` Message-ID: <1bdea3b316b6ab88d5e9c8541e8f9348fc36c998.1711360631.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When running `git maintenance run --auto`, then the various subtasks will only run as needed. Thus, we for example end up only packing loose objects if we hit a certain threshold. Interestingly enough, the "pack-refs" task is actually _never_ executed when the auto-flag is set because it does not have a condition at all. As 41abfe15d9 (maintenance: add pack-refs task, 2021-02-09) mentions: The 'auto_condition' function pointer is left NULL for now. We could extend this in the future to have a condition check if pack-refs should be run during 'git maintenance run --auto'. It is not quite clear from that quote whether it is actually intended that the task doesn't run at all in this mode. Also, no test was added to verify this behaviour. Ultimately though, it feels quite surprising that `git maintenance run --auto --task=pack-refs` would quietly never do anything at all. In any case, now that we do have the logic in place to let ref backends decide whether or not to repack refs, it does make sense to wire it up accordingly. With the "reftable" backend we will thus now perform auto-compaction, which optimizes the refdb as needed. But for the "files" backend we now unconditionally pack refs as it does not yet know to handle the "auto" flag. Arguably, this can be seen as a bug fix given that previously the task never did anything at all. Eventually though we should amend the "files" backend to use some heuristics for auto compaction, as well. Signed-off-by: Patrick Steinhardt --- builtin/gc.c | 12 +++++++++++- t/t0601-reffiles-pack-refs.sh | 10 ++++++++++ t/t0610-reftable-basics.sh | 2 +- 3 files changed, 22 insertions(+), 2 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index bf1f2a621a..3c874b248b 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -206,6 +206,16 @@ struct maintenance_run_opts { enum schedule_priority schedule; }; +static int pack_refs_condition(void) +{ + /* + * The auto-repacking logic for refs is handled by the ref backends and + * exposed via `git pack-refs --auto`. We thus always return truish + * here and let the backend decide for us. + */ + return 1; +} + static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts) { struct child_process cmd = CHILD_PROCESS_INIT; @@ -1298,7 +1308,7 @@ static struct maintenance_task tasks[] = { [TASK_PACK_REFS] = { "pack-refs", maintenance_task_pack_refs, - NULL, + pack_refs_condition, }, }; diff --git a/t/t0601-reffiles-pack-refs.sh b/t/t0601-reffiles-pack-refs.sh index 219a495451..7d4ab0b91a 100755 --- a/t/t0601-reffiles-pack-refs.sh +++ b/t/t0601-reffiles-pack-refs.sh @@ -370,4 +370,14 @@ test_expect_success 'pack-refs does not drop broken refs during deletion' ' test_cmp expect actual ' +test_expect_success 'maintenance --auto unconditionally packs loose refs' ' + git update-ref refs/heads/something HEAD && + test_path_is_file .git/refs/heads/something && + git rev-parse refs/heads/something >expect && + git maintenance run --task=pack-refs --auto && + test_path_is_missing .git/refs/heads/something && + git rev-parse refs/heads/something >actual && + test_cmp expect actual +' + test_done diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index b37d8bf3b1..931d888bbb 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -387,7 +387,7 @@ test_expect_success 'pack-refs: compaction raises locking errors' ' test_cmp expect err ' -for command in pack-refs gc +for command in pack-refs gc "maintenance run --task=pack-refs" do test_expect_success "$command: auto compaction" ' test_when_finished "rm -rf repo" &&