From patchwork Mon Apr 8 16:16:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Justin Tobler X-Patchwork-Id: 13621357 Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 30364140386 for ; Mon, 8 Apr 2024 16:17:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712593026; cv=none; b=nud3yt8/QT0AiIEG3mM0BJv1qdOgfaI0AJkcOebXYEDEcn160RWejlFyrdX2AQDSErW/Glbu38vmsHRihIvJieQ0amP/kzPKkyEauC37JcdohsRU6qiISPqTKg/6tFlZx3yAtLveOAHVGMcb96MVLniurUGNlX43m94YKbrQxrA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712593026; c=relaxed/simple; bh=lDcTbEshzPX9Q9GzNBEIhQPLw45DqIjU2UzSFZAphCA=; h=Message-Id:In-Reply-To:References:From:Date:Subject:Content-Type: MIME-Version:To:Cc; b=W2TYkDn1ssa2J/+c8MV8WxIl+3nsnL4/e42EEcGO/obbROToT9Qrw7pHevZSkcCIvpHkfu9TtAlqRyefjxkowa5SKpGszSdq80aJDG4ZgI1/Q1sqKncgK1LpQ8TkBnmG6DY6MBHN05arosE2U0eRVxkW0JYCFhHrqB1G0DaUQFw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bI+lEhvV; arc=none smtp.client-ip=209.85.128.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bI+lEhvV" Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-416511f13aaso10719885e9.1 for ; Mon, 08 Apr 2024 09:17:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712593019; x=1713197819; darn=vger.kernel.org; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=haAH7RPGDGXrPMNWd5MpiDXlapJDFBYn8Gm0/3RlgVM=; b=bI+lEhvVOl8nIev58ju3Brn1MJ2Jt1+yPJAwE377VybFT4kn66XKlR44ftNB5r3evR vTn+pqVRPIsfV6WNUuIj4a9LfWoyBGR10/RMoPvm3rXRuwfjRMIfrlJJyqG2QAtAc2Ug C2+d75iASecU3ZLgPOycQx0Iz9Txrnk4WjPQzYNrBF8a2ndZcm9pSn8Dlg2TaYpzyQoZ vRb+fyqD7BPCYzoz7a7XRgJM9mX8vKRt3W4CnrUyHcka8wU8WDxiPrYiiQFmoB/WQUal KqhMbKpRoGfN0bVOf2fnNKzJ9uZeKMx+nJkT24QzrWm9bDlGMRBqfPUu8eN9BkgepGml qa5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712593019; x=1713197819; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=haAH7RPGDGXrPMNWd5MpiDXlapJDFBYn8Gm0/3RlgVM=; b=Wk7ZOuwZNIICIQKQmGej7fWDoLdp902H4xdyUUFiv/wPyZQb8yDUt5Ef5ydDeLQda4 MF1zF8JQvyI5WzaaZtJjVEMuHfr5sK0OnJTkltK+6/syqZUWmFX8ti8ht+oMt6ikLuHc YYJvlU/EwvfnTw84YG/F3DnbNQb/Uo20HxrEhG3ZK15KONjOJqYXvWFJlqaAl1IPKst3 EtLAkN6DS8WzFEQZcGQ5/OuAiGHh5xeKiECK+lPYf8LYsCzUbwO1n4J0+ApXm1GK3+QF Vcq6NuPaF1vrFx4u1OaAo6BoaDiM3qxLe+q42pZj683C67rTkZi8zr3Z80i5GkcZf0mu IFBQ== X-Gm-Message-State: AOJu0YxKZS5clVEaCCOaJQWKUEdLDiKsV4sb1TjuzIlKn09CKfI/rY4m YsS9iQLjazkWXb8xkswC6P2JQkWy4k04SlDfcLAeylT41L6CXxCjw0NXhrZ/ X-Google-Smtp-Source: AGHT+IGA+Rygk3rsm+Uiz8ovbAQZr4MdkK5g3A/e0Wv8vpBb6zLBD5AxP4ohrIY+CkPrwuMIbxV4fA== X-Received: by 2002:a05:600c:4750:b0:416:3de5:133d with SMTP id w16-20020a05600c475000b004163de5133dmr5081966wmo.7.1712593018456; Mon, 08 Apr 2024 09:16:58 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id i21-20020a05600c355500b0041665b6e111sm5159761wmq.12.2024.04.08.09.16.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Apr 2024 09:16:58 -0700 (PDT) Message-Id: <9c8f6b336ecea50b1c0476cd274481560aa79740.1712593016.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 08 Apr 2024 16:16:53 +0000 Subject: [PATCH v6 1/3] reftable/stack: expose option to disable auto-compaction Fcc: Sent Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 To: git@vger.kernel.org Cc: Patrick Steinhardt , Karthik Nayak , Han-Wen Nienhuys , Justin Tobler , Justin Tobler From: Justin Tobler From: Justin Tobler The reftable stack already has a variable to configure whether or not to run auto-compaction, but it is inaccessible to users of the library. There exist use cases where a caller may want to have more control over auto-compaction. Move the `disable_auto_compact` option into `reftable_write_options` to allow external callers to disable auto-compaction. This will be used in a subsequent commit. Signed-off-by: Justin Tobler --- reftable/reftable-writer.h | 3 +++ reftable/stack.c | 2 +- reftable/stack.h | 1 - reftable/stack_test.c | 11 ++++++----- 4 files changed, 10 insertions(+), 7 deletions(-) diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index 7c7cae5f99b..155bf0bbe2a 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -46,6 +46,9 @@ struct reftable_write_options { * is a single line, and add '\n' if missing. */ unsigned exact_log_message : 1; + + /* boolean: Prevent auto-compaction of tables. */ + unsigned disable_auto_compact : 1; }; /* reftable_block_stats holds statistics for a single block type */ diff --git a/reftable/stack.c b/reftable/stack.c index dde50b61d69..1a7cdad12c9 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -680,7 +680,7 @@ int reftable_addition_commit(struct reftable_addition *add) if (err) goto done; - if (!add->stack->disable_auto_compact) { + if (!add->stack->config.disable_auto_compact) { /* * Auto-compact the stack to keep the number of tables in * control. It is possible that a concurrent writer is already diff --git a/reftable/stack.h b/reftable/stack.h index d919455669e..c862053025f 100644 --- a/reftable/stack.h +++ b/reftable/stack.h @@ -19,7 +19,6 @@ struct reftable_stack { int list_fd; char *reftable_dir; - int disable_auto_compact; struct reftable_write_options config; diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 351e35bd86d..4fec823f14f 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -325,7 +325,7 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void) * we can ensure that we indeed honor this setting and have * better control over when exactly auto compaction runs. */ - st->disable_auto_compact = i != n; + st->config.disable_auto_compact = i != n; err = reftable_stack_new_addition(&add, st); EXPECT_ERR(err); @@ -497,6 +497,7 @@ static void test_reftable_stack_add(void) struct reftable_write_options cfg = { .exact_log_message = 1, .default_permissions = 0660, + .disable_auto_compact = 1, }; struct reftable_stack *st = NULL; char *dir = get_tmp_dir(__LINE__); @@ -508,7 +509,6 @@ static void test_reftable_stack_add(void) err = reftable_new_stack(&st, dir, cfg); EXPECT_ERR(err); - st->disable_auto_compact = 1; for (i = 0; i < N; i++) { char buf[256]; @@ -935,7 +935,9 @@ static void test_empty_add(void) static void test_reftable_stack_auto_compaction(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options cfg = { + .disable_auto_compact = 1, + }; struct reftable_stack *st = NULL; char *dir = get_tmp_dir(__LINE__); @@ -945,7 +947,6 @@ static void test_reftable_stack_auto_compaction(void) err = reftable_new_stack(&st, dir, cfg); EXPECT_ERR(err); - st->disable_auto_compact = 1; /* call manually below for coverage. */ for (i = 0; i < N; i++) { char name[100]; struct reftable_ref_record ref = { @@ -994,7 +995,7 @@ static void test_reftable_stack_add_performs_auto_compaction(void) * we can ensure that we indeed honor this setting and have * better control over when exactly auto compaction runs. */ - st->disable_auto_compact = i != n; + st->config.disable_auto_compact = i != n; strbuf_reset(&refname); strbuf_addf(&refname, "branch-%04d", i); From patchwork Mon Apr 8 16:16:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Justin Tobler X-Patchwork-Id: 13621356 Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64E6B14262C for ; Mon, 8 Apr 2024 16:17:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712593027; cv=none; b=aoqTGxQ+hklGfprJISs8ozUTPAjKd7Ro/keGbP0oX08JvAA42zA5GTtQox825wwDL1CA6R+skp3Fxmz2ePm2ZWJS3UOQFNNIs9XCgNwzVQSpRi3zKaWn42hWVyavNc5X4imEDDQwXoiccKVB95RUps+NARKZ9Xkxc/4pJgEg+tU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712593027; c=relaxed/simple; bh=vbNBvkdrGKCcNnJiGvMHmIEcN79GtkKwXJqd9PWudK4=; h=Message-Id:In-Reply-To:References:From:Date:Subject:Content-Type: MIME-Version:To:Cc; b=WWTA5QjVzGvJzTlOQDHD1godQLTwxm685JKvzdX7Nt1Jruo9mWSwiHfzzW2A9JBuW7kLt8FvdEv98MUo0rnK58UqwLqd+6SnaxLrWC3voEkRhbBhq3uvJ2ysc5vl7dJGP93iwlsSk54PtbRLgVG0+bcAdGE1CE6JvDzZAtC1W/o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UBQEHfXw; arc=none smtp.client-ip=209.85.221.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UBQEHfXw" Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-343d1003106so2782394f8f.1 for ; Mon, 08 Apr 2024 09:17:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712593019; x=1713197819; darn=vger.kernel.org; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=CbBSaV2toufpZjzIivSjI10z+Fp2kgqi4DhTCaAdHt0=; b=UBQEHfXwR9pb+6Qeoou13gzqV+n5hF64IxkUlN4KFBod08Zcb419ZEkBpDakTRNI4V ksSZkSoZki4nmjzLNrLaYcUsoX6umBNpTDMZpjF6F7uS7u/f3ZJXK9DrmFei7bOfovDY oQW0FoDwA642smbTY4a+F4BRPpGxTN4iiwDOaT8FjsKG8Dkh/aFZ++6swLVff/Ed05eU l9mA4K9TXeTYujf11isY3ry1NP9B5dssFF+Qp+5dqaIc4ETbn2ZmMcxKLrzcQt8jaUPs TNND1BxvDgwna20s2tlZuBw7E0TpGzmQV8R8G40Nf/4n7ncSYPqCGX0DIrQ7o4YMHra9 Jtpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712593019; x=1713197819; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CbBSaV2toufpZjzIivSjI10z+Fp2kgqi4DhTCaAdHt0=; b=Abl7HxyE7kwXSJmuCxouADeIp3luT+kmyuoOeM0a3OXreGRNq1UokHrSHMLk7YXWfv GZ4QsU6y2d2cFe4kMzvyd5eKe0LESbkATZvaHky/XM2DSEqhl2Nh8MDANe7kx9wyWBrU 9eXIWm/xH0DS+Ltd4sKwDcPzMwuIhyBCEJTY3GgPfv8divP0f4clOCEYNtTXfbrmYXrg jWGuAtrQOF7EP5VemG/mgI0EnzvzirciKyzn42unLERYKpP6MRJ+cSGdZMtbeEQh8mPe XTMFMf7YezuufpJx3Cr+RyKj8d+p9sAM2fuaYsPDESuUqjDa67Q5p1CUPqaW1g50wre4 U1dw== X-Gm-Message-State: AOJu0YwahhYCAxUfxy3KjPj+zwbVH9KtxhGKW9InGEvCRYVyrugG8uF4 L7msW9PqIb1psBZV0EouvuvWZKqliKn8W/pB25gUa5k7Xb+QrG5H1++pxGIn X-Google-Smtp-Source: AGHT+IEepHH5XkYp33v/lyl6EGqkm35/0tm05O+3fU212cwhSSIMuREtg+ICfQkkMGrjm0Q58Rfupg== X-Received: by 2002:a05:6000:e49:b0:343:41ef:ab30 with SMTP id dy9-20020a0560000e4900b0034341efab30mr6936207wrb.47.1712593019169; Mon, 08 Apr 2024 09:16:59 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j31-20020a5d6e5f000000b0033fc06f2d84sm9314881wrz.109.2024.04.08.09.16.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Apr 2024 09:16:58 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 08 Apr 2024 16:16:54 +0000 Subject: [PATCH v6 2/3] reftable/stack: add env to disable autocompaction Fcc: Sent Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 To: git@vger.kernel.org Cc: Patrick Steinhardt , Karthik Nayak , Han-Wen Nienhuys , Justin Tobler , Justin Tobler From: Justin Tobler From: Justin Tobler In future tests it will be neccesary to create repositories with a set number of tables. To make this easier, introduce the `GIT_TEST_REFTABLE_AUTOCOMPACTION` environment variable that, when set to false, disables autocompaction of reftables. Signed-off-by: Justin Tobler --- refs/reftable-backend.c | 3 +++ t/t0610-reftable-basics.sh | 21 +++++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 0bed6d2ab48..1cda48c5046 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -18,6 +18,7 @@ #include "../reftable/reftable-merged.h" #include "../setup.h" #include "../strmap.h" +#include "parse.h" #include "refs-internal.h" /* @@ -247,6 +248,8 @@ static struct ref_store *reftable_be_init(struct repository *repo, refs->write_options.block_size = 4096; refs->write_options.hash_id = repo->hash_algo->format_id; refs->write_options.default_permissions = calc_shared_perm(0666 & ~mask); + refs->write_options.disable_auto_compact = + !git_env_bool("GIT_TEST_REFTABLE_AUTOCOMPACTION", 1); /* * Set up the main reftable stack that is hosted in GIT_COMMON_DIR. diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index 931d888bbbc..c9e10b34684 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -299,6 +299,27 @@ test_expect_success 'ref transaction: writes cause auto-compaction' ' test_line_count = 1 repo/.git/reftable/tables.list ' +test_expect_success 'ref transaction: env var disables compaction' ' + test_when_finished "rm -rf repo" && + + git init repo && + test_commit -C repo A && + + start=$(wc -l X-Patchwork-Id: 13621358 Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 614A81422DB for ; Mon, 8 Apr 2024 16:17:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712593030; cv=none; b=PdlkLynAo0+QK+rAcZO8NVGfgr9yJErurkrew8qdbaTQPFx75NRT3kR87hmk2VpYBvXKU1gCY4gOMspFQWVyhsXg1EqtjK0Nho7VuGtM6nfINnW1FUKzpkHqHPH/V3yz/UE1mfFfXm/cBZp1RgxKi3b7mp7dPjTgrhLhBV1nvZc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712593030; c=relaxed/simple; bh=t4bu3VO3CdFdG3XgoR7nC0RSluZbspY/09hBfLI1Wkg=; h=Message-Id:In-Reply-To:References:From:Date:Subject:MIME-Version: Content-Type:To:Cc; b=Z56KSCwHe7W3H9/ha/nC/O8HdhKNThs+CdEdnaBoJOrDfLEJqCJ3Ljvk0hrozS2KsnNzV7eaXyPkFVv4XdwFugWalHC2ony6HnOGHD6rEGzWks43b0D+HrOkDEXB05xe1NHMbfChzQ2Q9UWxeWwL7G7zGRvZF20jGgy6VncqpX0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lIQXWUXA; arc=none smtp.client-ip=209.85.221.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lIQXWUXA" Received: by mail-wr1-f43.google.com with SMTP id ffacd0b85a97d-345e4097cb9so414039f8f.3 for ; Mon, 08 Apr 2024 09:17:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712593021; x=1713197821; darn=vger.kernel.org; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=5hbh+l1VXKX0a9XBC9dEfE3rTRjybVL9js5cSqQwDC4=; b=lIQXWUXA5iIf6ues1sV4jwtnh+iJqrVLkvRxqizZ3wCduxWtO8/3JE9kh6ImkbaLmT 2LRZ92QoFgQq7uDMyu0Iafa80MTC6sHM1sPGO/AAWwK1cub6uJllL2YhNhfdu1IDNYCb 6qlpmfBRywmqvv7gpUbhXLMjEEQiOoM9gvddKBQmlBNGsUMQ3pmVbbfxI/H+W82/qK19 6l9qV/j6B809DKPS4fL5colY867QRyGpZV2Yn9mr2fQm4njfb5NcQR9Rgnk8wLR9Yt5M g50XjWR4jz1QOyegPqFeoENzll57M1N9hB9o/p7nsDnOgWA1ECYuG7QzE7W3gv79OfV7 bKew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712593021; x=1713197821; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5hbh+l1VXKX0a9XBC9dEfE3rTRjybVL9js5cSqQwDC4=; b=elOKCBsAJL6Msu6XxrJkcWxRB6FTKuqCNjzM7dMPYzCDAzD5SQ7G6TsMv2y3TYndwr o1w2JFV6zKAPNgdalYH/3ujeXVFqrsYTXeaKhFrvBJxDXttnSCDZcD7rPjZYHMbQdDj5 /ul540LRLga400X7ArE6Chm7LSMUN1wjX1U3fHTWbJJXvVEC48nFkWRWh8voaCcFvyGX BcFOKQzDNdhI5sBSO0c0PLxKzn612HJFm1qkPP6DZLMYKV05/92BcC8IveGQSnJPgq1l Bv7PBLT+JDJ5w53h9QNzdFhFTjPguQSxYVsY4r58aFoAbqb63ZG5IwxC7J/L4Ozeprs0 Q9HQ== X-Gm-Message-State: AOJu0YxSy5IIE/DDYSAMhc4I3V2sDlSif0jXsYcVrnxi2oYif42dFSUa vhw9Hu1ufvgnbHnJ0VqD6C+Pz+6BA9UYRnB4DM1adE43OAb40J9PUkWrz0Zb X-Google-Smtp-Source: AGHT+IGArpjFSwmEs3BtzUbYXiTAeDvBUGcl3C+fsY9RCO/xFSNqImoqpfl0FIqhX7sDWjtCcqyVdw== X-Received: by 2002:a5d:5708:0:b0:341:bff9:2e4f with SMTP id a8-20020a5d5708000000b00341bff92e4fmr6463210wrv.44.1712593020548; Mon, 08 Apr 2024 09:17:00 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a17-20020a5d5091000000b00341ba91c1f5sm9325158wrt.102.2024.04.08.09.16.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Apr 2024 09:16:59 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 08 Apr 2024 16:16:55 +0000 Subject: [PATCH v6 3/3] reftable/stack: use geometric table compaction Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: Patrick Steinhardt , Karthik Nayak , Han-Wen Nienhuys , Justin Tobler , Justin Tobler From: Justin Tobler From: Justin Tobler To reduce the number of on-disk reftables, compaction is performed. Contiguous tables with the same binary log value of size are grouped into segments. The segment that has both the lowest binary log value and contains more than one table is set as the starting point when identifying the compaction segment. Since segments containing a single table are not initially considered for compaction, if the table appended to the list does not match the previous table log value, no compaction occurs for the new table. It is therefore possible for unbounded growth of the table list. This can be demonstrated by repeating the following sequence: git branch -f foo git branch -d foo Each operation results in a new table being written with no compaction occurring until a separate operation produces a table matching the previous table log value. Instead, to avoid unbounded growth of the table list, the compaction strategy is updated to ensure tables follow a geometric sequence after each operation by individually evaluating each table in reverse index order. This strategy results in a much simpler and more robust algorithm compared to the previous one while also maintaining a minimal ordered set of tables on-disk. When creating 10 thousand references, the new strategy has no performance impact: Benchmark 1: update-ref: create refs sequentially (revision = HEAD~) Time (mean ± σ): 26.516 s ± 0.047 s [User: 17.864 s, System: 8.491 s] Range (min … max): 26.447 s … 26.569 s 10 runs Benchmark 2: update-ref: create refs sequentially (revision = HEAD) Time (mean ± σ): 26.417 s ± 0.028 s [User: 17.738 s, System: 8.500 s] Range (min … max): 26.366 s … 26.444 s 10 runs Summary update-ref: create refs sequentially (revision = HEAD) ran 1.00 ± 0.00 times faster than update-ref: create refs sequentially (revision = HEAD~) Some tests in `t0610-reftable-basics.sh` assert the on-disk state of tables and are therefore updated to specify the correct new table count. Since compaction is more aggressive in ensuring tables maintain a geometric sequence, the expected table count is reduced in these tests. In `reftable/stack_test.c` tests related to `sizes_to_segments()` are removed because the function is no longer needed. Also, the `test_suggest_compaction_segment()` test is updated to better showcase and reflect the new geometric compaction behavior. Signed-off-by: Justin Tobler --- reftable/stack.c | 123 +++++++++++++++++++------------------ reftable/stack.h | 3 - reftable/stack_test.c | 66 ++++---------------- t/t0610-reftable-basics.sh | 50 ++++++++++----- 4 files changed, 111 insertions(+), 131 deletions(-) diff --git a/reftable/stack.c b/reftable/stack.c index 1a7cdad12c9..80266bcbab1 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -1216,75 +1216,76 @@ static int segment_size(struct segment *s) return s->end - s->start; } -int fastlog2(uint64_t sz) -{ - int l = 0; - if (sz == 0) - return 0; - for (; sz; sz /= 2) { - l++; - } - return l - 1; -} - -struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n) -{ - struct segment *segs = reftable_calloc(n, sizeof(*segs)); - struct segment cur = { 0 }; - size_t next = 0, i; - - if (n == 0) { - *seglen = 0; - return segs; - } - for (i = 0; i < n; i++) { - int log = fastlog2(sizes[i]); - if (cur.log != log && cur.bytes > 0) { - struct segment fresh = { - .start = i, - }; - - segs[next++] = cur; - cur = fresh; - } - - cur.log = log; - cur.end = i + 1; - cur.bytes += sizes[i]; - } - segs[next++] = cur; - *seglen = next; - return segs; -} - struct segment suggest_compaction_segment(uint64_t *sizes, size_t n) { - struct segment min_seg = { - .log = 64, - }; - struct segment *segs; - size_t seglen = 0, i; - - segs = sizes_to_segments(&seglen, sizes, n); - for (i = 0; i < seglen; i++) { - if (segment_size(&segs[i]) == 1) - continue; + struct segment seg = { 0 }; + uint64_t bytes; + size_t i; - if (segs[i].log < min_seg.log) - min_seg = segs[i]; - } + /* + * If there are no tables or only a single one then we don't have to + * compact anything. The sequence is geometric by definition already. + */ + if (n <= 1) + return seg; - while (min_seg.start > 0) { - size_t prev = min_seg.start - 1; - if (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) + /* + * Find the ending table of the compaction segment needed to restore the + * geometric sequence. Note that the segment end is exclusive. + * + * To do so, we iterate backwards starting from the most recent table + * until a valid segment end is found. If the preceding table is smaller + * than the current table multiplied by the geometric factor (2), the + * compaction segment end has been identified. + * + * Tables after the ending point are not added to the byte count because + * they are already valid members of the geometric sequence. Due to the + * properties of a geometric sequence, it is not possible for the sum of + * these tables to exceed the value of the ending point table. + * + * Example table size sequence requiring no compaction: + * 64, 32, 16, 8, 4, 2, 1 + * + * Example table size sequence where compaction segment end is set to + * the last table. Since the segment end is exclusive, the last table is + * excluded during subsequent compaction and the table with size 3 is + * the final table included: + * 64, 32, 16, 8, 4, 3, 1 + */ + for (i = n - 1; i > 0; i--) { + if (sizes[i - 1] < sizes[i] * 2) { + seg.end = i + 1; + bytes = sizes[i]; break; + } + } - min_seg.start = prev; - min_seg.bytes += sizes[prev]; + /* + * Find the starting table of the compaction segment by iterating + * through the remaining tables and keeping track of the accumulated + * size of all tables seen from the segment end table. The previous + * table is compared to the accumulated size because the tables from the + * segment end are merged backwards recursively. + * + * Note that we keep iterating even after we have found the first + * starting point. This is because there may be tables in the stack + * preceding that first starting point which violate the geometric + * sequence. + * + * Example compaction segment start set to table with size 32: + * 128, 32, 16, 8, 4, 3, 1 + */ + for (; i > 0; i--) { + uint64_t curr = bytes; + bytes += sizes[i - 1]; + + if (sizes[i - 1] < curr * 2) { + seg.start = i - 1; + seg.bytes = bytes; + } } - reftable_free(segs); - return min_seg; + return seg; } static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st) diff --git a/reftable/stack.h b/reftable/stack.h index c862053025f..d43efa47607 100644 --- a/reftable/stack.h +++ b/reftable/stack.h @@ -32,12 +32,9 @@ int read_lines(const char *filename, char ***lines); struct segment { size_t start, end; - int log; uint64_t bytes; }; -int fastlog2(uint64_t sz); -struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n); struct segment suggest_compaction_segment(uint64_t *sizes, size_t n); #endif diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 4fec823f14f..1df3ffce526 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -770,59 +770,13 @@ static void test_reftable_stack_hash_id(void) clear_dir(dir); } -static void test_log2(void) -{ - EXPECT(1 == fastlog2(3)); - EXPECT(2 == fastlog2(4)); - EXPECT(2 == fastlog2(5)); -} - -static void test_sizes_to_segments(void) -{ - uint64_t sizes[] = { 2, 3, 4, 5, 7, 9 }; - /* .................0 1 2 3 4 5 */ - - size_t seglen = 0; - struct segment *segs = - sizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes)); - EXPECT(segs[2].log == 3); - EXPECT(segs[2].start == 5); - EXPECT(segs[2].end == 6); - - EXPECT(segs[1].log == 2); - EXPECT(segs[1].start == 2); - EXPECT(segs[1].end == 5); - reftable_free(segs); -} - -static void test_sizes_to_segments_empty(void) -{ - size_t seglen = 0; - struct segment *segs = sizes_to_segments(&seglen, NULL, 0); - EXPECT(seglen == 0); - reftable_free(segs); -} - -static void test_sizes_to_segments_all_equal(void) -{ - uint64_t sizes[] = { 5, 5 }; - size_t seglen = 0; - struct segment *segs = - sizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes)); - EXPECT(seglen == 1); - EXPECT(segs[0].start == 0); - EXPECT(segs[0].end == 2); - reftable_free(segs); -} - static void test_suggest_compaction_segment(void) { - uint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 }; - /* .................0 1 2 3 4 5 6 */ + uint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 }; struct segment min = suggest_compaction_segment(sizes, ARRAY_SIZE(sizes)); - EXPECT(min.start == 2); - EXPECT(min.end == 7); + EXPECT(min.start == 1); + EXPECT(min.end == 10); } static void test_suggest_compaction_segment_nothing(void) @@ -933,6 +887,16 @@ static void test_empty_add(void) reftable_stack_destroy(st2); } +static int fastlog2(uint64_t sz) +{ + int l = 0; + if (sz == 0) + return 0; + for (; sz; sz /= 2) + l++; + return l - 1; +} + static void test_reftable_stack_auto_compaction(void) { struct reftable_write_options cfg = { @@ -1122,7 +1086,6 @@ static void test_reftable_stack_compaction_concurrent_clean(void) int stack_test_main(int argc, const char *argv[]) { RUN_TEST(test_empty_add); - RUN_TEST(test_log2); RUN_TEST(test_names_equal); RUN_TEST(test_parse_names); RUN_TEST(test_read_file); @@ -1143,9 +1106,6 @@ int stack_test_main(int argc, const char *argv[]) RUN_TEST(test_reftable_stack_update_index_check); RUN_TEST(test_reftable_stack_uptodate); RUN_TEST(test_reftable_stack_validate_refname); - RUN_TEST(test_sizes_to_segments); - RUN_TEST(test_sizes_to_segments_all_equal); - RUN_TEST(test_sizes_to_segments_empty); RUN_TEST(test_suggest_compaction_segment); RUN_TEST(test_suggest_compaction_segment_nothing); return 0; diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index c9e10b34684..8eec093788d 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -293,7 +293,7 @@ test_expect_success 'ref transaction: writes cause auto-compaction' ' test_line_count = 1 repo/.git/reftable/tables.list && test_commit -C repo --no-tag A && - test_line_count = 2 repo/.git/reftable/tables.list && + test_line_count = 1 repo/.git/reftable/tables.list && test_commit -C repo --no-tag B && test_line_count = 1 repo/.git/reftable/tables.list @@ -320,6 +320,19 @@ test_expect_success 'ref transaction: env var disables compaction' ' test_line_count -lt $expected repo/.git/reftable/tables.list ' +test_expect_success 'ref transaction: alternating table sizes are compacted' ' + test_when_finished "rm -rf repo" && + + git init repo && + test_commit -C repo A && + for i in $(test_seq 5) + do + git -C repo branch -f foo && + git -C repo branch -d foo || return 1 + done && + test_line_count = 2 repo/.git/reftable/tables.list +' + check_fsync_events () { local trace="$1" && shift && @@ -345,7 +358,7 @@ test_expect_success 'ref transaction: writes are synced' ' git -C repo -c core.fsync=reference \ -c core.fsyncMethod=fsync update-ref refs/heads/branch HEAD && check_fsync_events trace2.txt <<-EOF - "name":"hardware-flush","count":2 + "name":"hardware-flush","count":4 EOF ' @@ -377,7 +390,7 @@ test_expect_success 'ref transaction: fails gracefully when auto compaction fail done || exit 1 done && - test_line_count = 13 .git/reftable/tables.list + test_line_count = 10 .git/reftable/tables.list ) ' @@ -387,8 +400,8 @@ test_expect_success 'pack-refs: compacts tables' ' test_commit -C repo A && ls -1 repo/.git/reftable >table-files && - test_line_count = 4 table-files && - test_line_count = 3 repo/.git/reftable/tables.list && + test_line_count = 3 table-files && + test_line_count = 2 repo/.git/reftable/tables.list && git -C repo pack-refs && ls -1 repo/.git/reftable >table-files && @@ -429,7 +442,7 @@ test_expect_success "$command: auto compaction" ' # The tables should have been auto-compacted, and thus auto # compaction should not have to do anything. ls -1 .git/reftable >tables-expect && - test_line_count = 4 tables-expect && + test_line_count = 3 tables-expect && git $command --auto && ls -1 .git/reftable >tables-actual && test_cmp tables-expect tables-actual && @@ -447,7 +460,7 @@ test_expect_success "$command: auto compaction" ' git branch B && git branch C && rm .git/reftable/*.lock && - test_line_count = 5 .git/reftable/tables.list && + test_line_count = 4 .git/reftable/tables.list && git $command --auto && test_line_count = 1 .git/reftable/tables.list @@ -479,7 +492,7 @@ do umask $umask && git init --shared=true repo && test_commit -C repo A && - test_line_count = 3 repo/.git/reftable/tables.list + test_line_count = 2 repo/.git/reftable/tables.list ) && git -C repo pack-refs && test_expect_perms "-rw-rw-r--" repo/.git/reftable/tables.list && @@ -847,12 +860,16 @@ test_expect_success 'worktree: pack-refs in main repo packs main refs' ' test_when_finished "rm -rf repo worktree" && git init repo && test_commit -C repo A && + + GIT_TEST_REFTABLE_AUTOCOMPACTION=false \ git -C repo worktree add ../worktree && + GIT_TEST_REFTABLE_AUTOCOMPACTION=false \ + git -C worktree update-ref refs/worktree/per-worktree HEAD && - test_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list && - test_line_count = 4 repo/.git/reftable/tables.list && + test_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list && + test_line_count = 3 repo/.git/reftable/tables.list && git -C repo pack-refs && - test_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list && + test_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list && test_line_count = 1 repo/.git/reftable/tables.list ' @@ -860,13 +877,17 @@ test_expect_success 'worktree: pack-refs in worktree packs worktree refs' ' test_when_finished "rm -rf repo worktree" && git init repo && test_commit -C repo A && + + GIT_TEST_REFTABLE_AUTOCOMPACTION=false \ git -C repo worktree add ../worktree && + GIT_TEST_REFTABLE_AUTOCOMPACTION=false \ + git -C worktree update-ref refs/worktree/per-worktree HEAD && - test_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list && - test_line_count = 4 repo/.git/reftable/tables.list && + test_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list && + test_line_count = 3 repo/.git/reftable/tables.list && git -C worktree pack-refs && test_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list && - test_line_count = 4 repo/.git/reftable/tables.list + test_line_count = 3 repo/.git/reftable/tables.list ' test_expect_success 'worktree: creating shared ref updates main stack' ' @@ -880,6 +901,7 @@ test_expect_success 'worktree: creating shared ref updates main stack' ' test_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list && test_line_count = 1 repo/.git/reftable/tables.list && + GIT_TEST_REFTABLE_AUTOCOMPACTION=false \ git -C worktree update-ref refs/heads/shared HEAD && test_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list && test_line_count = 2 repo/.git/reftable/tables.list