From patchwork Fri May 10 10:29:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13661273 Received: from wfhigh1-smtp.messagingengine.com (wfhigh1-smtp.messagingengine.com [64.147.123.152]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24BE8168B01 for ; Fri, 10 May 2024 10:29:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.152 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336967; cv=none; b=s3bgSYudkZ3IFsfVvxA4mMHqFe+/gU0zYXVV5wfmP48Pi+VPel/RR5DVorVE41IVW8Z6C8GoGR51ax4vhhByOJhBhtzuiIdIHX/zCB0kuC0ygWBXsF3Ok+ykpKYthjiG4a4T1ZFcGGvRmcVxCZj0QFgf/nPMW62JpHD50abN3I0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336967; c=relaxed/simple; bh=4XKN8aafEheiMk7IJ0N51EU+SS+IcyoXCdEtIBcocec=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=fjUfUZ5zOuw1SgWC4ulYVFLKrtcbcjVhaPvgATKFa5txkaft/dfruQ14eikwPb5q1QAla+69aM3pH2SSEf6DfreYfpchK/YT8tJ5b0HQYgi4i0+CmomCZP78BpqAbO1qDToEfEJ3K6F/DIr5vWPKQS4RgdFdLIkj8Tg32NJGkg8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=cUQdfwZA; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Wzu9pnex; arc=none smtp.client-ip=64.147.123.152 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="cUQdfwZA"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Wzu9pnex" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.west.internal (Postfix) with ESMTP id 418911800080; Fri, 10 May 2024 06:29:24 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Fri, 10 May 2024 06:29:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715336963; x=1715423363; bh=Z9RfCj6Tfz SgUWUmUzb5BAsuT0PD4K4xHiQiKh3dYVI=; b=cUQdfwZAelTF5SKJUV8bXcsB0g /cmx2cb355TxiXKbCXIkU3ZyBis0EyAle7g4Vs13Or4QMGUQ1CA02tgfCWUCq8B2 ATz1JBWBVNcYGEEn/6r+OuCK2mDgLHd7vCvSx5VKQJuHhO61H0FEu9MSxe2ZXpvT IdnZjsA4xHh+2Tl7CYCTTqwPj/oa0dXk5RmULXig0/uI4lsnoJKyyMhT++rck1MW CPbc6G/0BmzZn5MIL5cXVx4HwH0bICWC00qUVI9u7MrLM1qmzhNBp7+6EA5eDzva HFfIfURacH2IqdN252bJDg09Wrdzvci8RgYq6YCCJQmw2jvLDIZBwqE8/9og== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715336963; x=1715423363; bh=Z9RfCj6TfzSgUWUmUzb5BAsuT0PD 4K4xHiQiKh3dYVI=; b=Wzu9pnexk1hFSosPAhanjn7qU6WrvrecNTAMCIerXQCk QgcYZdI7hYKOwfUiXao9zIR/1UFyvH6ThT7cmy3AENvWrWkK1BvEOQmvV64WhOsh siJqQdX8SZ08oPQCfR+XedvL30Har7oBzHjP6Teu20VrJiofRckuVh/E3+I7MmG1 w+v9YPjTdqJqxH3CWeGULL2h/KP20WoCUiwfQP8J3fL8SHBVz+XYdvBu9RwDCw8y 44LYRKIAy1syvVbIvx9tO0X1pGWkKYagsy/kOFAubJuBvErxZz2GmiknaHe4p+gT 67lWOibeGu2RJtnZeF22kxsYx/+2/o2yrmJw5vqCKg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefkedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 May 2024 06:29:22 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 6c905aba (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 10 May 2024 10:29:10 +0000 (UTC) Date: Fri, 10 May 2024 12:29:20 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler Subject: [PATCH v2 01/11] reftable: consistently refer to `reftable_write_options` as `opts` Message-ID: <7efa56630681b6aca64c04fd19fb020ffbb76152.1715336798.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Throughout the reftable library the `reftable_write_options` are sometimes referred to as `cfg` and sometimes as `opts`. Unify these to consistently use `opts` to avoid confusion. While at it, touch up the coding style a bit by removing unneeded braces around one-line statements and newlines between variable declarations. Signed-off-by: Patrick Steinhardt --- reftable/dump.c | 4 +- reftable/reftable-stack.h | 2 +- reftable/stack.c | 43 +++++++------- reftable/stack.h | 2 +- reftable/stack_test.c | 114 +++++++++++++++++--------------------- 5 files changed, 75 insertions(+), 90 deletions(-) diff --git a/reftable/dump.c b/reftable/dump.c index 26e0393c7d..9c770a10cc 100644 --- a/reftable/dump.c +++ b/reftable/dump.c @@ -27,9 +27,9 @@ license that can be found in the LICENSE file or at static int compact_stack(const char *stackdir) { struct reftable_stack *stack = NULL; - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; - int err = reftable_new_stack(&stack, stackdir, cfg); + int err = reftable_new_stack(&stack, stackdir, opts); if (err < 0) goto done; diff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h index 1b602dda58..9c8e4eef49 100644 --- a/reftable/reftable-stack.h +++ b/reftable/reftable-stack.h @@ -29,7 +29,7 @@ struct reftable_stack; * stored in 'dir'. Typically, this should be .git/reftables. */ int reftable_new_stack(struct reftable_stack **dest, const char *dir, - struct reftable_write_options config); + struct reftable_write_options opts); /* returns the update_index at which a next table should be written. */ uint64_t reftable_stack_next_update_index(struct reftable_stack *st); diff --git a/reftable/stack.c b/reftable/stack.c index 80266bcbab..3979657793 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -56,15 +56,14 @@ static int reftable_fd_flush(void *arg) } int reftable_new_stack(struct reftable_stack **dest, const char *dir, - struct reftable_write_options config) + struct reftable_write_options opts) { struct reftable_stack *p = reftable_calloc(1, sizeof(*p)); struct strbuf list_file_name = STRBUF_INIT; int err = 0; - if (config.hash_id == 0) { - config.hash_id = GIT_SHA1_FORMAT_ID; - } + if (opts.hash_id == 0) + opts.hash_id = GIT_SHA1_FORMAT_ID; *dest = NULL; @@ -75,7 +74,7 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir, p->list_file = strbuf_detach(&list_file_name, NULL); p->list_fd = -1; p->reftable_dir = xstrdup(dir); - p->config = config; + p->opts = opts; err = reftable_stack_reload_maybe_reuse(p, 1); if (err < 0) { @@ -257,7 +256,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st, char **names, /* success! */ err = reftable_new_merged_table(&new_merged, new_tables, - new_readers_len, st->config.hash_id); + new_readers_len, st->opts.hash_id); if (err < 0) goto done; @@ -580,8 +579,8 @@ static int reftable_stack_init_addition(struct reftable_addition *add, } goto done; } - if (st->config.default_permissions) { - if (chmod(add->lock_file->filename.buf, st->config.default_permissions) < 0) { + if (st->opts.default_permissions) { + if (chmod(add->lock_file->filename.buf, st->opts.default_permissions) < 0) { err = REFTABLE_IO_ERROR; goto done; } @@ -680,7 +679,7 @@ int reftable_addition_commit(struct reftable_addition *add) if (err) goto done; - if (!add->stack->config.disable_auto_compact) { + if (!add->stack->opts.disable_auto_compact) { /* * Auto-compact the stack to keep the number of tables in * control. It is possible that a concurrent writer is already @@ -758,9 +757,9 @@ int reftable_addition_add(struct reftable_addition *add, err = REFTABLE_IO_ERROR; goto done; } - if (add->stack->config.default_permissions) { + if (add->stack->opts.default_permissions) { if (chmod(get_tempfile_path(tab_file), - add->stack->config.default_permissions)) { + add->stack->opts.default_permissions)) { err = REFTABLE_IO_ERROR; goto done; } @@ -768,7 +767,7 @@ int reftable_addition_add(struct reftable_addition *add, tab_fd = get_tempfile_fd(tab_file); wr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd, - &add->stack->config); + &add->stack->opts); err = write_table(wr, arg); if (err < 0) goto done; @@ -855,14 +854,14 @@ static int stack_compact_locked(struct reftable_stack *st, } tab_fd = get_tempfile_fd(tab_file); - if (st->config.default_permissions && - chmod(get_tempfile_path(tab_file), st->config.default_permissions) < 0) { + if (st->opts.default_permissions && + chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) { err = REFTABLE_IO_ERROR; goto done; } wr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, - &tab_fd, &st->config); + &tab_fd, &st->opts); err = stack_write_compact(st, wr, first, last, config); if (err < 0) goto done; @@ -910,7 +909,7 @@ static int stack_write_compact(struct reftable_stack *st, st->readers[last]->max_update_index); err = reftable_new_merged_table(&mt, subtabs, subtabs_len, - st->config.hash_id); + st->opts.hash_id); if (err < 0) { reftable_free(subtabs); goto done; @@ -1100,9 +1099,9 @@ static int stack_compact_range(struct reftable_stack *st, goto done; } - if (st->config.default_permissions) { + if (st->opts.default_permissions) { if (chmod(get_lock_file_path(&tables_list_lock), - st->config.default_permissions) < 0) { + st->opts.default_permissions) < 0) { err = REFTABLE_IO_ERROR; goto done; } @@ -1292,7 +1291,7 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st) { uint64_t *sizes = reftable_calloc(st->merged->stack_len, sizeof(*sizes)); - int version = (st->config.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2; + int version = (st->opts.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2; int overhead = header_size(version) - 1; int i = 0; for (i = 0; i < st->merged->stack_len; i++) { @@ -1368,7 +1367,7 @@ static int stack_check_addition(struct reftable_stack *st, int len = 0; int i = 0; - if (st->config.skip_name_check) + if (st->opts.skip_name_check) return 0; err = reftable_block_source_from_file(&src, new_tab_name); @@ -1500,11 +1499,11 @@ int reftable_stack_clean(struct reftable_stack *st) int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id) { struct reftable_stack *stack = NULL; - struct reftable_write_options cfg = { .hash_id = hash_id }; + struct reftable_write_options opts = { .hash_id = hash_id }; struct reftable_merged_table *merged = NULL; struct reftable_table table = { NULL }; - int err = reftable_new_stack(&stack, stackdir, cfg); + int err = reftable_new_stack(&stack, stackdir, opts); if (err < 0) goto done; diff --git a/reftable/stack.h b/reftable/stack.h index d43efa4760..97d7ebc043 100644 --- a/reftable/stack.h +++ b/reftable/stack.h @@ -20,7 +20,7 @@ struct reftable_stack { char *reftable_dir; - struct reftable_write_options config; + struct reftable_write_options opts; struct reftable_reader **readers; size_t readers_len; diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 1df3ffce52..3316d55f19 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -150,7 +150,7 @@ static void test_reftable_stack_add_one(void) char *dir = get_tmp_dir(__LINE__); struct strbuf scratch = STRBUF_INIT; int mask = umask(002); - struct reftable_write_options cfg = { + struct reftable_write_options opts = { .default_permissions = 0660, }; struct reftable_stack *st = NULL; @@ -163,7 +163,7 @@ static void test_reftable_stack_add_one(void) }; struct reftable_ref_record dest = { NULL }; struct stat stat_result = { 0 }; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_test_ref, &ref); @@ -186,7 +186,7 @@ static void test_reftable_stack_add_one(void) strbuf_addstr(&scratch, "/tables.list"); err = stat(scratch.buf, &stat_result); EXPECT(!err); - EXPECT((stat_result.st_mode & 0777) == cfg.default_permissions); + EXPECT((stat_result.st_mode & 0777) == opts.default_permissions); strbuf_reset(&scratch); strbuf_addstr(&scratch, dir); @@ -195,7 +195,7 @@ static void test_reftable_stack_add_one(void) strbuf_addstr(&scratch, st->readers[0]->name); err = stat(scratch.buf, &stat_result); EXPECT(!err); - EXPECT((stat_result.st_mode & 0777) == cfg.default_permissions); + EXPECT((stat_result.st_mode & 0777) == opts.default_permissions); #else (void) stat_result; #endif @@ -209,7 +209,7 @@ static void test_reftable_stack_add_one(void) static void test_reftable_stack_uptodate(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st1 = NULL; struct reftable_stack *st2 = NULL; char *dir = get_tmp_dir(__LINE__); @@ -232,10 +232,10 @@ static void test_reftable_stack_uptodate(void) /* simulate multi-process access to the same stack by creating two stacks for the same directory. */ - err = reftable_new_stack(&st1, dir, cfg); + err = reftable_new_stack(&st1, dir, opts); EXPECT_ERR(err); - err = reftable_new_stack(&st2, dir, cfg); + err = reftable_new_stack(&st2, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st1, &write_test_ref, &ref1); @@ -257,8 +257,7 @@ static void test_reftable_stack_uptodate(void) static void test_reftable_stack_transaction_api(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; struct reftable_addition *add = NULL; @@ -271,8 +270,7 @@ static void test_reftable_stack_transaction_api(void) }; struct reftable_ref_record dest = { NULL }; - - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); reftable_addition_destroy(add); @@ -301,12 +299,12 @@ static void test_reftable_stack_transaction_api(void) static void test_reftable_stack_transaction_api_performs_auto_compaction(void) { char *dir = get_tmp_dir(__LINE__); - struct reftable_write_options cfg = {0}; + struct reftable_write_options opts = {0}; struct reftable_addition *add = NULL; struct reftable_stack *st = NULL; int i, n = 20, err; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 0; i <= n; i++) { @@ -325,7 +323,7 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void) * we can ensure that we indeed honor this setting and have * better control over when exactly auto compaction runs. */ - st->config.disable_auto_compact = i != n; + st->opts.disable_auto_compact = i != n; err = reftable_stack_new_addition(&add, st); EXPECT_ERR(err); @@ -361,13 +359,13 @@ static void test_reftable_stack_auto_compaction_fails_gracefully(void) .value_type = REFTABLE_REF_VAL1, .value.val1 = {0x01}, }; - struct reftable_write_options cfg = {0}; + struct reftable_write_options opts = {0}; struct reftable_stack *st; struct strbuf table_path = STRBUF_INIT; char *dir = get_tmp_dir(__LINE__); int err; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, write_test_ref, &ref); @@ -442,8 +440,7 @@ static int write_error(struct reftable_writer *wr, void *arg) static void test_reftable_stack_update_index_check(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; struct reftable_ref_record ref1 = { @@ -459,7 +456,7 @@ static void test_reftable_stack_update_index_check(void) .value.symref = "master", }; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_test_ref, &ref1); @@ -474,12 +471,11 @@ static void test_reftable_stack_update_index_check(void) static void test_reftable_stack_lock_failure(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err, i; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) { err = reftable_stack_add(st, &write_error, &i); @@ -494,7 +490,7 @@ static void test_reftable_stack_add(void) { int i = 0; int err = 0; - struct reftable_write_options cfg = { + struct reftable_write_options opts = { .exact_log_message = 1, .default_permissions = 0660, .disable_auto_compact = 1, @@ -507,7 +503,7 @@ static void test_reftable_stack_add(void) struct stat stat_result; int N = ARRAY_SIZE(refs); - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -566,7 +562,7 @@ static void test_reftable_stack_add(void) strbuf_addstr(&path, "/tables.list"); err = stat(path.buf, &stat_result); EXPECT(!err); - EXPECT((stat_result.st_mode & 0777) == cfg.default_permissions); + EXPECT((stat_result.st_mode & 0777) == opts.default_permissions); strbuf_reset(&path); strbuf_addstr(&path, dir); @@ -575,7 +571,7 @@ static void test_reftable_stack_add(void) strbuf_addstr(&path, st->readers[0]->name); err = stat(path.buf, &stat_result); EXPECT(!err); - EXPECT((stat_result.st_mode & 0777) == cfg.default_permissions); + EXPECT((stat_result.st_mode & 0777) == opts.default_permissions); #else (void) stat_result; #endif @@ -593,7 +589,7 @@ static void test_reftable_stack_add(void) static void test_reftable_stack_log_normalize(void) { int err = 0; - struct reftable_write_options cfg = { + struct reftable_write_options opts = { 0, }; struct reftable_stack *st = NULL; @@ -617,7 +613,7 @@ static void test_reftable_stack_log_normalize(void) .update_index = 1, }; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); input.value.update.message = "one\ntwo"; @@ -650,8 +646,7 @@ static void test_reftable_stack_tombstone(void) { int i = 0; char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; struct reftable_ref_record refs[2] = { { NULL } }; @@ -660,8 +655,7 @@ static void test_reftable_stack_tombstone(void) struct reftable_ref_record dest = { NULL }; struct reftable_log_record log_dest = { NULL }; - - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); /* even entries add the refs, odd entries delete them. */ @@ -729,8 +723,7 @@ static void test_reftable_stack_tombstone(void) static void test_reftable_stack_hash_id(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; @@ -740,24 +733,24 @@ static void test_reftable_stack_hash_id(void) .value.symref = "target", .update_index = 1, }; - struct reftable_write_options cfg32 = { .hash_id = GIT_SHA256_FORMAT_ID }; + struct reftable_write_options opts32 = { .hash_id = GIT_SHA256_FORMAT_ID }; struct reftable_stack *st32 = NULL; - struct reftable_write_options cfg_default = { 0 }; + struct reftable_write_options opts_default = { 0 }; struct reftable_stack *st_default = NULL; struct reftable_ref_record dest = { NULL }; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_test_ref, &ref); EXPECT_ERR(err); /* can't read it with the wrong hash ID. */ - err = reftable_new_stack(&st32, dir, cfg32); + err = reftable_new_stack(&st32, dir, opts32); EXPECT(err == REFTABLE_FORMAT_ERROR); - /* check that we can read it back with default config too. */ - err = reftable_new_stack(&st_default, dir, cfg_default); + /* check that we can read it back with default opts too. */ + err = reftable_new_stack(&st_default, dir, opts_default); EXPECT_ERR(err); err = reftable_stack_read_ref(st_default, "master", &dest); @@ -790,8 +783,7 @@ static void test_suggest_compaction_segment_nothing(void) static void test_reflog_expire(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; struct reftable_log_record logs[20] = { { NULL } }; int N = ARRAY_SIZE(logs) - 1; @@ -802,8 +794,7 @@ static void test_reflog_expire(void) }; struct reftable_log_record log = { NULL }; - - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 1; i <= N; i++) { @@ -866,21 +857,19 @@ static int write_nothing(struct reftable_writer *wr, void *arg) static void test_empty_add(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; char *dir = get_tmp_dir(__LINE__); - struct reftable_stack *st2 = NULL; - - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_nothing, NULL); EXPECT_ERR(err); - err = reftable_new_stack(&st2, dir, cfg); + err = reftable_new_stack(&st2, dir, opts); EXPECT_ERR(err); clear_dir(dir); reftable_stack_destroy(st); @@ -899,16 +888,15 @@ static int fastlog2(uint64_t sz) static void test_reftable_stack_auto_compaction(void) { - struct reftable_write_options cfg = { + struct reftable_write_options opts = { .disable_auto_compact = 1, }; struct reftable_stack *st = NULL; char *dir = get_tmp_dir(__LINE__); - int err, i; int N = 100; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -938,13 +926,13 @@ static void test_reftable_stack_auto_compaction(void) static void test_reftable_stack_add_performs_auto_compaction(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; struct strbuf refname = STRBUF_INIT; char *dir = get_tmp_dir(__LINE__); int err, i, n = 20; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 0; i <= n; i++) { @@ -959,7 +947,7 @@ static void test_reftable_stack_add_performs_auto_compaction(void) * we can ensure that we indeed honor this setting and have * better control over when exactly auto compaction runs. */ - st->config.disable_auto_compact = i != n; + st->opts.disable_auto_compact = i != n; strbuf_reset(&refname); strbuf_addf(&refname, "branch-%04d", i); @@ -986,14 +974,13 @@ static void test_reftable_stack_add_performs_auto_compaction(void) static void test_reftable_stack_compaction_concurrent(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st1 = NULL, *st2 = NULL; char *dir = get_tmp_dir(__LINE__); - int err, i; int N = 3; - err = reftable_new_stack(&st1, dir, cfg); + err = reftable_new_stack(&st1, dir, opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -1010,7 +997,7 @@ static void test_reftable_stack_compaction_concurrent(void) EXPECT_ERR(err); } - err = reftable_new_stack(&st2, dir, cfg); + err = reftable_new_stack(&st2, dir, opts); EXPECT_ERR(err); err = reftable_stack_compact_all(st1, NULL); @@ -1036,14 +1023,13 @@ static void unclean_stack_close(struct reftable_stack *st) static void test_reftable_stack_compaction_concurrent_clean(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL; char *dir = get_tmp_dir(__LINE__); - int err, i; int N = 3; - err = reftable_new_stack(&st1, dir, cfg); + err = reftable_new_stack(&st1, dir, opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -1060,7 +1046,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void) EXPECT_ERR(err); } - err = reftable_new_stack(&st2, dir, cfg); + err = reftable_new_stack(&st2, dir, opts); EXPECT_ERR(err); err = reftable_stack_compact_all(st1, NULL); @@ -1069,7 +1055,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void) unclean_stack_close(st1); unclean_stack_close(st2); - err = reftable_new_stack(&st3, dir, cfg); + err = reftable_new_stack(&st3, dir, opts); EXPECT_ERR(err); err = reftable_stack_clean(st3); From patchwork Fri May 10 10:29:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13661274 Received: from wfhigh1-smtp.messagingengine.com (wfhigh1-smtp.messagingengine.com [64.147.123.152]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 03A64168B01 for ; Fri, 10 May 2024 10:29:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.152 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336971; cv=none; b=AfFCEGEVGU2jZ6v/214KucpSiXcsukF7jwS3oIvsiOX35WLI2pN6dqbJGP0Jrmp/1ihzdKV7gKH16rD8SRFGLDr0zf7wdm1FYmrLbtpYEElCKyTxz3MGEaRwsG0e6MqQaSfzBLJjQjOIkGcX3THUbccjzBTj6zR1Bw40IUYkHa8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336971; c=relaxed/simple; bh=gu/fthYt8H/q7rhSnojtHqzpVupq2uxZt+5jzmWFh7c=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=HCBBL/VehboUbMHGtTnusLZtkmAjs2jCG7eJ5aBAEkvyDh3jMOCmyVZnIGxknJ5de46waPI58OXVRucVV+YhnvWGiS9n6WxLEnN94B5ofSmrJUJb7rIZ92pTgJEyzyDGfow6RVwHU9/fO+f56qUa9guhymdNhsT128KZSC5y77M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=CQEKNUsl; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=e0VKwEuh; arc=none smtp.client-ip=64.147.123.152 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="CQEKNUsl"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="e0VKwEuh" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfhigh.west.internal (Postfix) with ESMTP id 1AA31180009A; Fri, 10 May 2024 06:29:29 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 10 May 2024 06:29:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715336968; x=1715423368; bh=3d0FFhFQHf INkXefT8jArDIFCWHcBC6W3EEMM8bSXlA=; b=CQEKNUslGxhrNNxUmU7/SR6bQb Y8brxzBFCmedZpQioPBzECxYV/nqusrn79AX4GpSPzC1FVhKvO6pjLirc4GNfNxf IEkUWPe0dS2LWv/IlgQo5lzXn+9eO3tfRV6hoEJqfNtiKydQi6DA68Tr46lIMfK/ 1CGhM+Dp7fNho+edmUyNi7JRhWgypJvESpO+OGSbdGPqaq94uxywH/X44DZBxhwG keW3DU/SIqnxQL+18Bz8rtBejaqBJcr0zEtmkMferd8cIQEsTvg6hHceKHj4cvku O8YfX2ZylgjwZP7WTVkPs11kIeklzGlvYiyU7Ttufbl2wvHVQfPYGVUhtalQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715336968; x=1715423368; bh=3d0FFhFQHfINkXefT8jArDIFCWHc BC6W3EEMM8bSXlA=; b=e0VKwEuhfV7HD+DbegAaTO4z3Pz9yzprQXK0JNtwQYaW zmsfDjzJsc/5sb8LQNKSVB1jBSmCkV/bckLreXeVI9ZkVQj3usWPHm8DrklAQGZR w9xopBw/7Y5D6mzNMeVb6CK4y8wdQx/RJSQpKD+uXzVvWR7lu643aBaid5RT+Lci LRGVaMmUOo/Fz4tbBOIfIfsUkwOktFTgCmLa2YRPcAnLSZICiOXJJzBbZ/UlByhB FRFzYJsjPmWJMXs7Xchpe2yl2oofowidP4IRX1yn8f8ugChPCpgTGTWoymRrt7nu FNOOaRtgwioW8WzqQNt2NKgaIMTF2xRnKOWQsEnsmQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefkedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 May 2024 06:29:27 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 859208f4 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 10 May 2024 10:29:15 +0000 (UTC) Date: Fri, 10 May 2024 12:29:25 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler Subject: [PATCH v2 02/11] reftable: consistently pass write opts as value Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We sometimes pass the refatble write options as value and sometimes as a pointer. This is quite confusing and makes the reader wonder whether the options get modified sometimes. In fact, `reftable_new_writer()` does cause the caller-provided options to get updated when some values aren't set up. This is quite unexpected, but didn't cause any harm until now. Refactor the code to consistently pass the options as a value so that they are local to the subsystem they are being passed into so that we can avoid weirdness like this. Signed-off-by: Patrick Steinhardt --- reftable/merged_test.c | 6 +++--- reftable/readwrite_test.c | 26 +++++++++++++------------- reftable/refname_test.c | 2 +- reftable/reftable-writer.h | 2 +- reftable/stack.c | 4 ++-- reftable/writer.c | 12 +++++++----- 6 files changed, 27 insertions(+), 25 deletions(-) diff --git a/reftable/merged_test.c b/reftable/merged_test.c index 530fc82d1c..4ac81de8d4 100644 --- a/reftable/merged_test.c +++ b/reftable/merged_test.c @@ -42,7 +42,7 @@ static void write_test_table(struct strbuf *buf, } } - w = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts); + w = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts); reftable_writer_set_limits(w, min, max); for (i = 0; i < n; i++) { @@ -70,7 +70,7 @@ static void write_test_log_table(struct strbuf *buf, .exact_log_message = 1, }; struct reftable_writer *w = NULL; - w = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts); + w = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts); reftable_writer_set_limits(w, update_index, update_index); for (i = 0; i < n; i++) { @@ -403,7 +403,7 @@ static void test_default_write_opts(void) struct reftable_write_options opts = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_ref_record rec = { .refname = "master", diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c index a6dbd214c5..27631a041b 100644 --- a/reftable/readwrite_test.c +++ b/reftable/readwrite_test.c @@ -51,7 +51,7 @@ static void write_table(char ***names, struct strbuf *buf, int N, .hash_id = hash_id, }; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts); struct reftable_ref_record ref = { NULL }; int i = 0, n; struct reftable_log_record log = { NULL }; @@ -129,7 +129,7 @@ static void test_log_buffer_size(void) .message = "commit: 9\n", } } }; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); /* This tests buffer extension for log compression. Must use a random hash, to ensure that the compressed part is larger than the original. @@ -172,7 +172,7 @@ static void test_log_overflow(void) }, }; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); memset(msg, 'x', sizeof(msg) - 1); reftable_writer_set_limits(w, update_index, update_index); @@ -199,7 +199,7 @@ static void test_log_write_read(void) struct reftable_block_source source = { NULL }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); const struct reftable_stats *stats = NULL; reftable_writer_set_limits(w, 0, N); for (i = 0; i < N; i++) { @@ -288,7 +288,7 @@ static void test_log_zlib_corruption(void) struct reftable_block_source source = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); const struct reftable_stats *stats = NULL; char message[100] = { 0 }; int err, i, n; @@ -526,7 +526,7 @@ static void test_table_refs_for(int indexed) struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_iterator it = { NULL }; int j; @@ -619,7 +619,7 @@ static void test_write_empty_table(void) struct reftable_write_options opts = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_block_source source = { NULL }; struct reftable_reader *rd = NULL; struct reftable_ref_record rec = { NULL }; @@ -657,7 +657,7 @@ static void test_write_object_id_min_length(void) }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_ref_record ref = { .update_index = 1, .value_type = REFTABLE_REF_VAL1, @@ -692,7 +692,7 @@ static void test_write_object_id_length(void) }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_ref_record ref = { .update_index = 1, .value_type = REFTABLE_REF_VAL1, @@ -726,7 +726,7 @@ static void test_write_empty_key(void) struct reftable_write_options opts = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_ref_record ref = { .refname = "", .update_index = 1, @@ -749,7 +749,7 @@ static void test_write_key_order(void) struct reftable_write_options opts = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_ref_record refs[2] = { { .refname = "b", @@ -792,7 +792,7 @@ static void test_write_multiple_indices(void) struct reftable_reader *reader; int err, i; - writer = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, &opts); + writer = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, opts); reftable_writer_set_limits(writer, 1, 1); for (i = 0; i < 100; i++) { struct reftable_ref_record ref = { @@ -869,7 +869,7 @@ static void test_write_multi_level_index(void) struct reftable_reader *reader; int err; - writer = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, &opts); + writer = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, opts); reftable_writer_set_limits(writer, 1, 1); for (size_t i = 0; i < 200; i++) { struct reftable_ref_record ref = { diff --git a/reftable/refname_test.c b/reftable/refname_test.c index b9cc62554e..3468253be7 100644 --- a/reftable/refname_test.c +++ b/reftable/refname_test.c @@ -30,7 +30,7 @@ static void test_conflict(void) struct reftable_write_options opts = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_ref_record rec = { .refname = "a/b", .value_type = REFTABLE_REF_SYMREF, diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index 155bf0bbe2..44cb986465 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -92,7 +92,7 @@ struct reftable_stats { struct reftable_writer * reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), int (*flush_func)(void *), - void *writer_arg, struct reftable_write_options *opts); + void *writer_arg, struct reftable_write_options opts); /* Set the range of update indices for the records we will add. When writing a table into a stack, the min should be at least diff --git a/reftable/stack.c b/reftable/stack.c index 3979657793..7b4fff7c9e 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -767,7 +767,7 @@ int reftable_addition_add(struct reftable_addition *add, tab_fd = get_tempfile_fd(tab_file); wr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd, - &add->stack->opts); + add->stack->opts); err = write_table(wr, arg); if (err < 0) goto done; @@ -861,7 +861,7 @@ static int stack_compact_locked(struct reftable_stack *st, } wr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, - &tab_fd, &st->opts); + &tab_fd, st->opts); err = stack_write_compact(st, wr, first, last, config); if (err < 0) goto done; diff --git a/reftable/writer.c b/reftable/writer.c index 1d9ff0fbfa..ad2f2e6c65 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -122,20 +122,22 @@ static struct strbuf reftable_empty_strbuf = STRBUF_INIT; struct reftable_writer * reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), int (*flush_func)(void *), - void *writer_arg, struct reftable_write_options *opts) + void *writer_arg, struct reftable_write_options opts) { struct reftable_writer *wp = reftable_calloc(1, sizeof(*wp)); strbuf_init(&wp->block_writer_data.last_key, 0); - options_set_defaults(opts); - if (opts->block_size >= (1 << 24)) { + + options_set_defaults(&opts); + if (opts.block_size >= (1 << 24)) { /* TODO - error return? */ abort(); } + wp->last_key = reftable_empty_strbuf; - REFTABLE_CALLOC_ARRAY(wp->block, opts->block_size); + REFTABLE_CALLOC_ARRAY(wp->block, opts.block_size); wp->write = writer_func; wp->write_arg = writer_arg; - wp->opts = *opts; + wp->opts = opts; wp->flush = flush_func; writer_reinit_block_writer(wp, BLOCK_TYPE_REF); From patchwork Fri May 10 10:29:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13661275 Received: from wfout7-smtp.messagingengine.com (wfout7-smtp.messagingengine.com [64.147.123.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02192168B01 for ; Fri, 10 May 2024 10:29:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336976; cv=none; b=sq7B0h7OwmQJ9FHvnQCfh+f26MliJ0NGnkIDH+sXBn5UGv0aiSoq3EJiaMm3Ykc/fQpkxgp8fWlE269Xs2RB1suI1iVRJOdo8sPYfb5QNNrNSH8/tCbXRvK+PNcukVwc/fiiYXKrKSihLHtQIR/K7LjNTvWCRxjM9iRUWe1vNEw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336976; c=relaxed/simple; bh=1KmcghFF7+83vwY4DcYxDOdcg+gwlwUFWFy/dI0jsds=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=WenhDgm3ekxcR68v4RCSGUcndKlsHYwkoLUIYsvVdYWdWVEadVDejLLwoURleg/PZeK0HFaHNSP1kZaGB7SGA4/zjjWRbMt24ZOpV4CdvIn4jw3lq5/T/64XokTvlpGIIABIqXJCY9mXLB9INogGXvSTIMcJvEf0WUpdEjDN708= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=p93SvoMS; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=hqmWsqoy; arc=none smtp.client-ip=64.147.123.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="p93SvoMS"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="hqmWsqoy" Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfout.west.internal (Postfix) with ESMTP id 124231C000E5; Fri, 10 May 2024 06:29:34 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Fri, 10 May 2024 06:29:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715336973; x=1715423373; bh=Tj/KG2yxNu E/1QAgHFbx55l6WQ//XkhmH3M5iWee9b0=; b=p93SvoMS0QE/jgfwdk8AYZ1GWX GEYXo4PQXFV9kGAyySZKnVJml7zRPPC3VHJTQ4pXEFqCNF540salbeTwF4SIi8o0 7Gxs2cQUz18i1AadGE2+brX85xFfpOWvzYha613Ace9nxkMwTnI88AMJI7Cc5dIK TMelUPRFM5A8l+KB94TuGXL99kqESnWK95QvY1DQFSyJFQdaw32GJybAh8O4iVrz kHUSgBvQaLCgVkbgYkHnI27qnMy+vyyg0fGNhmENTM562J8V+SbCH16C8X4rj4dp tTNWHFS7oJNUhLmcPBLrnDXTSrEE4msxSPU+gQxxCQV6pul7SErGZnRYYAkg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715336973; x=1715423373; bh=Tj/KG2yxNuE/1QAgHFbx55l6WQ// XkhmH3M5iWee9b0=; b=hqmWsqoyhoovCCqvK1715pbRWVlrDTd/DWQokIRvCiSw lM5rUwOI2jLWtOEiX/Tr8jWtYPryn7dMAnePSYSyQZ8JAckc7q0LPVuHu1KhKles 6W27tIoXdFXU+Q8UDcQamV5iW3zGvWzm+s/BmMMo4Lj9s7HB/Aiy1DotD4tRa6vp hEGX4g+kpuO9scWWqXlOHTx4WFfwftvPz6iwz9VoYw/Ex4DGoWyENhSpp7uCzRH1 YrsciVKRtKu9Guak8/b1ZCyYuX32DdlY4CsM17I0AqX8m7nzOdGyaTAhM8kv+bXB thDf7/LPUp7S7IcsTLkNcYisc4wd9AIJxKgylvTNbQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefkedgvdelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 May 2024 06:29:32 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 07043d59 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 10 May 2024 10:29:19 +0000 (UTC) Date: Fri, 10 May 2024 12:29:30 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler Subject: [PATCH v2 03/11] reftable/writer: drop static variable used to initialize strbuf Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We have a static variable in the reftable writer code that is merely used to initialize the `last_key` of the writer. Convert the code to instead use `strbuf_init()` and drop the variable. Signed-off-by: Patrick Steinhardt --- reftable/writer.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/reftable/writer.c b/reftable/writer.c index ad2f2e6c65..7df6e53699 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -117,15 +117,12 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ) w->block_writer->restart_interval = w->opts.restart_interval; } -static struct strbuf reftable_empty_strbuf = STRBUF_INIT; - struct reftable_writer * reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), int (*flush_func)(void *), void *writer_arg, struct reftable_write_options opts) { struct reftable_writer *wp = reftable_calloc(1, sizeof(*wp)); - strbuf_init(&wp->block_writer_data.last_key, 0); options_set_defaults(&opts); if (opts.block_size >= (1 << 24)) { @@ -133,7 +130,8 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), abort(); } - wp->last_key = reftable_empty_strbuf; + strbuf_init(&wp->block_writer_data.last_key, 0); + strbuf_init(&wp->last_key, 0); REFTABLE_CALLOC_ARRAY(wp->block, opts.block_size); wp->write = writer_func; wp->write_arg = writer_arg; From patchwork Fri May 10 10:29:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13661276 Received: from wfhigh1-smtp.messagingengine.com (wfhigh1-smtp.messagingengine.com [64.147.123.152]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D932F16D4D6 for ; Fri, 10 May 2024 10:29:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.152 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336981; cv=none; b=Cm9Tb/Q/UrQWc2RlSVmz9AmCckSlevtmOs4gOfV1HmQGp3mzac0SupX3OGUwUuYYKzFDoo9RZwvkcCtjnVKxYw/W7jKWmKKPlFwo1YTQjdNBuv4jffRd828FPsYdtjuchnT7DOxnLLE7XV+o2wzfgRfZ5i5WZZzXIoLiGwCJFi4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336981; c=relaxed/simple; bh=bTdvNkhNt1Dd7R0p9HaQQbZdFDtQm/dZCQoz4i4yAsw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=uSYeNMgmVDwfvJf7zlStlm8XaoGkB0UQ+c7rQ3aGPWp3+q391sMLJwquuphymMXX0/TsEEVc9odqjRCPHJ1OTV7r8yjeUZnOzezEa37OXUGvL5+VnNI2iyVxRCzdRS0T2i/TxfQY7XbBmdr5x/79amaUHHwcYV0S3Mpu8CX1T3s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=vBVIFC2m; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Xul7QzvC; arc=none smtp.client-ip=64.147.123.152 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="vBVIFC2m"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Xul7QzvC" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfhigh.west.internal (Postfix) with ESMTP id EDFFB180008D; Fri, 10 May 2024 06:29:38 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 10 May 2024 06:29:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715336978; x=1715423378; bh=f4lcxOJ8s7 yfqr9ikE6inqnuBR2tNzMVz4kWzHwA7Z8=; b=vBVIFC2miCbv0QVYfpiF2tbJV0 mY3zhNUl8ScYn9hOpaGYrZU/LwolS2DJDuX/MU3CfT3zeh2XIuW5u84tgUC9sWzl iAkf4F0LQdNKe4w3gPt4JiFG1BHGvsGyJZAofsB0ZXQi7XUW13O0/4RSpYqZEz1n YsDSeCwCnElUthkCMcywSQc12q+P7XnqUkMVm6Fp/b9b7M8LeAz+1w4e6Sss9btY ojTcsj7BhMqv/lwthAQVaVAWju/X2sj/C3gdzgQB57W5w06Z3XsPyk+1ADiRWt6c o+9Rcfm5PsInOpR9JRTXkNX2tU7NWze/Brqh67LgTcUqd/SJfa5SIUfLV0qg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715336978; x=1715423378; bh=f4lcxOJ8s7yfqr9ikE6inqnuBR2t NzMVz4kWzHwA7Z8=; b=Xul7QzvC19xPfpJsNJq51u6YAxH5X/3/aeGIY2qNkj3+ n1HBkZsKZVMhQXXM5oOd8om8UbE7pK5IVG3/b5Bb8yCtMRbCxyQfGXOxJkPrBENe nIDhVco+wEa1PTtreS4VrQZRWhNkozf9DltCDaRmIXdfY9pceFnlB4FWTYb6UKPI 3CPL6iYxm+HMXncbKOWLnvouKP1P8BNseIU/ijpkc84HrCuOpY9CW/3dvi0fvNHR wxD1aJAQz+j3NDckF37Wg/mXy1ei6czHsBsgy2giXKGvJLDhOg9C9zlludVkLvpV thnchfoutrAAdHHM5l56gRlWZWb8KWDwtYKCCjJbIQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefkedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 May 2024 06:29:37 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 652b5fa0 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 10 May 2024 10:29:25 +0000 (UTC) Date: Fri, 10 May 2024 12:29:35 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler Subject: [PATCH v2 04/11] reftable/writer: improve error when passed an invalid block size Message-ID: <5e7cbb7b193c578f7c946a5077a79421b0ac57f2.1715336798.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The reftable format only supports block sizes up to 16MB. When the writer is being passed a value bigger than that it simply calls abort(3P), which isn't all that helpful due to the lack of a proper error message. Improve this by calling `BUG()` instead. Signed-off-by: Patrick Steinhardt --- reftable/writer.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/reftable/writer.c b/reftable/writer.c index 7df6e53699..374b7d15ed 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -125,10 +125,8 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), struct reftable_writer *wp = reftable_calloc(1, sizeof(*wp)); options_set_defaults(&opts); - if (opts.block_size >= (1 << 24)) { - /* TODO - error return? */ - abort(); - } + if (opts.block_size >= (1 << 24)) + BUG("configured block size exceeds 16MB"); strbuf_init(&wp->block_writer_data.last_key, 0); strbuf_init(&wp->last_key, 0); From patchwork Fri May 10 10:29:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13661277 Received: from wfout7-smtp.messagingengine.com (wfout7-smtp.messagingengine.com [64.147.123.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B144416C844 for ; Fri, 10 May 2024 10:29:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336986; cv=none; b=I12nVc86iicUoI8jmJrdLUbo3K9MFQj9jrDjWP+InOFFNdLrG/xY8dGT4GZ0XwsvnzOd83TpdmzTPqc1XvVNULjLOefdWF9SkGNO86X1SBk97CjrkT1EQT5j4pY62z7+7JcyPvWMp/t+67Xd0dRu+ugJnPdSanI0PkQr7E59UUE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336986; c=relaxed/simple; bh=VYjmdAe83h4bfBirPVgRa1l18TlkShUjbRj4pmPXFgk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=c9ueD0KDBqEiVXLqHmWETVi4BaaRmPDyL4JpYgQnCmWjDKmYjKokHVgIHVnOx3oKB3gyOj6/6upaix/RiYds+FLzbwzqFxCmI8q5SrGoLpG+JIVBREuOmNt+HUzFIvzhYJ5HHUgRDuCe07vJT6fAbl6rqvFItU2KdS/f+3EfsFQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=bFqo5dBk; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Ea5T7Q5p; arc=none smtp.client-ip=64.147.123.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="bFqo5dBk"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Ea5T7Q5p" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.west.internal (Postfix) with ESMTP id CAA211C000E5; Fri, 10 May 2024 06:29:43 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Fri, 10 May 2024 06:29:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715336983; x=1715423383; bh=lJ2VUvM+5L XM1o8XtVAXTRGCkiPShI81i4116Y6L+rI=; b=bFqo5dBkYPm9Fxy3a7pYoiTeIj WEhFGKEKJ0TKVtfltPDCMJ8yOBBHxvryD/x7OGfkUd2aa4KwFHuGAQ7sPVALvxLG kYEfg3mfjSuIXxqwL+vLPdMc9++oJxQNvba2me+2WXKpGM2xdLYjvBJwxg/jRY6D a7jt7/rb3e6AV4efCzt/5B6lJjfXESoRcPe987YMps9UIYw6gNqElOd/i5v1BG5U FdGT7VOi988+bS6hACEP19CXd1yWktC/CCywoTED6VnLpI2L4EK4ptVKlJeFo0SO DSu/k4+enPD20pnwm/wb/QSe/93gBr/kL2Y4/3RykBIUXKLwlolDToLUdkHg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715336983; x=1715423383; bh=lJ2VUvM+5LXM1o8XtVAXTRGCkiPS hI81i4116Y6L+rI=; b=Ea5T7Q5pGo0YWNSQsAyj89yK/fEcgKWGyaMOGUIDvzoo pmzAHIQAnklh9EETdT9h1SM2a7eEK68sYxkgM7YPdGT4KCnrWXIyUSPyYtlSNRHl dVJsTmuZ7cTcC0bfuNIcWObrQovM+KneEPHUwaPQPQDeYz+qTjvZUNNTBdAlKvAO lcx6+yh6zYuW7BIfxcqMrTOARCfOZ2wz4M1jWCN16WjnUjNqYxrTA/281tgAhjwJ D4eVXsc1EY2swHJjZzU1xXpAhhVyulrtd/i8XUka7BbJtLThICV7HkNFdu7k1dGg 2Oz+tI1V+3nGcZz6gzTImy3kna6eriv4euzUNyFanA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefkedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 May 2024 06:29:42 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 8324d31b (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 10 May 2024 10:29:30 +0000 (UTC) Date: Fri, 10 May 2024 12:29:40 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler Subject: [PATCH v2 05/11] reftable/dump: support dumping a table's block structure Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We're about to introduce new configs that will allow users to have more control over how exactly reftables are written. To verify that these configs are effective we will need to take a peak into the actual blocks written by the reftable backend. Introduce a new mode to the dumping logic that prints out the block structure. This logic can be invoked via `test-tool dump-reftables -b`. Signed-off-by: Patrick Steinhardt --- reftable/dump.c | 8 ++- reftable/reader.c | 63 ++++++++++++++++++ reftable/reftable-reader.h | 2 + t/t0613-reftable-write-options.sh | 102 ++++++++++++++++++++++++++++++ 4 files changed, 174 insertions(+), 1 deletion(-) create mode 100755 t/t0613-reftable-write-options.sh diff --git a/reftable/dump.c b/reftable/dump.c index 9c770a10cc..24476cc2a9 100644 --- a/reftable/dump.c +++ b/reftable/dump.c @@ -48,6 +48,7 @@ static void print_help(void) printf("usage: dump [-cst] arg\n\n" "options: \n" " -c compact\n" + " -b dump blocks\n" " -t dump table\n" " -s dump stack\n" " -6 sha256 hash format\n" @@ -58,6 +59,7 @@ static void print_help(void) int reftable_dump_main(int argc, char *const *argv) { int err = 0; + int opt_dump_blocks = 0; int opt_dump_table = 0; int opt_dump_stack = 0; int opt_compact = 0; @@ -67,6 +69,8 @@ int reftable_dump_main(int argc, char *const *argv) for (; argc > 1; argv++, argc--) if (*argv[1] != '-') break; + else if (!strcmp("-b", argv[1])) + opt_dump_blocks = 1; else if (!strcmp("-t", argv[1])) opt_dump_table = 1; else if (!strcmp("-6", argv[1])) @@ -88,7 +92,9 @@ int reftable_dump_main(int argc, char *const *argv) arg = argv[1]; - if (opt_dump_table) { + if (opt_dump_blocks) { + err = reftable_reader_print_blocks(arg); + } else if (opt_dump_table) { err = reftable_reader_print_file(arg); } else if (opt_dump_stack) { err = reftable_stack_print_directory(arg, opt_hash_id); diff --git a/reftable/reader.c b/reftable/reader.c index 481dff10d4..f23c8523db 100644 --- a/reftable/reader.c +++ b/reftable/reader.c @@ -856,3 +856,66 @@ int reftable_reader_print_file(const char *tablename) reftable_reader_free(r); return err; } + +int reftable_reader_print_blocks(const char *tablename) +{ + struct { + const char *name; + int type; + } sections[] = { + { + .name = "ref", + .type = BLOCK_TYPE_REF, + }, + { + .name = "obj", + .type = BLOCK_TYPE_OBJ, + }, + { + .name = "log", + .type = BLOCK_TYPE_LOG, + }, + }; + struct reftable_block_source src = { 0 }; + struct table_iter ti = TABLE_ITER_INIT; + struct reftable_reader *r = NULL; + size_t i; + int err; + + err = reftable_block_source_from_file(&src, tablename); + if (err < 0) + goto done; + + err = reftable_new_reader(&r, &src, tablename); + if (err < 0) + goto done; + + printf("header:\n"); + printf(" block_size: %d\n", r->block_size); + + for (i = 0; i < ARRAY_SIZE(sections); i++) { + err = reader_start(r, &ti, sections[i].type, 0); + if (err < 0) + goto done; + if (err > 0) + continue; + + printf("%s:\n", sections[i].name); + + while (1) { + printf(" - length: %u\n", ti.br.block_len); + printf(" restarts: %u\n", ti.br.restart_count); + + err = table_iter_next_block(&ti); + if (err < 0) + goto done; + if (err > 0) + break; + } + } + +done: + reftable_reader_free(r); + table_iter_close(&ti); + return err; +} diff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h index 4a4bc2fdf8..4a04857773 100644 --- a/reftable/reftable-reader.h +++ b/reftable/reftable-reader.h @@ -97,5 +97,7 @@ void reftable_table_from_reader(struct reftable_table *tab, /* print table onto stdout for debugging. */ int reftable_reader_print_file(const char *tablename); +/* print blocks onto stdout for debugging. */ +int reftable_reader_print_blocks(const char *tablename); #endif diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh new file mode 100755 index 0000000000..462980c37c --- /dev/null +++ b/t/t0613-reftable-write-options.sh @@ -0,0 +1,102 @@ +#!/bin/sh + +test_description='reftable write options' + +GIT_TEST_DEFAULT_REF_FORMAT=reftable +export GIT_TEST_DEFAULT_REF_FORMAT +# Disable auto-compaction for all tests as we explicitly control repacking of +# refs. +GIT_TEST_REFTABLE_AUTOCOMPACTION=false +export GIT_TEST_REFTABLE_AUTOCOMPACTION +# Block sizes depend on the hash function, so we force SHA1 here. +GIT_TEST_DEFAULT_HASH=sha1 +export GIT_TEST_DEFAULT_HASH +# Block sizes also depend on the actual refs we write, so we force "master" to +# be the default initial branch name. +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +test_expect_success 'default write options' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + git pack-refs && + cat >expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 129 + restarts: 2 + log: + - length: 262 + restarts: 2 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'disabled reflog writes no log blocks' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + git pack-refs && + cat >expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 129 + restarts: 2 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'many refs results in multiple blocks' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 200) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 4049 + restarts: 11 + - length: 1136 + restarts: 3 + log: + - length: 4041 + restarts: 4 + - length: 4015 + restarts: 3 + - length: 4014 + restarts: 3 + - length: 4012 + restarts: 3 + - length: 3289 + restarts: 3 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_done From patchwork Fri May 10 10:29:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13661278 Received: from wfhigh1-smtp.messagingengine.com (wfhigh1-smtp.messagingengine.com [64.147.123.152]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7BE0216D317 for ; Fri, 10 May 2024 10:29:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.152 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336991; cv=none; b=ZFI5qRCMEJShzN2khF56wMDT0GfM7Y7nQ6rBO9gFrP9H3ssDx4zrnINcp1n3Xye6/WBoswBWISyFNNdoOO/2PXwESocdjzWDFmei40fQNRgywfW3rsMm0EVxKFkrHGybk70eieVkI04yYqrXDnnCGHxi6yfMWlq+vFUM5GlBmqM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336991; c=relaxed/simple; bh=txZBop4l8BE+ntN/zQjxlXqmJbPYgXMgnSYrKyAM+bE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=V609kvVwDNCyU+L6eiJ0/QOmbg25XqQs5bMIz7aDkC24aKLkQ5TMHSMldWVo98kYbo1cYGiSdAwsS6ZKIeA3Hg4ZKvp11hzNxfRFz6GARgMzG6BYeT3HrZBSperBYg2fJHQmJA6xZKuvIty8h9hrOBrq7GpFwv1XQL8BWtVIcn0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=BzxfeRv3; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=fmCq2qGj; arc=none smtp.client-ip=64.147.123.152 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="BzxfeRv3"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="fmCq2qGj" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.west.internal (Postfix) with ESMTP id 9BC291800080; Fri, 10 May 2024 06:29:48 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Fri, 10 May 2024 06:29:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715336988; x=1715423388; bh=IkmrLE/mUs OJ9WJn0TX40aMWkv8lzFxIra/xRyUWJSA=; b=BzxfeRv3dbyM2IOh6QUaowH/J4 x6DxwNOnz6e8WwlhYquhccUgyz7RJ+dR7rCV0/gQRGZc2Pp2zi8sy5Ocdy/RGgCy R/J2sdZn2or2kjREflpq9tHq2jKYc/a19CmUpLWPJGS6QpK9Z6D4fMAV6xqYs/YX EwZ6j9Vo4S/4RsFFNV9LxqJoJgibY6ftP1Qkw8Y+pvsfMR48KAQhJxV28yrJMyXV jhpqFmyMPMhDmWg3TY5c7Zkw2PnLWy/nfq+wOWDk6uJ8DVPEotFhaOdqkgaE1m79 snOud9rfFJk2UPVoCaEj3I6oTtxxELEpBeIfTjRw0ZADZfxUd3Nkv9qlXW0Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715336988; x=1715423388; bh=IkmrLE/mUsOJ9WJn0TX40aMWkv8l zFxIra/xRyUWJSA=; b=fmCq2qGjoshmAJWM78kQWPsjvxH0c1Y6NwRwrxxGFkTK elHs6kq4H+3qEPmndYazpu6pNCiiN95ork/EPySAtVkFohfRJTDwC2YNmJIKdRJt nllhG5y8qZBtjlqZnI8W6bldh0h0QSBtXXhY64rxjH0c7C9q+vQYUg4rnEpzzr9U Al/2047c3okkkJ29b1n+9BdI8PB64NkZuCSYnTuliSm+PEa7eNhzzMUiX7B9MzOK sUh0e2LUdL5H6cvmBaaqz/OwvMnnOA9/CwicZqHCrwr0x809takoka3ZAa58ixfp uvu5cJeZ07dwKuVRwfN92RRufss79JAWPcMC2innVw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefkedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepfeenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 May 2024 06:29:47 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 861b7430 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 10 May 2024 10:29:35 +0000 (UTC) Date: Fri, 10 May 2024 12:29:45 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler Subject: [PATCH v2 06/11] refs/reftable: allow configuring block size Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Add a new option `reftable.blockSize` that allows the user to control the block size used by the reftable library. Signed-off-by: Patrick Steinhardt --- Documentation/config.txt | 2 + Documentation/config/reftable.txt | 14 ++++++ refs/reftable-backend.c | 31 ++++++++++++- t/t0613-reftable-write-options.sh | 72 +++++++++++++++++++++++++++++++ 4 files changed, 118 insertions(+), 1 deletion(-) create mode 100644 Documentation/config/reftable.txt diff --git a/Documentation/config.txt b/Documentation/config.txt index 70b448b132..fa1469e5e7 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -497,6 +497,8 @@ include::config/rebase.txt[] include::config/receive.txt[] +include::config/reftable.txt[] + include::config/remote.txt[] include::config/remotes.txt[] diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt new file mode 100644 index 0000000000..fa7c4be014 --- /dev/null +++ b/Documentation/config/reftable.txt @@ -0,0 +1,14 @@ +reftable.blockSize:: + The size in bytes used by the reftable backend when writing blocks. + The block size is determined by the writer, and does not have to be a + power of 2. The block size must be larger than the longest reference + name or log entry used in the repository, as references cannot span + blocks. ++ +Powers of two that are friendly to the virtual memory system or +filesystem (such as 4kB or 8kB) are recommended. Larger sizes (64kB) can +yield better compression, with a possible increased cost incurred by +readers during access. ++ +The largest block size is `16777215` bytes (15.99 MiB). The default value is +`4096` bytes (4kB). A value of `0` will use the default value. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 1cda48c504..bd9999cefc 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -1,6 +1,7 @@ #include "../git-compat-util.h" #include "../abspath.h" #include "../chdir-notify.h" +#include "../config.h" #include "../environment.h" #include "../gettext.h" #include "../hash.h" @@ -230,6 +231,22 @@ static int read_ref_without_reload(struct reftable_stack *stack, return ret; } +static int reftable_be_config(const char *var, const char *value, + const struct config_context *ctx, + void *_opts) +{ + struct reftable_write_options *opts = _opts; + + if (!strcmp(var, "reftable.blocksize")) { + unsigned long block_size = git_config_ulong(var, value, ctx->kvi); + if (block_size > 16777215) + die("reftable block size cannot exceed 16MB"); + opts->block_size = block_size; + } + + return 0; +} + static struct ref_store *reftable_be_init(struct repository *repo, const char *gitdir, unsigned int store_flags) @@ -245,12 +262,24 @@ static struct ref_store *reftable_be_init(struct repository *repo, base_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable); strmap_init(&refs->worktree_stacks); refs->store_flags = store_flags; - refs->write_options.block_size = 4096; + refs->write_options.hash_id = repo->hash_algo->format_id; refs->write_options.default_permissions = calc_shared_perm(0666 & ~mask); refs->write_options.disable_auto_compact = !git_env_bool("GIT_TEST_REFTABLE_AUTOCOMPACTION", 1); + git_config(reftable_be_config, &refs->write_options); + + /* + * It is somewhat unfortunate that we have to mirror the default block + * size of the reftable library here. But given that the write options + * wouldn't be updated by the library here, and given that we require + * the proper block size to trim reflog message so that they fit, we + * must set up a proper value here. + */ + if (!refs->write_options.block_size) + refs->write_options.block_size = 4096; + /* * Set up the main reftable stack that is hosted in GIT_COMMON_DIR. * This stack contains both the shared and the main worktree refs. diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh index 462980c37c..8bdbc6ec70 100755 --- a/t/t0613-reftable-write-options.sh +++ b/t/t0613-reftable-write-options.sh @@ -99,4 +99,76 @@ test_expect_success 'many refs results in multiple blocks' ' ) ' +test_expect_success 'tiny block size leads to error' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + cat >expect <<-EOF && + error: unable to compact stack: entry too large + EOF + test_must_fail git -c reftable.blockSize=50 pack-refs 2>err && + test_cmp expect err + ) +' + +test_expect_success 'small block size leads to multiple ref blocks' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit A && + test_commit B && + git -c reftable.blockSize=100 pack-refs && + + cat >expect <<-EOF && + header: + block_size: 100 + ref: + - length: 53 + restarts: 1 + - length: 74 + restarts: 1 + - length: 38 + restarts: 1 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'small block size fails with large reflog message' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit A && + perl -e "print \"a\" x 500" >logmsg && + cat >expect <<-EOF && + fatal: update_ref failed for ref ${SQ}refs/heads/logme${SQ}: reftable: transaction failure: entry too large + EOF + test_must_fail git -c reftable.blockSize=100 \ + update-ref -m "$(cat logmsg)" refs/heads/logme HEAD 2>err && + test_cmp expect err + ) +' + +test_expect_success 'block size exceeding maximum supported size' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit A && + test_commit B && + cat >expect <<-EOF && + fatal: reftable block size cannot exceed 16MB + EOF + test_must_fail git -c reftable.blockSize=16777216 pack-refs 2>err && + test_cmp expect err + ) +' + test_done From patchwork Fri May 10 10:29:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13661279 Received: from wfhigh1-smtp.messagingengine.com (wfhigh1-smtp.messagingengine.com [64.147.123.152]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DB0316C443 for ; Fri, 10 May 2024 10:29:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.152 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336996; cv=none; b=LbNM6B3TTzYptxesXOgePfZMqX1BMZYm/9p5OqSgrbWVVS2hTtd8aiQ1dhFTc5tO7mSas2P/pqKS9qIFO3Y0S+dYcGaJqHBODJKOOKSCoMG0IXvldQDhDmgblvQgMx5BE+egeJEHDEqDrK9ZKJENYD5YMeYvRjdN4RhgcF7tfeY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715336996; c=relaxed/simple; bh=QPg5ZB/M/IGceeUAMAmd8EOlRjqGFtvCWvzo6Sl20WI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=eRUNKPkxGs+ZEOxCSRbJxjk3f6DLiUNoWiHta3C26ftSkp+R1NKBtNg4AlsVNrpVvFAugxrhnJJ2A6FJvhVTlVK3xFdxJ3TJj3Sw4x+FFORK8y2af9D2qN/pB17uMZZtlAg5x9JuknvDGxNF6BmBZCxzwxrtGF250/Xb/+0ch+0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=b6OuZrZ8; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=NnjF9QEW; arc=none smtp.client-ip=64.147.123.152 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="b6OuZrZ8"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="NnjF9QEW" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.west.internal (Postfix) with ESMTP id 829F818000BA; Fri, 10 May 2024 06:29:54 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Fri, 10 May 2024 06:29:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715336994; x=1715423394; bh=X8u/shzkQA QX0JpI/efeOG8pkPrL8vibrojcuSAeQhM=; b=b6OuZrZ8EBbVJOrQPXnbYsl2fz IH7aaYgguVbSMZaIp4GDPBGWbkxcWwPQmcfTz1EkUEY7/MtzIrEpc1b2PzHNfrnd eeBKzuVgRcUbbe29qL/kMoFvRbLDZT8SgRWtKvCWyN2aMAn+Fmi1PORXh10onCFo Pzij+2SufwlpD8h2DMmaph9zkWeSI4DGEfShPmES6rFQo06s/UUCmrpjfvITqZEr L/7go6D2n9TK2eOWhLlUBo/FOdZpfzeblQDFljL38Rz2JBFpJX+8KpTFubM2UD6n Wi62KyaXl0p+X5plG7NbBXPADFZuEEylcWvNYWkrOHb2jC1Hn5q1Y9kENmXw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715336994; x=1715423394; bh=X8u/shzkQAQX0JpI/efeOG8pkPrL 8vibrojcuSAeQhM=; b=NnjF9QEWJMvDwxUfAu5yW8oYzuiuLVIbIvM+dP/G6eSP JOyTpEt0I8TBWEotp2bIojbVbx7cq3qCyU6Rvo2MZdpuM4hkyEqZjWF0EbiBCZ5f snTxIL2UClkFFbbi4savkbyX+aqg6/NQ1CwlOQGQ0HSiwxCnUWIwJ5tSh25JoP7C JOKUlG1tUarDriTt7vpEZGXzAPX/tZ7CDijdSDTcwdSXjqoeTGNx14PAKb9GL2wV ILQB/b+2asdNk1zRDQ+t4aaAMwPc/zTZwh74VQLxmFz1Urj+M00hIaLnKxFYIsky 2ehFbBSvqC2BvCUazR5SF6su2WUJvO2PmIcWGdyk8g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefkedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepgeenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 May 2024 06:29:53 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 39f125cc (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 10 May 2024 10:29:40 +0000 (UTC) Date: Fri, 10 May 2024 12:29:51 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler Subject: [PATCH v2 07/11] reftable: use `uint16_t` to track restart interval Message-ID: <05e8d1df2d9921426b803d71b22ba3dba188836c.1715336798.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The restart interval can at most be `UINT16_MAX` as specified in the technical documentation of the reftable format. Furthermore, it cannot ever be negative. Regardless of that we use an `int` to track the restart interval. Change the type to use an `uint16_t` instead. Signed-off-by: Patrick Steinhardt --- reftable/block.h | 2 +- reftable/reftable-writer.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reftable/block.h b/reftable/block.h index ea4384a7e2..cd5577105d 100644 --- a/reftable/block.h +++ b/reftable/block.h @@ -25,7 +25,7 @@ struct block_writer { uint32_t header_off; /* How often to restart keys. */ - int restart_interval; + uint16_t restart_interval; int hash_size; /* Offset of next uint8_t to write. */ diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index 44cb986465..4cd8ebe6c7 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -28,7 +28,7 @@ struct reftable_write_options { unsigned skip_index_objects : 1; /* how often to write complete keys in each block. */ - int restart_interval; + uint16_t restart_interval; /* 4-byte identifier ("sha1", "s256") of the hash. * Defaults to SHA1 if unset From patchwork Fri May 10 10:29:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13661280 Received: from wfout7-smtp.messagingengine.com (wfout7-smtp.messagingengine.com [64.147.123.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57F1B165FA9 for ; Fri, 10 May 2024 10:30:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715337001; cv=none; b=bP0M6AVpA0Eup7rEVPWuBHNptWpnlje5Nm6LLNeQGEcCXkkt5X26/EtTKWyLS3cHqgQsAKP0rQfUllu03HniH5Zcjam+cxS+Ud2LE1ZKUfa6pcU1agwMyloNZ+MHEP1Hzqey/WQq2HI0DmzAo+vnD/TXJ03uyWWLgookhKv9RDw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715337001; c=relaxed/simple; bh=VsfLnbHiG2m07njE5cPOYPUZeD+f9MtCqe2SX0WQX2M=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=H7cEaJ/XCJIHDxyjfWMidUhRALyNARnQV7RlEWwKPov0SSPZyElu3mLZWPWqX5nNoqYvRHnq2UtMn3WXpG95EKSe/aHN7hYintO0oINTVmIvuFqGe6+V2bMyj2pgtTEnmp1SkniqKQxw5ZdKLvAbh45mBM4BZam25+D+xJbV+4Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=TXfzBrxW; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=ZMogNNgb; arc=none smtp.client-ip=64.147.123.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="TXfzBrxW"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="ZMogNNgb" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.west.internal (Postfix) with ESMTP id 5A6371C000D3; Fri, 10 May 2024 06:29:59 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Fri, 10 May 2024 06:29:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715336998; x=1715423398; bh=RYS9Y2gvp0 ExQ3iP7CTzvHI2QaBTTQwBlP92Tc/qzQ0=; b=TXfzBrxW0WveweZk9zAsbAohoE 5/KS+R4l0p4QCeogydjr8wBxqzxYw5xDSdaVkZEPpb/LBPbtv1RodZfWBkJogfts NVR/9tXhoncW4GV7BeyURQUBN+A4qceM3tLTppGL2QmJDnn6f040QfF2bR9RYFM7 3AuvmELz+4Ko08E1M9wSYTsuOoU7MyB6r0PZM3rT7Nq6bhUtQbTC+F2piKptgU3O JTbddxpBkGVH53pVwLMvkFjjuAPH+YXYh2RH1TWlO5Bi4aGe/SE/dPQj79BTPrjs ClkPUS5PV5AYAHkGv1Tw+KkptlA0mnrVp3XzWUBxO0kfC7PQsmKpJ6rAbxKw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715336998; x=1715423398; bh=RYS9Y2gvp0ExQ3iP7CTzvHI2QaBT TQwBlP92Tc/qzQ0=; b=ZMogNNgb4uvEJ3p+kEnRwJfT4lqVx7DHrx7Q/U4hQxVj 5FXKVTv3qNmcqzMSZYFLXsYi3Zy1xdaEocyKkqswlsDU0U+m3Uzn0Mn1o6uHYIKe 9taG1IbFJuK3EgQ4C/FLmEVbE7PWnjjbHYmJzjTC5MTqhISqTN0UcrBXTJS4l+BQ emBGtGMhK3XSZ+eVw/+pDTCL0mI1/E1k90inBpVgTs03+VjiUxrXcp3+LEM8Hvrd RbM74jVvXAdOEZjfvAFPhTKK3YcagcdQzhYNtBqTJo85ULDY4dyLU4uYu/cufmz5 fiNpkAaAUqPQPY7MDFNQErIaxCNCFylxaoVJcxkMtQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefkedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepgeenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 May 2024 06:29:58 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id c11b8a59 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 10 May 2024 10:29:45 +0000 (UTC) Date: Fri, 10 May 2024 12:29:55 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler Subject: [PATCH v2 08/11] refs/reftable: allow configuring restart interval Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Add a new option `reftable.restartInterval` that allows the user to control the restart interval when writing reftable records used by the reftable library. Signed-off-by: Patrick Steinhardt --- Documentation/config/reftable.txt | 19 ++++++++++++++ refs/reftable-backend.c | 5 ++++ t/t0613-reftable-write-options.sh | 43 +++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+) diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt index fa7c4be014..16b915c75e 100644 --- a/Documentation/config/reftable.txt +++ b/Documentation/config/reftable.txt @@ -12,3 +12,22 @@ readers during access. + The largest block size is `16777215` bytes (15.99 MiB). The default value is `4096` bytes (4kB). A value of `0` will use the default value. + +reftable.restartInterval:: + The interval at which to create restart points. The reftable backend + determines the restart points at file creation. The process is + arbitrary, but every 16 or 64 records is recommended. Every 16 may be + more suitable for smaller block sizes (4k or 8k), every 64 for larger + block sizes (64k). ++ +More frequent restart points reduces prefix compression and increases +space consumed by the restart table, both of which increase file size. ++ +Less frequent restart points makes prefix compression more effective, +decreasing overall file size, with increased penalties for readers +walking through more records after the binary search step. ++ +A maximum of `65535` restart points per block is supported. ++ +The default value is to create restart points every 16 records. A value of `0` +will use the default value. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index bd9999cefc..9972dfc1a3 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -242,6 +242,11 @@ static int reftable_be_config(const char *var, const char *value, if (block_size > 16777215) die("reftable block size cannot exceed 16MB"); opts->block_size = block_size; + } else if (!strcmp(var, "reftable.restartinterval")) { + unsigned long restart_interval = git_config_ulong(var, value, ctx->kvi); + if (restart_interval > UINT16_MAX) + die("reftable block size cannot exceed %u", (unsigned)UINT16_MAX); + opts->restart_interval = restart_interval; } return 0; diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh index 8bdbc6ec70..e0a5b26f58 100755 --- a/t/t0613-reftable-write-options.sh +++ b/t/t0613-reftable-write-options.sh @@ -171,4 +171,47 @@ test_expect_success 'block size exceeding maximum supported size' ' ) ' +test_expect_success 'restart interval at every single record' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 10) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 566 + restarts: 13 + log: + - length: 1393 + restarts: 12 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'restart interval exceeding maximum supported interval' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + cat >expect <<-EOF && + fatal: reftable block size cannot exceed 65535 + EOF + test_must_fail git -c reftable.restartInterval=65536 pack-refs 2>err && + test_cmp expect err + ) +' + test_done From patchwork Fri May 10 10:30:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13661281 Received: from wfout7-smtp.messagingengine.com (wfout7-smtp.messagingengine.com [64.147.123.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C9755467A for ; Fri, 10 May 2024 10:30:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715337006; cv=none; b=GKigXhj78QC9qVzo3Q+d7lJGXz1dr/L59Za5IdImRYqnK44B+MC9Xz/ijfawEGWqgnlRUONWzvbZ4NvhrxjxPGf9YGFPUTqjkrvog1FxIt/DNNcHuurORCzLD4qRC3UYypBe5idg0w9jM16n3MYAbxSa55quTDEiLNVYbojmptg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715337006; c=relaxed/simple; bh=jlrl32PHZ06yIBttA3hkNThi9QUxNVP9pq1AoJgN4dA=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=KY+h7XS2nOj2ZL5Xke/MeL+cb10G8FQxDkBMaJYSD1Zvdt6pZVi5N/ySjPTrgumyqBoE1xyHkpXd6OmiCqRspoSIwo1ltAHgc6zti7cLjkQaxlcMW/pqsYR/bUNMj8nYu+p24H1d77oabZFItvwMwVes0Jjenq+lXgfYxYN5fHo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=TYCi9cmT; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=E5u0bbc7; arc=none smtp.client-ip=64.147.123.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="TYCi9cmT"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="E5u0bbc7" Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfout.west.internal (Postfix) with ESMTP id 928EC1C000D3; Fri, 10 May 2024 06:30:04 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Fri, 10 May 2024 06:30:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715337004; x=1715423404; bh=V1hNh4T/yt DNcWNn3EU1CI74zS6631AzKSw5ILiVqt8=; b=TYCi9cmTX+33U0ESAsvccNiYaM mEvL4+uJta84VNOEqp2g5HVSE7fgmqkNH0p1NPDAXEmu5u9Yk7uYpKtLCE8EhPPn donRmMoIYJmtrLuRVs/pmtQATPdzRbv6SbStTKTI5pT9rQDsIY0OU7r1/GCc+8zg dUrHqGBHT0DM6x6zRFNToTrRNHnk0iB0HWA+4IGQi550Baj/fFS6QPT53zk3bR/n Njgn/6bfuYYbLsvk6lngHk7TEDQ10eDQdbK4+VBs3Bs+RNmWpVuHLkBMdKza/g1k xuwkn32j/1PAzUAz+34jFJH43ozid1Hw3eyLaImRZw/KLb0PslkzGue2GPlg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715337004; x=1715423404; bh=V1hNh4T/ytDNcWNn3EU1CI74zS66 31AzKSw5ILiVqt8=; b=E5u0bbc7fIKufpXqNVCLUW5fbJGGdeH130xigbD6eB1T VrDpdx22BP3ebB+dDaq6ymPm/+YLnPF4REdfbitP2vStB6BTU3WALd+hWU5Acmpo cRiiPg9kOSdAGiNRnyH0ytixnyUB5ul33m8Z+BoZjUKEm5rGf02SL0DSy4Hi09pu GORu/Q2fbLXObmHrP2i2xsDegQx9OC9wzXBWKUtJNB6qOh2vu1tdzOXPnnVaxQ5F wqL3V/1za8OXQp6aPN0IrEaciMpGaTik3JF9pZ3KyG66buqgZmoiCvLzVxtGdTBt VUKdb04Q1OfyeaW2Dw61wU/fCvySkonwnqecWP5Npw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefkedgvdelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepvdevudfhteeiveehheeiffduheevjeffveefieeiudetffeitdejgefgvdfghedu necuffhomhgrihhnpegrlhhlohifrghnhihshhgrudhinhifrghnthdrihhtnecuvehluh hsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhssehpkhhsrdhi mh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 May 2024 06:30:02 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 36fb6d3d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 10 May 2024 10:29:50 +0000 (UTC) Date: Fri, 10 May 2024 12:30:00 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler Subject: [PATCH v2 09/11] refs/reftable: allow disabling writing the object index Message-ID: <6bc240fd0cb99d888e5d19423a53a475b6a455ca.1715336798.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Besides the expected "ref" and "log" records, the reftable library also writes "obj" records. These are basically a reverse mapping of object IDs to their respective ref records so that it becomes efficient to figure out which references point to a specific object. The motivation for this data structure is the "uploadpack.allowTipSHA1InWant" config, which allows a client to fetch any object by its hash that has a ref pointing to it. This reverse index is not used by Git at all though, and the expectation is that most hosters nowadays use "uploadpack.allowAnySHA1InWant". It may thus be preferable for many users to disable writing these optional object indices altogether to safe some precious disk space. Add a new config "reftable.indexObjects" that allows the user to disable the object index altogether. Signed-off-by: Patrick Steinhardt --- Documentation/config/reftable.txt | 6 +++ refs/reftable-backend.c | 2 + t/t0613-reftable-write-options.sh | 69 +++++++++++++++++++++++++++++++ 3 files changed, 77 insertions(+) diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt index 16b915c75e..6e4466f3c5 100644 --- a/Documentation/config/reftable.txt +++ b/Documentation/config/reftable.txt @@ -31,3 +31,9 @@ A maximum of `65535` restart points per block is supported. + The default value is to create restart points every 16 records. A value of `0` will use the default value. + +reftable.indexObjects:: + Whether the reftable backend shall write object blocks. Object blocks + are a reverse mapping of object ID to the references pointing to them. ++ +The default value is `true`. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 9972dfc1a3..63b75f770d 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -247,6 +247,8 @@ static int reftable_be_config(const char *var, const char *value, if (restart_interval > UINT16_MAX) die("reftable block size cannot exceed %u", (unsigned)UINT16_MAX); opts->restart_interval = restart_interval; + } else if (!strcmp(var, "reftable.indexobjects")) { + opts->skip_index_objects = !git_config_bool(var, value); } return 0; diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh index e0a5b26f58..e2708e11d5 100755 --- a/t/t0613-reftable-write-options.sh +++ b/t/t0613-reftable-write-options.sh @@ -214,4 +214,73 @@ test_expect_success 'restart interval exceeding maximum supported interval' ' ) ' +test_expect_success 'object index gets written by default with ref index' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 5) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 100 + ref: + - length: 53 + restarts: 1 + - length: 95 + restarts: 1 + - length: 71 + restarts: 1 + - length: 80 + restarts: 1 + obj: + - length: 11 + restarts: 1 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'object index can be disabled' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 5) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 100 + ref: + - length: 53 + restarts: 1 + - length: 95 + restarts: 1 + - length: 71 + restarts: 1 + - length: 80 + restarts: 1 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + test_done From patchwork Fri May 10 10:30:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13661282 Received: from wfhigh1-smtp.messagingengine.com (wfhigh1-smtp.messagingengine.com [64.147.123.152]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0CC0B28F3 for ; Fri, 10 May 2024 10:30:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.152 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715337011; cv=none; b=VVm9utD1FPcxVaG8d1xFkilNGTO0SWzaCr2kjosZ8uGdFFoScTsQc4nO3gfeVbOkQYpl13I3Qv+y9y0RYPqQlzFdtrCloHxCg8RIP+bKG4j4QqLNI4dkyB2VzXzZ+qioZG0E1E9XXKmjLU7+nKxyNVrCOLMNOvn0L4rHjKHWn1k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715337011; c=relaxed/simple; bh=yNVLrZju5o702Hzs0rv9rMNridaG5C1V/lddD8+8fNo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=e1TUilwlnc7VL83feC+sukCZJr730jZpe71aB9LgU47hlXwcMDlDQSsYRvpIKJNgFPssW2ApIr0vnL7b5wYZWWraJEi4zrLAt1EBDl9uDbb1ffb79VaMSB/sp4GVBACduoW5ScP0F+NhtFRCPU5SK1Ax4ecdn47sxxckT57Vtiw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=F4ES207Q; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Coc/JeSh; arc=none smtp.client-ip=64.147.123.152 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="F4ES207Q"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Coc/JeSh" Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfhigh.west.internal (Postfix) with ESMTP id 6BC011800080; Fri, 10 May 2024 06:30:09 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Fri, 10 May 2024 06:30:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715337009; x=1715423409; bh=QyjI5Ret6w ujypSKhG/mvuVjgkJ9denZDhdhGk0hf00=; b=F4ES207QTU3fKIUtKEV8p16/hx jVzETLNQ0TeE4EPWjGH9qxWQnM2Ycu6leatXVdL7uGHv/Kgwu+xEvjMVobFawDLH tpKx5EZ/wSCNCCx/twkRYhmgq4lB/9sjTO+Iuq+mmM17F97HO8w9I0RTV8Op5ZCK tHpGMV1DU3jRn7XAP1stSZW+walJMhK7VXvrrSA19Qc//LkyV5GJg6+pvTVhgL0n xNamj/jqo4yDPvB4UtAbJg4RglFpO30Zee+Lsu0qJv5wyTHrRDwtWzLujS8ctMn9 81lmEOJRsXvMwcvyKYeXdpjd8Fbpsi9pD3CEuGHaYBerDq0nBrUT1ZUupDgQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715337009; x=1715423409; bh=QyjI5Ret6wujypSKhG/mvuVjgkJ9 denZDhdhGk0hf00=; b=Coc/JeSh/zAMatec+TTnsowqkOKCojU42CFv87gG6HPU eTXnUb5wjsYq7G7pLfF2Qr0BBlwApaZJoiADUdVYgztXywSpwW4May1E4xkMR4UV o19RC0SIoWkYPfjuYUA5xBaoQUEub+pe+EVl12IZIePznhLXI6OlnN5O5mAkc3dS HqsWb4LnIHUXwO5iKu89G/r2E3DtW8iATbr3sJ11UaKCjvZsxHrgrCyMPf0CiL/u 8uFkK2MCB4UMEFjkiDGjCHrFtXOtZnxe/QRHxeZ1xhDI3KPPa6zB/QLtnVSL1cK2 EjOwovnTceMyyOk3+cV39eBqbIhAKwkVIrGlyWDr/w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefkedgvdelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 May 2024 06:30:08 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 12030245 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 10 May 2024 10:29:55 +0000 (UTC) Date: Fri, 10 May 2024 12:30:05 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler Subject: [PATCH v2 10/11] reftable: make the compaction factor configurable Message-ID: <9d4c1f034038df2ae232b6665a0d9d7ee5833c5f.1715336798.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When auto-compacting, the reftable library packs references such that the sizes of the tables form a geometric sequence. The factor for this geometric sequence is hardcoded to 2 right now. We're about to expose this as a config option though, so let's expose the factor via write options. Signed-off-by: Patrick Steinhardt --- reftable/constants.h | 1 + reftable/reftable-writer.h | 6 ++++++ reftable/stack.c | 14 ++++++++++---- reftable/stack.h | 3 ++- reftable/stack_test.c | 4 ++-- 5 files changed, 21 insertions(+), 7 deletions(-) diff --git a/reftable/constants.h b/reftable/constants.h index 5eee72c4c1..f6beb843eb 100644 --- a/reftable/constants.h +++ b/reftable/constants.h @@ -17,5 +17,6 @@ license that can be found in the LICENSE file or at #define MAX_RESTARTS ((1 << 16) - 1) #define DEFAULT_BLOCK_SIZE 4096 +#define DEFAULT_GEOMETRIC_FACTOR 2 #endif diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index 4cd8ebe6c7..155457b042 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -49,6 +49,12 @@ struct reftable_write_options { /* boolean: Prevent auto-compaction of tables. */ unsigned disable_auto_compact : 1; + + /* + * Geometric sequence factor used by auto-compaction to decide which + * tables to compact. Defaults to 2 if unset. + */ + uint8_t auto_compaction_factor; }; /* reftable_block_stats holds statistics for a single block type */ diff --git a/reftable/stack.c b/reftable/stack.c index 7b4fff7c9e..762954b181 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -10,6 +10,7 @@ license that can be found in the LICENSE file or at #include "../write-or-die.h" #include "system.h" +#include "constants.h" #include "merged.h" #include "reader.h" #include "refname.h" @@ -1215,12 +1216,16 @@ static int segment_size(struct segment *s) return s->end - s->start; } -struct segment suggest_compaction_segment(uint64_t *sizes, size_t n) +struct segment suggest_compaction_segment(uint64_t *sizes, size_t n, + uint8_t factor) { struct segment seg = { 0 }; uint64_t bytes; size_t i; + if (!factor) + factor = DEFAULT_GEOMETRIC_FACTOR; + /* * If there are no tables or only a single one then we don't have to * compact anything. The sequence is geometric by definition already. @@ -1252,7 +1257,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n) * 64, 32, 16, 8, 4, 3, 1 */ for (i = n - 1; i > 0; i--) { - if (sizes[i - 1] < sizes[i] * 2) { + if (sizes[i - 1] < sizes[i] * factor) { seg.end = i + 1; bytes = sizes[i]; break; @@ -1278,7 +1283,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n) uint64_t curr = bytes; bytes += sizes[i - 1]; - if (sizes[i - 1] < curr * 2) { + if (sizes[i - 1] < curr * factor) { seg.start = i - 1; seg.bytes = bytes; } @@ -1304,7 +1309,8 @@ int reftable_stack_auto_compact(struct reftable_stack *st) { uint64_t *sizes = stack_table_sizes_for_compaction(st); struct segment seg = - suggest_compaction_segment(sizes, st->merged->stack_len); + suggest_compaction_segment(sizes, st->merged->stack_len, + st->opts.auto_compaction_factor); reftable_free(sizes); if (segment_size(&seg) > 0) return stack_compact_range_stats(st, seg.start, seg.end - 1, diff --git a/reftable/stack.h b/reftable/stack.h index 97d7ebc043..5b45cff4f7 100644 --- a/reftable/stack.h +++ b/reftable/stack.h @@ -35,6 +35,7 @@ struct segment { uint64_t bytes; }; -struct segment suggest_compaction_segment(uint64_t *sizes, size_t n); +struct segment suggest_compaction_segment(uint64_t *sizes, size_t n, + uint8_t factor); #endif diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 3316d55f19..f6c11ef18d 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -767,7 +767,7 @@ static void test_suggest_compaction_segment(void) { uint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 }; struct segment min = - suggest_compaction_segment(sizes, ARRAY_SIZE(sizes)); + suggest_compaction_segment(sizes, ARRAY_SIZE(sizes), 2); EXPECT(min.start == 1); EXPECT(min.end == 10); } @@ -776,7 +776,7 @@ static void test_suggest_compaction_segment_nothing(void) { uint64_t sizes[] = { 64, 32, 16, 8, 4, 2 }; struct segment result = - suggest_compaction_segment(sizes, ARRAY_SIZE(sizes)); + suggest_compaction_segment(sizes, ARRAY_SIZE(sizes), 2); EXPECT(result.start == result.end); } From patchwork Fri May 10 10:30:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13661283 Received: from wfhigh1-smtp.messagingengine.com (wfhigh1-smtp.messagingengine.com [64.147.123.152]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 378EA79DD5 for ; Fri, 10 May 2024 10:30:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.152 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715337018; cv=none; b=AQIFAIGND5uLfKK6HrQWZVczHpLvuRfY0sR5wQmVJcWYK8rbxz/dhmrdbe2evNnz4vfbWODQVCWXfC8L3rZm2Qz65GGDrhy8A2DF80nVnhVfX7Hpt3yt1uaUFuAfZRR9d60uiBGEA2088otZl8krmAFUQcsHSZuHf/Q/wBwRjUA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715337018; c=relaxed/simple; bh=Gl7ZTVMOJ/h6fzUrAqsNOuGAtq0XFZAu7QvREBWZ7Mo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=M2ZVGiZL05Ave4S0tVK6+07tW6fmES0kHYrgNNsIWcfm6Jq8aiUBmweM8KzsRsL1PdneAZWsv8vuQECgeizgCLpY/mxMNmCKprxLUNMKzbOqrVIJZbDKOxvnSmjPKFDQqq/CrewwnqJ+bywO8f1zu+kCH48Wvgl6Ylv34HNCFnw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=jJuIwMy4; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=BiKNm1RH; arc=none smtp.client-ip=64.147.123.152 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="jJuIwMy4"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="BiKNm1RH" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.west.internal (Postfix) with ESMTP id 50F4118000BA; Fri, 10 May 2024 06:30:15 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Fri, 10 May 2024 06:30:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715337014; x=1715423414; bh=fSMPnY6FHe 5yndlLV7GMeubYQklGc0YtLWSPOUuSBNE=; b=jJuIwMy4NSRvGlSwIq4fydkty8 7wtRs9USH02H6I5Ovo6N+0ynXcX2CB+DuAJhk8q9YIZr5VioyEI6t+iRAu/Bg+QN eNDMw4uZA1QxjVDXy3MQXtbQIqoBKnk9DflosWAdRRup7cXF1uqNJhYJFRwaz4UF 9GOAln0gwE0tV4zQ7aSULY1XcuvMDymE04xj6SvV9RO8VCgm/ef3Q/PeAVm6fS2p 2G9Y9ZHUS/NC7ah/ZQrVUA5edxy6xMxiKzcy0vQ2pVKJ85k4tbwGPkji/yAeaaMJ eOfB8yuA4qpeskQPH1Lm3P3yuwrpfSYhpQGfvzvAxzX20roh3eOcJjsHXrlg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715337014; x=1715423414; bh=fSMPnY6FHe5yndlLV7GMeubYQklG c0YtLWSPOUuSBNE=; b=BiKNm1RHhBUVA8OXjqJ3akO6/D24lYOuJgKrCBO9Cg+M 5HzlHY/q543Zq+2fzrEE6aBQJrRr6LRvryqi4J9/Tao+cfxVwtudRHEiEXDxdXxY 0uPEatnL7mAd4fa+6FLWFwA8NaV4jNK9CaEAZwwOqlrFC5Czmf+h6UNYEBJx7PjE j6I0jDtFKMKm2IIgj9cshXYk1UsV2WBhOKWBWLi2Q8JnTpA4JP0D5yWsf0YKek9r GTtyWQ2fu3sQg0rTxTVZJlw46Z0QRzGmTPOwByZXeENVQDIQc4UEEbmEuk2ZthyC oDfoUCvU2zBB0GVreI6Fke9Cik739JV2bdoDaU/Dmg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefkedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepieenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 May 2024 06:30:13 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id b284ffc3 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 10 May 2024 10:30:01 +0000 (UTC) Date: Fri, 10 May 2024 12:30:11 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler Subject: [PATCH v2 11/11] refs/reftable: allow configuring geometric factor Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Allow configuring the geometric factor used by the auto-compaction algorithm whenever a new table is appended to the stack of tables. Signed-off-by: Patrick Steinhardt --- Documentation/config/reftable.txt | 10 ++++++++++ refs/reftable-backend.c | 5 +++++ 2 files changed, 15 insertions(+) diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt index 6e4466f3c5..f928d1029b 100644 --- a/Documentation/config/reftable.txt +++ b/Documentation/config/reftable.txt @@ -37,3 +37,13 @@ reftable.indexObjects:: are a reverse mapping of object ID to the references pointing to them. + The default value is `true`. + +reftable.geometricFactor:: + Whenever the reftable backend appends a new table to the stack, it + performs auto compaction to ensure that there is only a handful of + tables. The backend does this by ensuring that tables form a geometric + sequence regarding the respective sizes of each table. ++ +By default, the geometric sequence uses a factor of 2, meaning that for any +table, the next-biggest table must at least be twice as big. A maximum factor +of 256 is supported. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 63b75f770d..c4bd0fe1f9 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -249,6 +249,11 @@ static int reftable_be_config(const char *var, const char *value, opts->restart_interval = restart_interval; } else if (!strcmp(var, "reftable.indexobjects")) { opts->skip_index_objects = !git_config_bool(var, value); + } else if (!strcmp(var, "reftable.geometricfactor")) { + unsigned long factor = git_config_ulong(var, value, ctx->kvi); + if (factor > UINT8_MAX) + die("reftable geometric factor cannot exceed %u", (unsigned)UINT8_MAX); + opts->auto_compaction_factor = factor; } return 0;