From patchwork Thu May 2 06:52:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13651330 Received: from fhigh8-smtp.messagingengine.com (fhigh8-smtp.messagingengine.com [103.168.172.159]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CFA524A0E for ; Thu, 2 May 2024 06:52:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.159 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632732; cv=none; b=FBPUsZktldN1H9xsP501s9YkCByTJWiXX/KKUGY2jvtPOYODs5uerZ5qT1/IMU/xBzqAyAXw3IMdcCB+ZYJ32SvoPxNKKHSIoXBLiOBqxKUSBDXjt7srpqYIr5JWXsRT1Y/kVqB1+A8ml9XlLlTXQTC7X0IjfKG7O0huQjTKdew= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632732; c=relaxed/simple; bh=DMa7YorWs6qN2HC4D/sOzWM8AZexY4TFGppjp3KhHUM=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Tm0LhFVWCQJ60TFg2Wu8jSK7fmcVl2aQUeCdMFBSIZTi9ejP+jsKPUbOuUso+TqEpraED4i6C6SmvMQqoubCYgQsajYwYCVC6FoBwUkG4q6uNL7c13/xw1x5wfSrRD50EaysqGUb5u8D7/nEvW455zkx1RGSL/Jo7oPmsDmdhMs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=roc7V4lK; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Grd9aj6V; arc=none smtp.client-ip=103.168.172.159 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="roc7V4lK"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Grd9aj6V" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 6AEFD11400CA for ; Thu, 2 May 2024 02:52:09 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Thu, 02 May 2024 02:52:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1714632729; x=1714719129; bh=jOZ/nVo3ci uX/XcMkTwPXvB6JqMdUK802enhZbN2XHs=; b=roc7V4lK8HrY60yqf5rXndOfX2 pTyjjktQYdgl0InsI3JbCWbL38RvgMTG1QzxCgWNjD/+KABTPZeJVjP1+yZI6esa nV3EpMGMt74fZ1EjbA4ELSlGgtTOZJccStqVVr85XQH4Pg9bMsDoyB0Ygao/Antu 018NitrKytvqr6Cc7M3I23gUEuA3nazhHUpm8XxK9jXkudasua19uFYQ7rVIkq6t +eelCF2G43XJ7B8xCrYOaAD7t7rIaLfS+xjV24G5wk00r8/6zfHkdcF9hvBIV3ou 758oKOD1Hx6nIZlWZJ92FDUwTzYTIqxjS3Z7IE1u8SuzvjhWDYf1eUJOwgTg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1714632729; x=1714719129; bh=jOZ/nVo3ciuX/XcMkTwPXvB6JqMd UK802enhZbN2XHs=; b=Grd9aj6Ved1TdyFRwoqJrjEggEpOmBIRnCq2WJtGRHGp z3fMvV1YRfJdNlbEBa1+OflSn6VjfJXADM9hTPPrjO2ZlQKcZ7MswdwOYwZJyIn6 oRO2vyEM2Zz5JcocZWeKZXojwnMa3pP1Qs5CNLkuedqhctkC8oF8ngGhye5VSKBi bl9QcvuHLUz0KCztS5Mog197iRn9vdnBVAQ5g8r6nFmBt3yrjd5drg5G+00JPxHY ASS3FlyZ7TxQ0YEEsDKOKjWjfk0cn6wSlW7V6RgJFrd0u43kCwT/UF3zmh+IEIIa 2vev3KFaqfMBkt3Yv7RQWGbbjisyvQGZmlVZ/HeRCA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddujedgudduudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 2 May 2024 02:52:08 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id d92da9b5 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 2 May 2024 06:51:43 +0000 (UTC) Date: Thu, 2 May 2024 08:52:06 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 08/11] refs/reftable: allow configuring restart interval Message-ID: <653ec4dfa59b20e0ca9ef76bb5f2c1725c2bc078.1714630191.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Add a new option `reftable.restartInterval` that allows the user to control the restart interval when writing reftable records used by the reftable library. Signed-off-by: Patrick Steinhardt --- Documentation/config/reftable.txt | 19 ++++++++++++++ refs/reftable-backend.c | 6 +++++ t/t0613-reftable-write-options.sh | 43 +++++++++++++++++++++++++++++++ 3 files changed, 68 insertions(+) diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt index fa7c4be014..16b915c75e 100644 --- a/Documentation/config/reftable.txt +++ b/Documentation/config/reftable.txt @@ -12,3 +12,22 @@ readers during access. + The largest block size is `16777215` bytes (15.99 MiB). The default value is `4096` bytes (4kB). A value of `0` will use the default value. + +reftable.restartInterval:: + The interval at which to create restart points. The reftable backend + determines the restart points at file creation. The process is + arbitrary, but every 16 or 64 records is recommended. Every 16 may be + more suitable for smaller block sizes (4k or 8k), every 64 for larger + block sizes (64k). ++ +More frequent restart points reduces prefix compression and increases +space consumed by the restart table, both of which increase file size. ++ +Less frequent restart points makes prefix compression more effective, +decreasing overall file size, with increased penalties for readers +walking through more records after the binary search step. ++ +A maximum of `65535` restart points per block is supported. ++ +The default value is to create restart points every 16 records. A value of `0` +will use the default value. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index c2c47a3bc1..a786143de2 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -243,6 +243,12 @@ static int reftable_be_config(const char *var, const char *value, die("reftable block size cannot exceed 16MB"); opts->block_size = block_size; return 0; + } else if (!strcmp(var, "reftable.restartinterval")) { + unsigned long restart_interval = git_config_ulong(var, value, ctx->kvi); + if (restart_interval > UINT16_MAX) + die("reftable block size cannot exceed %u", (unsigned)UINT16_MAX); + opts->restart_interval = restart_interval; + return 0; } return 0; diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh index 8bdbc6ec70..e0a5b26f58 100755 --- a/t/t0613-reftable-write-options.sh +++ b/t/t0613-reftable-write-options.sh @@ -171,4 +171,47 @@ test_expect_success 'block size exceeding maximum supported size' ' ) ' +test_expect_success 'restart interval at every single record' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 10) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 566 + restarts: 13 + log: + - length: 1393 + restarts: 12 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'restart interval exceeding maximum supported interval' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + cat >expect <<-EOF && + fatal: reftable block size cannot exceed 65535 + EOF + test_must_fail git -c reftable.restartInterval=65536 pack-refs 2>err && + test_cmp expect err + ) +' + test_done