From patchwork Thu Apr 4 05:48:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13617308 Received: from fhigh7-smtp.messagingengine.com (fhigh7-smtp.messagingengine.com [103.168.172.158]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27B294CE1B for ; Thu, 4 Apr 2024 05:48:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.158 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712209719; cv=none; b=s8G+DKDni/fUlz9E5EZzZMBbAkCISDEVqGAcUBHLxw1THq47ydQox0G2MmrV8Fbkyb/PKKrr0VYxtjwm2p2Jr2crfnuC/GJ39aM9v0RQhTqTQRz/uJw+m+BZx8cDKaL3lw+vHdQxgVzjlD5A+pdK9a+6GiJ8duXAfJttZtlvVQ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712209719; c=relaxed/simple; bh=vO6PHnpDo6QIv4/et9J3hYZH9umrOstXMd0pzKb42yc=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=dys0boZovRMoaYDAN+D9GUjKV4vIIrT0Q71bJkzixtQAfNzZRdd8hq8Nq+SnvWin7SarqPjbSbXyWM1RfsTyS/p1cbFXijUDgyS5Hol+GiRNy0FfVTX+vrPPPLkWd3Z5fQkEmlRHuZHG2+pHe803cy0F7x/T/lUosP7Bx6x6dg8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=Nb4A+Vf2; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=bDs/mbC/; arc=none smtp.client-ip=103.168.172.158 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="Nb4A+Vf2"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="bDs/mbC/" Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 2F8A1114010A; Thu, 4 Apr 2024 01:48:37 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Thu, 04 Apr 2024 01:48:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1712209717; x=1712296117; bh=x316v33e6U 3ej5yuNCPG5F96q1u52bzfiSqEaBcVy2Y=; b=Nb4A+Vf2SWpYXXh0Yz64jfOsw2 eVifetoErwA+tEkeAm1191V5iBCMg9KHCX7BJUuBNft95kNNmItUdXyJAFGT2m8q DDFnnkrxL9Z2BYXmonjHub/pwCw+GtRyTeEkZHA5gBc/lsDIXsSqP4LNqMbcygQn eDiH5vLBn1kCmIiXooJMEgWzb6hKhZghBZNXh+1Igaj6c4IZQW2KC3oUsRJkhsX6 GHhsDUpy3EtdwN2MKPOiPIs7tZ62QQnP5eADLug6v8xbiAaAKl1vkb95xl4OB+X4 KNrBXjzv025jm47car5yy9sTW65ZA4l4VLDFH+MxidNYYN/g5/wA28uVLVHA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1712209717; x=1712296117; bh=x316v33e6U3ej5yuNCPG5F96q1u5 2bzfiSqEaBcVy2Y=; b=bDs/mbC/tISywahUZKbaNUok22rF4rNbVDzxCwvHyuM4 DLER9erYVWKoyM3ti52cbDpLHCQBtqftXZbTyJLcY65T/7mEgEVw0wMCbN7a2C8Z hKCea7UFWi9dHDMfxeo7ewc+bkmGG3hf7Vc+9c4CRp9nXr6tQdIrYtTZe48Q63kS 00BArJaSBv6D/pBB2y7cVNUMLuXnKf88qTau1PXdVYN0WifJtQPAZWk/rnvN7SNW Unr9tdbZNl7anRVkRjx0IcVDKbXsrooZwgyKQKmUDAQo02TCkjf0WJaYOGc2j5xk eQpuEu5jY+Bt1BouYK8RzGPpV03vn3JQJIxSbiRRXw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrudefjedguddttdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvfevuffkfhggtggujgesgh dtreertddtvdenucfhrhhomheprfgrthhrihgtkhcuufhtvghinhhhrghrughtuceophhs sehpkhhsrdhimheqnecuggftrfgrthhtvghrnhepueektdevtdffveeljeetgfehheeige ekleduvdeffeeghefgledttdehjeelffetnecuvehluhhsthgvrhfuihiivgeptdenucfr rghrrghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 4 Apr 2024 01:48:36 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 6d73a381 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 4 Apr 2024 05:48:33 +0000 (UTC) Date: Thu, 4 Apr 2024 07:48:33 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano , Han-Wen Nienhuys Subject: [PATCH v2 07/11] reftable/writer: refactorings for `writer_flush_nonempty_block()` Message-ID: <8f1c5b416986f6d4934bbfefe18ad7ed55231671.1712209149.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Large parts of the reftable library do not conform to Git's typical code style. Refactor `writer_flush_nonempty_block()` such that it conforms better to it and add some documentation that explains some of its more intricate behaviour. Signed-off-by: Patrick Steinhardt --- reftable/writer.c | 72 +++++++++++++++++++++++++++++------------------ 1 file changed, 44 insertions(+), 28 deletions(-) diff --git a/reftable/writer.c b/reftable/writer.c index 0ad5eb8887..d347ec4cc6 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -659,58 +659,74 @@ static void writer_clear_index(struct reftable_writer *w) w->index_cap = 0; } -static const int debug = 0; - static int writer_flush_nonempty_block(struct reftable_writer *w) { + struct reftable_index_record index_record = { + .last_key = STRBUF_INIT, + }; uint8_t typ = block_writer_type(w->block_writer); - struct reftable_block_stats *bstats = - writer_reftable_block_stats(w, typ); - uint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0; - int raw_bytes = block_writer_finish(w->block_writer); - int padding = 0; - int err = 0; - struct reftable_index_record ir = { .last_key = STRBUF_INIT }; + struct reftable_block_stats *bstats; + int raw_bytes, padding = 0, err; + uint64_t block_typ_off; + + /* + * Finish the current block. This will cause the block writer to emit + * restart points and potentially compress records in case we are + * writing a log block. + * + * Note that this is still happening in memory. + */ + raw_bytes = block_writer_finish(w->block_writer); if (raw_bytes < 0) return raw_bytes; - if (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) { + /* + * By default, all records except for log records are padded to the + * block size. + */ + if (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) padding = w->opts.block_size - raw_bytes; - } - if (block_typ_off > 0) { + bstats = writer_reftable_block_stats(w, typ); + block_typ_off = (bstats->blocks == 0) ? w->next : 0; + if (block_typ_off > 0) bstats->offset = block_typ_off; - } - bstats->entries += w->block_writer->entries; bstats->restarts += w->block_writer->restart_len; bstats->blocks++; w->stats.blocks++; - if (debug) { - fprintf(stderr, "block %c off %" PRIu64 " sz %d (%d)\n", typ, - w->next, raw_bytes, - get_be24(w->block + w->block_writer->header_off + 1)); - } - - if (w->next == 0) { + /* + * If this is the first block we're writing to the table then we need + * to also write the reftable header. + */ + if (!w->next) writer_write_header(w, w->block); - } err = padded_write(w, w->block, raw_bytes, padding); if (err < 0) return err; + /* + * Add an index record for every block that we're writing. If we end up + * having more than a threshold of index records we will end up writing + * an index section in `writer_finish_section()`. Each index record + * contains the last record key of the block it is indexing as well as + * the offset of that block. + * + * Note that this also applies when flushing index blocks, in which + * case we will end up with a multi-level index. + */ REFTABLE_ALLOC_GROW(w->index, w->index_len + 1, w->index_cap); - - ir.offset = w->next; - strbuf_reset(&ir.last_key); - strbuf_addbuf(&ir.last_key, &w->block_writer->last_key); - w->index[w->index_len] = ir; - + index_record.offset = w->next; + strbuf_reset(&index_record.last_key); + strbuf_addbuf(&index_record.last_key, &w->block_writer->last_key); + w->index[w->index_len] = index_record; w->index_len++; + w->next += padding + raw_bytes; w->block_writer = NULL; + return 0; }