From patchwork Wed May 29 22:55:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13679520 Received: from mail-qv1-f51.google.com (mail-qv1-f51.google.com [209.85.219.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F378D1CB31B for ; Wed, 29 May 2024 22:55:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023324; cv=none; b=GBMS9YH75FF7IQ3NOd5qlpIzKahDnX40cG3AR+F2CzglHNjDvC9z7Fr/2+UAINDRKmWGwyEB05bS8CdopPLImTe1hdfG1SrkRTfiQdeIoTprxT/9zza2fhtCOcvedhWXqDW3aj4nvV2oR8VBvCVFza1lHd/tICOivqXF1ZCE8jQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023324; c=relaxed/simple; bh=ncEr/vdaghLw4HaNscI2g7Vk7X2GkWVl7hYiIoOHUag=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=CuBho1j3oNyyhhFlIND0hTUzhU89E3YvHE1yfhKaD1bZWT4Hev+LiKE1J+ILSoy/LGW2MozE7Vhd3/thYbSKAwJPyMt9HKNuzJWFGUJI+ejR9mA+ZeWEFm1NKTSrMAgnhJ0bTdF/XI0RpZpl4gJBWDmWUoeIdjdQwmGzC7w8v3Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=BTp8ZMap; arc=none smtp.client-ip=209.85.219.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="BTp8ZMap" Received: by mail-qv1-f51.google.com with SMTP id 6a1803df08f44-6ab9d01c479so1089286d6.1 for ; Wed, 29 May 2024 15:55:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1717023321; x=1717628121; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=j2qnGTH7FSQCQFCFupyWglJmKwfpyviSgK3xaNDTBnA=; b=BTp8ZMapQ7rIXGoXdivbDl4OjcpAckN3G9JY+hJWcFcc3SJZNmRzhdFSpp/4bQlt2Z L8THm0z3LRvRBrq+kPjsJ0EWlNWdhmyFyDo7cRCROFhHSCw8jFCfIW8B99rtGy1WZVmx latBKri9CRSkxhNVpvFqVJFDEi3sGhcvE/srS2HmERJIZ8NkKTorm8/XKbye+WbtJWGI PJoOIsgfoZvZwTj/ew9B85S9hAeTnlOPYd9ZliO8IBf75cAeRsgBRtBBHU6hsiGIeonR GIzCWaENgME8ssmiVJrQm1Sb9wHybLfOzzTna8o6RH4HKuADEHtx2ybCHAlM+p67XYZq FXBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717023321; x=1717628121; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=j2qnGTH7FSQCQFCFupyWglJmKwfpyviSgK3xaNDTBnA=; b=KaG176hr7RjPOU73Hf2a6DOuFby+7ueVl8AWeAvYvQJKps0Dev/++xJX3gDZx2IRC2 Zwa1UvtK9xjnnh6VnOLZphGWic+8gmPVASuLFOg/C4vThDcGNwprupeB8OBQcT9lmWPo o1Udx0wveWD7MNET5z7gDKcLAyaT2WzW+R8zcG9BKGQ2Dz1KSzhp8n/wUjR4vMblOfsw bTSDbaRP5WA8X0X7YAq/cX21q8Sw5cYW9Bbfa+v8AFnPy6DDZWiSy0ikH77mqyESRI3O gCgxwSd39nwe2LYXuTuHrsMLh5XA/ykEJlBXR4MjIuyLhuADnPtA4HAQopUhwb2/ZE91 pmng== X-Gm-Message-State: AOJu0YySRUTe8jRYOE2rCpDazarz8S7l+NTBZXV+l6puCGwfjHb5Rtz1 qzkjh6aAR2hASsUB53ERG6YLMJhe135XEnewP23LQ9l8+RG1RZ571H5dBg2lOjOynyyg5qvOJU4 HR8I= X-Google-Smtp-Source: AGHT+IGpGu1XA8wx8LQ8yrvMyCp6DnM920ofPRE+3nQs3cmhY/OygySWZuHMM8vdGiRlz9dXQ9Vedg== X-Received: by 2002:a05:6214:2f83:b0:6a0:6676:f1c8 with SMTP id 6a1803df08f44-6ae0cce535amr4309586d6.59.1717023321375; Wed, 29 May 2024 15:55:21 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6ac20a134f5sm57709746d6.146.2024.05.29.15.55.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 15:55:20 -0700 (PDT) Date: Wed, 29 May 2024 18:55:19 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v2 1/8] midx-write.c: tolerate `--preferred-pack` without bitmaps Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When passing a preferred pack to the MIDX write machinery, we ensure that the given preferred pack is non-empty since 5d3cd09a808 (midx: reject empty `--preferred-pack`'s, 2021-08-31). However packs are only loaded (via `write_midx_internal()`, though a subsequent patch will refactor this code out to its own function) when the `MIDX_WRITE_REV_INDEX` flag is set. So if a caller runs: $ git multi-pack-index write --preferred-pack=... with both (a) an existing MIDX, and (b) specifies a pack from that MIDX as the preferred one, without passing `--bitmap`, then the check added in 5d3cd09a808 will result in a segfault. Note that packs loaded from disk which don't appear in an existing MIDX do not trigger this issue, as those packs are loaded unconditionally. We conditionally load packs from a MIDX since we tolerate MIDXs whose packs do not resolve (i.e., via the MIDX write after removing unreferenced packs via 'git multi-pack-index expire'). In practice, this isn't possible to trigger when running `git multi-pack-index write` from `git repack`, as the latter always passes `--stdin-packs`, which prevents us from loading an existing MIDX, as it forces all packs to be read from disk. But a future commit in this series will change that behavior to unconditionally load an existing MIDX, even with `--stdin-packs`, making this behavior trigger-able from 'repack' much more easily. Prevent this from being an issue by removing the segfault altogether by calling `prepare_midx_pack()` on packs loaded from an existing MIDX when either the `MIDX_WRITE_REV_INDEX` flag is set *or* we specified a `--preferred-pack`. Signed-off-by: Taylor Blau --- midx-write.c | 8 +++++++- t/t5319-multi-pack-index.sh | 23 +++++++++++++++++++++++ 2 files changed, 30 insertions(+), 1 deletion(-) diff --git a/midx-write.c b/midx-write.c index 2fa120c213..591b79bcbf 100644 --- a/midx-write.c +++ b/midx-write.c @@ -935,11 +935,17 @@ static int write_midx_internal(const char *object_dir, for (i = 0; i < ctx.m->num_packs; i++) { ALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc); - if (flags & MIDX_WRITE_REV_INDEX) { + if (flags & MIDX_WRITE_REV_INDEX || + preferred_pack_name) { /* * If generating a reverse index, need to have * packed_git's loaded to compare their * mtimes and object count. + * + * If a preferred pack is specified, + * need to have packed_git's loaded to + * ensure the chosen preferred pack has + * a non-zero object count. */ if (prepare_midx_pack(the_repository, ctx.m, i)) { error(_("could not load pack")); diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index dd09134db0..10d2a6bf92 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -350,6 +350,29 @@ test_expect_success 'preferred packs must be non-empty' ' ) ' +test_expect_success 'preferred pack from existing MIDX without bitmaps' ' + git init preferred-without-bitmaps && + ( + cd preferred-without-bitmaps && + + test_commit one && + pack="$(git pack-objects --all $objdir/pack/pack X-Patchwork-Id: 13679521 Received: from mail-qk1-f175.google.com (mail-qk1-f175.google.com [209.85.222.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E6FD1CB31B for ; Wed, 29 May 2024 22:55:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023327; cv=none; b=f95kAnH6kKtX/gJEBk3AM/3lPZyzPJ5Wf1o5WPVPk/ITt+iTUTb79KQjxCq9EUQQlVgDcZt6Pfoo3A9gkxasr6dz+k2WNk4hjTWgEEwMvoa6eGoj3qiUM9LnGlLlwHEAdRw08IJ5tDel5L6zAEpJxzXkjX5dv62YLdyVjqIRpDc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023327; c=relaxed/simple; bh=IpXTMwUILuLFPACSIhLeVaJycwURNGnqr850stQJI/A=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=fN/iEH7EGB+5X7s8ReO+8jocvlYqnQa1c7A5+fx1RhimxYz3oPBqZC9KTNaMpCd4M82C0qTJ2wZCBOEz+XSBxGKjQIk/QRWF1v38rKfbBoAULd+ZCVQtwRfYQDYC8cdTwx3YUB/ll5Sx9QEj8A3mmwmz/uUm4hJHuGOl27CDqic= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=cZPZKYY0; arc=none smtp.client-ip=209.85.222.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="cZPZKYY0" Received: by mail-qk1-f175.google.com with SMTP id af79cd13be357-794e29c7775so23702685a.3 for ; Wed, 29 May 2024 15:55:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1717023325; x=1717628125; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=4BDj1iFzT9mvQaAR3L1LF+fq7Hesr51hL9BkM3aDzZQ=; b=cZPZKYY00px46dB+8QSgufIcGmb7UhugsqIDu02HAqLldkQRR9uec6hBVYbeJgeBau whbJij9YIheVzsHLqYqLLGhXsyYHmASe+2ukjs9+cqVGSJIsa+WrOafhutTlnWU5rmeb TKqpJFVkaPHxh9NmiJbsFBW5yRcZjh5s4fVKdKCXjoAtoxn8puoIPQrP5xcZsBYDvJCm F45UQHq1CS5TmyI46UawP5I+2Hl92EpH+NmKycx2h59h9sqpGZbxHsr2bMwqbsnZoedQ gjfQlJMQoVatLEVK+cjfp6Z3vzfz+wRBQ9lEtHRPFH5Nd4/2EIkAVWI/XdMAD9Ye9rXw 45Mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717023325; x=1717628125; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=4BDj1iFzT9mvQaAR3L1LF+fq7Hesr51hL9BkM3aDzZQ=; b=oo5HcnuSIlhj8WcgTEXlcz5jnX01Myd4o9HbEublbvgL2mtF3JIzpBmsYFmUEUnpEP PZzJN0dI592MDgRGkJ3fl2xBWi48YhGNEVKJp/50reKFTObrM8ffkgQ1osXdUKghPRBG +Wh7QKRVQVB4WrUe2ymEie1QbUOGk4InLgqaCtXOK7hg5tCiYjmRfmRowLm21jkjBZWO SwkoydmB86T0HQYbIkVYrh1K2QEHCgHYIxVrKNvwApyyWDNsqvNOI1r/q+81GWCW7Sd7 5Q18jKxobVbAcVMAoL2VaANXoqZagSiZ68HruLQ+qvLvwIHajjhpIzOpSMJWmB1or+/O ENqg== X-Gm-Message-State: AOJu0YzGGgJGl7Leg//9cLhJcLZjEj4Q97FC++D4kR03F+g5x67MtSf8 xtsQyS9ifTxFd0BencWkHEurhu+kngDK5u5LV3SlyEB8oZdFLf1LOoEOK2uVAlZoUBQ1ZL9zrVc KmaU= X-Google-Smtp-Source: AGHT+IGy2AJAnVu4G6senu/LlX8um29g5SFmxnQTYLikCbUH7tdx7EE1sPeIriyMDiWFffZmxzWaYg== X-Received: by 2002:a05:620a:284c:b0:794:bb6b:25db with SMTP id af79cd13be357-794e9dc5c41mr71496485a.33.1717023324780; Wed, 29 May 2024 15:55:24 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id af79cd13be357-794abcdbb69sm508975685a.67.2024.05.29.15.55.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 15:55:24 -0700 (PDT) Date: Wed, 29 May 2024 18:55:23 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v2 2/8] midx-write.c: reduce argument count for `get_sorted_entries()` Message-ID: <9d422efe6e8eea13f48d3f52e4c69c94f8e72114.1717023301.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `midx-write.c::get_sorted_entries()` is responsible for constructing the array of OIDs from a given list of packs which will comprise the MIDX being written. The singular call-site for this function looks something like: ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr, ctx.preferred_pack_idx); This function has five formal arguments, all of which are members of the shared `struct write_midx_context` used to track various pieces of information about the MIDX being written. The function `get_sorted_entries()` dates back to fe1ed56f5e4 (midx: sort and deduplicate objects from packfiles, 2018-07-12), which came shortly after 396f257018a (multi-pack-index: read packfile list, 2018-07-12). The latter patch introduced the `pack_list` structure, which was a precursor to the structure we now know as `write_midx_context` (c.f. 577dc49696a (midx: rename pack_info to write_midx_context, 2021-02-18)). At the time, `get_sorted_entries()` likely could have used the pack_list structure introduced earlier in 396f257018a, but understandably did not since the structure only contained three fields (only two of which were relevant to `get_sorted_entries()`) at the time. Simplify the declaration of this function by taking a single pointer to the whole `struct write_midx_context` instead of various members within it. Since this function is now computing the entire result (populating both `ctx->entries`, and `ctx->entries_nr`), rename it to something that doesn't start with "get_" to make clear that this function has a side-effect. Signed-off-by: Taylor Blau --- midx-write.c | 45 +++++++++++++++++++-------------------------- 1 file changed, 19 insertions(+), 26 deletions(-) diff --git a/midx-write.c b/midx-write.c index 591b79bcbf..15965ceb70 100644 --- a/midx-write.c +++ b/midx-write.c @@ -299,21 +299,16 @@ static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout, * Copy only the de-duplicated entries (selected by most-recent modified time * of a packfile containing the object). */ -static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, - struct pack_info *info, - uint32_t nr_packs, - size_t *nr_objects, - int preferred_pack) +static void compute_sorted_entries(struct write_midx_context *ctx) { uint32_t cur_fanout, cur_pack, cur_object; size_t alloc_objects, total_objects = 0; struct midx_fanout fanout = { 0 }; - struct pack_midx_entry *deduplicated_entries = NULL; - uint32_t start_pack = m ? m->num_packs : 0; + uint32_t start_pack = ctx->m ? ctx->m->num_packs : 0; - for (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) + for (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++) total_objects = st_add(total_objects, - info[cur_pack].p->num_objects); + ctx->info[cur_pack].p->num_objects); /* * As we de-duplicate by fanout value, we expect the fanout @@ -323,26 +318,26 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, alloc_objects = fanout.alloc = total_objects > 3200 ? total_objects / 200 : 16; ALLOC_ARRAY(fanout.entries, fanout.alloc); - ALLOC_ARRAY(deduplicated_entries, alloc_objects); - *nr_objects = 0; + ALLOC_ARRAY(ctx->entries, alloc_objects); + ctx->entries_nr = 0; for (cur_fanout = 0; cur_fanout < 256; cur_fanout++) { fanout.nr = 0; - if (m) - midx_fanout_add_midx_fanout(&fanout, m, cur_fanout, - preferred_pack); + if (ctx->m) + midx_fanout_add_midx_fanout(&fanout, ctx->m, cur_fanout, + ctx->preferred_pack_idx); - for (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) { - int preferred = cur_pack == preferred_pack; + for (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++) { + int preferred = cur_pack == ctx->preferred_pack_idx; midx_fanout_add_pack_fanout(&fanout, - info, cur_pack, + ctx->info, cur_pack, preferred, cur_fanout); } - if (-1 < preferred_pack && preferred_pack < start_pack) - midx_fanout_add_pack_fanout(&fanout, info, - preferred_pack, 1, + if (-1 < ctx->preferred_pack_idx && ctx->preferred_pack_idx < start_pack) + midx_fanout_add_pack_fanout(&fanout, ctx->info, + ctx->preferred_pack_idx, 1, cur_fanout); midx_fanout_sort(&fanout); @@ -356,17 +351,16 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, &fanout.entries[cur_object].oid)) continue; - ALLOC_GROW(deduplicated_entries, st_add(*nr_objects, 1), + ALLOC_GROW(ctx->entries, st_add(ctx->entries_nr, 1), alloc_objects); - memcpy(&deduplicated_entries[*nr_objects], + memcpy(&ctx->entries[ctx->entries_nr], &fanout.entries[cur_object], sizeof(struct pack_midx_entry)); - (*nr_objects)++; + ctx->entries_nr++; } } free(fanout.entries); - return deduplicated_entries; } static int write_midx_pack_names(struct hashfile *f, void *data) @@ -1060,8 +1054,7 @@ static int write_midx_internal(const char *object_dir, } } - ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr, - ctx.preferred_pack_idx); + compute_sorted_entries(&ctx); ctx.large_offsets_needed = 0; for (i = 0; i < ctx.entries_nr; i++) { From patchwork Wed May 29 22:55:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13679522 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A75DC1CB31B for ; Wed, 29 May 2024 22:55:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023333; cv=none; b=K0MMR7D3Z+W22RqrXHH8rglLKbilVjRUzmdZPKnAX3k8F8LWrWlyp6Myfsyde9mW5rAzZnMOXcBJ1DkfJJPJKmspfppGnXo7Le62PUoyvrjk6greN5YKmBd9ZbbEH5QoqE6J4nA5fI4SpmFofULq9chXsbbo8ep9+CW/nDCL+E0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023333; c=relaxed/simple; bh=UGg7U1T4RfZOkgeOhgOrGv6/wfK1HYUq41dRSyIoQ/o=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=jt9AQldewaO1dCNXno70Pa8nXNDWFRg77maQU3rNP2Z45w6eepkxQAg601kn4MzPF82J+OOpDIRc59g7w6bkaWTcx6SgsL5J0JiuwYg9unxPYDwzsNIgibzFORss9VXLFFuVj59UlQ5iSOI1SWyMxCJDcmR8w/1i7uyGOitapP8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=cey+pUZX; arc=none smtp.client-ip=209.85.160.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="cey+pUZX" Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-43e23ce7b07so1088051cf.1 for ; Wed, 29 May 2024 15:55:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1717023330; x=1717628130; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=gJcXgSFD2N6ZIbBhxeMGvaAbGOu3HrfqZVycpkjFW8g=; b=cey+pUZXcWz/h1M3j9S2lleySxuCUPv+35hrNwF12X6z97hiBE9mtUXZbTzG4zBtMo FIY1baGYA4zCP3zyuXrBWc+yFCGLXXEbX87uGO/8i68RYltJeL67TRzsVxp809KlFyKw fDYJfcATUzj14Bdd/ZvUI7/wB8RzOI2CzoD9y/WRzSlj4aqooUbwAl6t3SvvteXqwhGi +5KkTYkFuy91OHPLyE4nhvk+EqAqRV/5aBAZllG9S2iTBLd8SaKf9fnJ/EitFofwNz9J 9MSW3M42OSgoNQy3dq9JIqzDR4lgjwBVMz2fbC4DIiBuVw4lA5ExJmMhEc62B/ZE+s/D f7TA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717023330; x=1717628130; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=gJcXgSFD2N6ZIbBhxeMGvaAbGOu3HrfqZVycpkjFW8g=; b=BnQTlMMjRQGIBAIzZ+C8kcIIX0mH5mtNnReQ+kpCyss0RykviBt+zwQtIP9ajj4IiS ozmRw5n9rjqEJwencmKHu6wchKC23KWP2IRRGmN9OKNFnM4RmWSc4dUzijbR/T1x8Jkx 1c1e9Q9lzYBBKw3SGJKCzbVEESzjMS0GEFfdLq5wuElOVgIc8KvuKQkhfFu7X5ZsONuL eI0uVuJU+29kaq5waYPCrDMlJA63b/xmyEkLVmduFEbAC+nxCYl4/G7lvhdFsMvbYl1M KFsei3enNrWFOX9AA4NICsfNLSCNEBp3r+EsVHs/LbH+17POqmFZroxnj+VZ3K7coMcv H9gg== X-Gm-Message-State: AOJu0YwM90Ic5kC3EVznvKmR5fDSUlc8Ko7Qb0O1JRGOk1sC89G/JAwN kAalITlvi5pQf1hPQxvPG7FEu/KvNXt1+6Is+Zm8JtTbeA1YUpoAclRu1MKwZrz82d0hUImexyo K0aM= X-Google-Smtp-Source: AGHT+IGB4qErMSv+ELYwQhog3ufp+pxICmuynzhDmG33+nNABnuj0qz6jXeeqrhHWV5Zv83xZ7BZNQ== X-Received: by 2002:a05:622a:1491:b0:439:b41a:56c1 with SMTP id d75a77b69052e-43fe92ccc10mr7807931cf.34.1717023330061; Wed, 29 May 2024 15:55:30 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-43fb17fa357sm59135961cf.45.2024.05.29.15.55.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 15:55:29 -0700 (PDT) Date: Wed, 29 May 2024 18:55:28 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v2 3/8] midx-write.c: pass `start_pack` to `compute_sorted_entries()` Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `compute_sorted_entries()` is broadly responsible for building an array of the objects to be written into a MIDX based on the provided list of packs. If we have loaded an existing MIDX, however, we may not use all of its packs, despite loading them into the ctx->info array. The existing implementation simply skips past the first ctx->m->num_packs (if ctx->m is non-NULL, indicating that we loaded an existing MIDX). This is because we read objects in packs from an existing MIDX via the MIDX itself, rather than from the pack-level fanout to guarantee a de-duplicated result (see: a40498a1265 (midx: use existing midx when writing new one, 2018-07-12)). Future changes (outside the scope of this patch series) to the MIDX code will require us to skip *at most* that number[^1]. We could tag each pack with a bit that indicates the pack's contents should be included in the MIDX. But we can just as easily determine the number of packs to skip by passing in the number of packs we learned about after processing an existing MIDX. [^1]: Kind of. The real number will be bounded by the number of packs in a MIDX layer, and the number of packs in its base layer(s), but that concept hasn't been fully defined yet. Signed-off-by: Taylor Blau --- midx-write.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/midx-write.c b/midx-write.c index 15965ceb70..949a66e973 100644 --- a/midx-write.c +++ b/midx-write.c @@ -299,12 +299,12 @@ static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout, * Copy only the de-duplicated entries (selected by most-recent modified time * of a packfile containing the object). */ -static void compute_sorted_entries(struct write_midx_context *ctx) +static void compute_sorted_entries(struct write_midx_context *ctx, + uint32_t start_pack) { uint32_t cur_fanout, cur_pack, cur_object; size_t alloc_objects, total_objects = 0; struct midx_fanout fanout = { 0 }; - uint32_t start_pack = ctx->m ? ctx->m->num_packs : 0; for (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++) total_objects = st_add(total_objects, @@ -889,7 +889,7 @@ static int write_midx_internal(const char *object_dir, { struct strbuf midx_name = STRBUF_INIT; unsigned char midx_hash[GIT_MAX_RAWSZ]; - uint32_t i; + uint32_t i, start_pack; struct hashfile *f = NULL; struct lock_file lk; struct write_midx_context ctx = { 0 }; @@ -957,6 +957,8 @@ static int write_midx_internal(const char *object_dir, } } + start_pack = ctx.nr; + ctx.pack_paths_checked = 0; if (flags & MIDX_PROGRESS) ctx.progress = start_delayed_progress(_("Adding packfiles to multi-pack-index"), 0); @@ -1054,7 +1056,7 @@ static int write_midx_internal(const char *object_dir, } } - compute_sorted_entries(&ctx); + compute_sorted_entries(&ctx, start_pack); ctx.large_offsets_needed = 0; for (i = 0; i < ctx.entries_nr; i++) { From patchwork Wed May 29 22:55:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13679523 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3B731CB31B for ; Wed, 29 May 2024 22:55:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023337; cv=none; b=BPHAAwGt3so1zT/EWkjbaG1P5BP3mGABoPnk4dwh+qvGF2T/YMtlIqBq3/J+BY+lZ2uTVvJ3zpDDQTSwfwfdKgxbOvfW71awG058K0+GJy4oSZ2wdMxvPYVxTlS3wM6sqjFz8ts+8SnRVGzrXaj6N2eBLzepVF0D6H3Ipt4DiZU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023337; c=relaxed/simple; bh=9fVKigdbb/OOVXYdUeZZ7nJyoZoHb+i+tRG8ZCbLnLg=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=BpRpDEOUMtn97+8FhrYVxro3pyikV0I9gRIGrc9qF3SGsls5UAyLIwR3NQ89YQ6oa/5mFbvIz6n5jJ84b64QPPRLYIA/kMoIZ4n+LI/j3kDGSrR1FO1jprv8vkzw62ipXit9+bCveDQEOjDWzoXwHOE3OF1nawqgEE/mwJkMsQo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=2IALDrdw; arc=none smtp.client-ip=209.85.222.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="2IALDrdw" Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-794b1052194so21205185a.1 for ; Wed, 29 May 2024 15:55:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1717023334; x=1717628134; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=I1ByDU/0X+OFSmFqI3+rjM1YjOApGJwluw7Oy18PYU8=; b=2IALDrdwZQkkF6nB75ZdCNVM/1ZWzoZoP6lRIWKjVRpxQuP3Gqom6EqHrbFumxJ5z+ brFzCwb938dEFWQsBYLwpLvVRL0g3jSpkAPVAktH4YAtfseeR/g0gStLaI7zCT8lDj+q SldZft0JR3//j4X4J/j/7jAN8OwyR1FbipLqOcZ6uQMYlygQbqlJFIEPk9uyTzs3Y/KH 9RNTfItQbLdYz+Kfr8UJE5ayKwIibz9v7ZnqpsxMUN/XEopaiw84yKHViwGyGKCxS81f AxeYCYHW2FGUH/ahcC8U08xP84UhoIJBHiOxyOD4mP1JzORC50yJFK1wrz6ETK/Opouw 7eWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717023334; x=1717628134; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=I1ByDU/0X+OFSmFqI3+rjM1YjOApGJwluw7Oy18PYU8=; b=OooxpsqSl4hJBg3Pa+9pq6Zw4JRfiUBRnPcGwrIWqhUvanYbWIR7Y7vrmJnFEWpQBf V1ztdeQVNVP8tGYrMhWDhtPlIuurZ6gGd0Ga0mluLjsnl1XSZC6osY130PxGiDXYIORp 7L2wuN+6b9snLuIPEDcOXsghyUJFMoeJzKnhdhaaud44P2pMtFerIt9lX3bjrWPjNacW d2sgMYeTviOp3agtybczVWYhCCh5YsVO+iORxOtkwzyLsYEb2388ND6t8laKSsNGP0cV Kwz0EU6t1vnEVHl+OiM12qSClLeP1dUZo8PxXl3MaM34pcsLgcBvtuGGptE8DWMHSTW3 UlFg== X-Gm-Message-State: AOJu0YzJIZpyHQF5TupFbz9f84yV0+KSaws8M14theC5AIg25WWVpts1 r5AOv4BtEZbEs1x5I4bdWfyaHhD04+HpkHLEIVY/9UUc7hKAUVpLySW/e654aA7238EPbq/rCEO DP1Y= X-Google-Smtp-Source: AGHT+IFWgORROs0zvCVuF7wnrV9aIx5hpix0M6ECxR02xk1zYcY08pY7YK9LrB5+/GuqIQXgDmMtxg== X-Received: by 2002:a05:620a:9:b0:794:b4e0:a29a with SMTP id af79cd13be357-794e9e33bdbmr59453485a.63.1717023334362; Wed, 29 May 2024 15:55:34 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id af79cd13be357-794abd06397sm508864185a.96.2024.05.29.15.55.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 15:55:33 -0700 (PDT) Date: Wed, 29 May 2024 18:55:32 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v2 4/8] midx-write.c: extract `should_include_pack()` Message-ID: <9cd92574659d49b5365b97cc2352b4cc8c09959d.1717023301.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The add_pack_to_midx() callback used via for_each_file_in_pack_dir() is used to add packs with .idx files to the MIDX being written. Within this function, we have a pair of checks that discards packs which: - appear in an existing MIDX, if we successfully read an existing MIDX from disk - or, appear in the "to_include" list, if invoking the MIDX write machinery with the `--stdin-packs` command-line argument. A future commit will want to call a slight variant of these checks from the code that reuses all packs from an existing MIDX, as well as the current location via add_pack_to_midx(). The latter will be modified in subsequent commits to only reuse packs which appear in the to_include list, if one was given. Prepare for that step by extracting these checks as a subroutine that may be called from both places. Signed-off-by: Taylor Blau --- midx-write.c | 48 ++++++++++++++++++++++++++++-------------------- 1 file changed, 28 insertions(+), 20 deletions(-) diff --git a/midx-write.c b/midx-write.c index 949a66e973..a80709726a 100644 --- a/midx-write.c +++ b/midx-write.c @@ -100,6 +100,32 @@ struct write_midx_context { struct string_list *to_include; }; +static int should_include_pack(const struct write_midx_context *ctx, + const char *file_name) +{ + /* + * Note that at most one of ctx->m and ctx->to_include are set, + * so we are testing midx_contains_pack() and + * string_list_has_string() independently (guarded by the + * appropriate NULL checks). + * + * We could support passing to_include while reusing an existing + * MIDX, but don't currently since the reuse process drags + * forward all packs from an existing MIDX (without checking + * whether or not they appear in the to_include list). + * + * If we added support for that, these next two conditional + * should be performed independently (likely checking + * to_include before the existing MIDX). + */ + if (ctx->m && midx_contains_pack(ctx->m, file_name)) + return 0; + else if (ctx->to_include && + !string_list_has_string(ctx->to_include, file_name)) + return 0; + return 1; +} + static void add_pack_to_midx(const char *full_path, size_t full_path_len, const char *file_name, void *data) { @@ -108,29 +134,11 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, if (ends_with(file_name, ".idx")) { display_progress(ctx->progress, ++ctx->pack_paths_checked); - /* - * Note that at most one of ctx->m and ctx->to_include are set, - * so we are testing midx_contains_pack() and - * string_list_has_string() independently (guarded by the - * appropriate NULL checks). - * - * We could support passing to_include while reusing an existing - * MIDX, but don't currently since the reuse process drags - * forward all packs from an existing MIDX (without checking - * whether or not they appear in the to_include list). - * - * If we added support for that, these next two conditional - * should be performed independently (likely checking - * to_include before the existing MIDX). - */ - if (ctx->m && midx_contains_pack(ctx->m, file_name)) - return; - else if (ctx->to_include && - !string_list_has_string(ctx->to_include, file_name)) + + if (!should_include_pack(ctx, file_name)) return; ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); - p = add_packed_git(full_path, full_path_len, 0); if (!p) { warning(_("failed to add packfile '%s'"), From patchwork Wed May 29 22:55:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13679524 Received: from mail-qv1-f47.google.com (mail-qv1-f47.google.com [209.85.219.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23F101CB31B for ; Wed, 29 May 2024 22:55:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023340; cv=none; b=MJNdrQK0Ap0tceGeV44OuEpOAa1B84F1ZPD7TsS9qaFvltZkdYal+yKl+ucDARV9KMUNUHvVdOeLw5kHb8ZIiojyLcUI02aI6FwpRehcn8r2MOADY0fra7Y+vbpOAjuZLji4V1zCP36y/zOZJgqnNal740wEVoo31HapwmGxcEA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023340; c=relaxed/simple; bh=OH+ZXOWuaWbHXLfzLFS4LJf/1r9XV3xOjR+8f4pdeGk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=puU2LABM3DBcNCIm+lQXQHySpWdyiC8dzwd/sZ90AdT1N15YSjgCYWUuGaWWLupTt20o3CR8qvzNbvQajXID5NMXl+aXHo/zJo1bNj9nJncCZV8RxWC+nvsFNS/vA9QNPnK7y+aadZxmq/oSoY9OnDdJ+KsjqiPKUnzbhbfGoFE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=OuJOhNN5; arc=none smtp.client-ip=209.85.219.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="OuJOhNN5" Received: by mail-qv1-f47.google.com with SMTP id 6a1803df08f44-6ab9e011f62so5036996d6.0 for ; Wed, 29 May 2024 15:55:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1717023338; x=1717628138; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=Kjazu+Ufq2AIDDiAtTjbXZ571NaUKcrxOM7b/1ju0f4=; b=OuJOhNN5Tm+Y3Gy8YIVuLMTJJ4LFu0Sa505x5guyRchDB8pHrmA+51SzCRpY1/Kr/f qmwUkM+VHwu4nPmSOtwr7ssWk0U06bBg+mZARx3JPiBFEmfn2lVw7HAmET8YG+KUgkSF zFkmsdowFlq7IJQIZmxMQL3NGnxXEeiRmrXUDrogperzl/55hV265Bm/m3vw7l/rWtmP LUqwJMTPGzH2d6KNxAiUVnzs6lSw7+pBTKRiaE66Jjg0jGBlsVXQCIKILY+x++fbxmRx eHLLDMo1GxsNq48XR9Mt8CIEUFeJxBA0wnDdriG3T3oax8XVtIRYe8JksUl7YCaeDxHF s4MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717023338; x=1717628138; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Kjazu+Ufq2AIDDiAtTjbXZ571NaUKcrxOM7b/1ju0f4=; b=mZFyOWtBnbDlroVFz8GsNmdl+GXlFA6Gy9oISCkZ15rNMPnKV96P/bh/ywDtocr+wn MqaTCfNaQd+cKhjuLvvnKxXaVDAKmhwu+dmiVbj1xg6vVmzR5ddjRPD+641P0cSUpqhU 8kfNi/6YDTJBXr+zh6vzfmdYZvUfYr+b9K1fysQbPBcHI2s0ho7zz+fuZ2i/5oL1eqGw xo6Q8JxIpKI1pfd+DEyjtIf11rElAg1V2G93+cFFIalJgi8U/y8v01rb8OaSKUl8jox7 2QsOxho65+95ZKWHiTSmx6ovx25mZAagiPwaHVDfbNcvwUQ37x6gLeAyDXVyo5fZMIyE /XUQ== X-Gm-Message-State: AOJu0Yy324ti6E2dF7AD3g1qDz9v1tHSXXhWr0Y537+qVKcC6pCTZw9P 2zN1qWwr7wcPKsUW81dgpfLW39aTRD9PB/snuzBTcJqWvbuRv8arv0b4J60xWcx2sEfiPwFWlIc 41Ms= X-Google-Smtp-Source: AGHT+IF5sCULN1LiyDGnjfjQlY7uxfjm8qZzGbOTKRXnGbTzeJ/efDymRmmJAzYWAt46tM25PygNxQ== X-Received: by 2002:a0c:c308:0:b0:6ae:ff6:4d17 with SMTP id 6a1803df08f44-6ae0ff64f70mr2145656d6.32.1717023337651; Wed, 29 May 2024 15:55:37 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6ae09b63a84sm3400826d6.73.2024.05.29.15.55.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 15:55:37 -0700 (PDT) Date: Wed, 29 May 2024 18:55:36 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v2 5/8] midx-write.c: extract `fill_packs_from_midx()` Message-ID: <1bb890e87cf6db9e464d7753e781a9e5d70e0b12.1717023301.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When write_midx_internal() loads an existing MIDX, all packs are copied forward into the new MIDX. Improve the readability of write_midx_internal() by extracting this functionality out into a separate function. Signed-off-by: Taylor Blau --- midx-write.c | 68 +++++++++++++++++++++++++++++----------------------- 1 file changed, 38 insertions(+), 30 deletions(-) diff --git a/midx-write.c b/midx-write.c index a80709726a..c93e1ae305 100644 --- a/midx-write.c +++ b/midx-write.c @@ -888,6 +888,40 @@ static struct multi_pack_index *lookup_multi_pack_index(struct repository *r, return result; } +static int fill_packs_from_midx(struct write_midx_context *ctx, + const char *preferred_pack_name, uint32_t flags) +{ + uint32_t i; + + for (i = 0; i < ctx->m->num_packs; i++) { + ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); + + if (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) { + /* + * If generating a reverse index, need to have + * packed_git's loaded to compare their + * mtimes and object count. + * + * + * If a preferred pack is specified, need to + * have packed_git's loaded to ensure the chosen + * preferred pack has a non-zero object count. + */ + if (prepare_midx_pack(the_repository, ctx->m, i)) + return error(_("could not load pack")); + + if (open_pack_index(ctx->m->packs[i])) + die(_("could not open index for %s"), + ctx->m->packs[i]->pack_name); + } + + fill_pack_info(&ctx->info[ctx->nr++], ctx->m->packs[i], + ctx->m->pack_names[i], i); + } + + return 0; +} + static int write_midx_internal(const char *object_dir, struct string_list *packs_to_include, struct string_list *packs_to_drop, @@ -933,36 +967,10 @@ static int write_midx_internal(const char *object_dir, ctx.info = NULL; ALLOC_ARRAY(ctx.info, ctx.alloc); - if (ctx.m) { - for (i = 0; i < ctx.m->num_packs; i++) { - ALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc); - - if (flags & MIDX_WRITE_REV_INDEX || - preferred_pack_name) { - /* - * If generating a reverse index, need to have - * packed_git's loaded to compare their - * mtimes and object count. - * - * If a preferred pack is specified, - * need to have packed_git's loaded to - * ensure the chosen preferred pack has - * a non-zero object count. - */ - if (prepare_midx_pack(the_repository, ctx.m, i)) { - error(_("could not load pack")); - result = 1; - goto cleanup; - } - - if (open_pack_index(ctx.m->packs[i])) - die(_("could not open index for %s"), - ctx.m->packs[i]->pack_name); - } - - fill_pack_info(&ctx.info[ctx.nr++], ctx.m->packs[i], - ctx.m->pack_names[i], i); - } + if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name, + flags) < 0) { + result = 1; + goto cleanup; } start_pack = ctx.nr; From patchwork Wed May 29 22:55:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13679525 Received: from mail-yb1-f182.google.com (mail-yb1-f182.google.com [209.85.219.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83C671CB33A for ; Wed, 29 May 2024 22:55:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023344; cv=none; b=Kbq2llZa9iNf1yhT3KJ8yx2lnfMOpfm7ew/hEYthUuaKmw5+q2PJNDPc625GvIkLf/wPTSVYrKV2+LDssurwIo9vlZo4/t8Sq41BjH1ejWPNvP+03CY3RwwZf/wACfJOCuzu+TbY3L5IwC/BQf3uVuMEvx+X2+gE6w1Unog4MOc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023344; c=relaxed/simple; bh=vAS1q+82/ZYTptq1S/8znx/mkne8PQm2BQv+RJZt10o=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=t+g+IZTMcMG21vJyED/ocMmPVyal/xgk3eNxCcm6kg7MgzHswjWFtCOhnGal2UZKlKvKTdHNNc8mIHnakKFcrtbj0wQc0vQ/uqBxvKBgBazlRhw0G9lHLwsMkqt7WMQbPyZF8frKMb40LhJ6kn2sVRot/p8ApJ95JOY2tVfXteQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=g9DjVbTQ; arc=none smtp.client-ip=209.85.219.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="g9DjVbTQ" Received: by mail-yb1-f182.google.com with SMTP id 3f1490d57ef6-dfa48f505a3so246401276.1 for ; Wed, 29 May 2024 15:55:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1717023341; x=1717628141; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=2qcWRfwRHol7Acenxsrnyfd+BOEBQe5/tlzf2MKi+KM=; b=g9DjVbTQFHgS0LPHN9RNk93jW7JVeYpaO/bMdyWgyxzXaQFhN0FTkwJXTZb0j0HcIv 6sYGItORqfH0mjTzusNBorSVIQQVkyDcv5Yxnunmfu0HDBAZsMlVUYuWTrmdGS5pjRjF DwCeCf/g9M6uoaVxnf2zj/5qfhfCUzSHkWo1Cb+TfwfJsulRWyjSTdi0Zh2iEpjjJQWr FbhijC/GAB+kmrX2yeZ5H6pmoaIJFjD3/eqaLCCSiXxbPAgpAWkW2te/NgcWLFrl66wV +rqtXjXIWaWUArwrgQyl3Ducg8z29gUsZ9c1s4KibvdnVZ8W6c6oVMoQA5M09Tyt5qxi PcSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717023341; x=1717628141; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=2qcWRfwRHol7Acenxsrnyfd+BOEBQe5/tlzf2MKi+KM=; b=Tz7cadp6Qrl2IY4siOUCfiTEwqDFkUPWW6SDIh8Ca5G/S+nk+Bzkk14ANR/4q3LqZO VTexeRJ8a77l3WCPGMSzeJT9HI5U2uDQxk1oa01qZ0Y26oAPCkAF3ejgQTBTvFg4futj qtGJyEFXoNiEo3OksEyliTDwYdz+DeNN0J5jgSDugCwE9VvFG5WPpAPjHTBNAVAkE0Rp zwGQCTv/qO6GaWVimvDWmN4T0mEHIt+JPJk9bmYPvVh/msdFbhC1yW9mXAV5KxUW+VVv yvUfPDyUA5IND2IRVji85G4G9nfUyH449LNRXp9k2UxuicOKcnbCt8I9wAVmhgf9AwqS szZQ== X-Gm-Message-State: AOJu0YzNa1QHkpW39ENv0j5gNSt2GbnrEL5wqSgsPTuNoMgdsyyW8Z/4 pfxNBhmRP6zeY9gyqSKpqvCM8M2opsjbZlL15caLRovB/pQnjHOF3MkpxuC6XpaWQ8n59Ig+ZV8 EH6Y= X-Google-Smtp-Source: AGHT+IHWfjq9hT5PyNgrGfl3DScLPGR9mjcPBVwEEPMYk/CdlFW4RgmZnGcNVwxZCjdRH3Z01gAMRg== X-Received: by 2002:a25:d6d6:0:b0:df4:e882:2808 with SMTP id 3f1490d57ef6-dfa5a7bab7fmr590731276.56.1717023341020; Wed, 29 May 2024 15:55:41 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id af79cd13be357-794abd30371sm510875085a.104.2024.05.29.15.55.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 15:55:40 -0700 (PDT) Date: Wed, 29 May 2024 18:55:39 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v2 6/8] midx-write.c: support reading an existing MIDX with `packs_to_include` Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Avoid unconditionally copying all packs from an existing MIDX into a new MIDX by checking that packs added via `fill_packs_from_midx()` don't appear in the `to_include` set, if one was provided. Do so by calling `should_include_pack()` from both `add_pack_to_midx()` and `fill_packs_from_midx()`. In order to make this work, teach `should_include_pack()` a new "exclude_from_midx" parameter, which allows skipping the first check. This is done so that the caller in `fill_packs_from_midx()` doesn't reject all of the packs it provided since they appear in an existing MIDX by definition. The sum total of this change is that we are now able to read and reference objects in an existing MIDX even when given a non-NULL `packs_to_include`. This is a prerequisite step for incremental MIDXs, which need to load any existing MIDX (if one is present) in order to determine whether or not an object already appears in an earlier portion of the MIDX to avoid duplicating it across multiple portions. Signed-off-by: Taylor Blau --- midx-write.c | 42 +++++++++++------------------------------- 1 file changed, 11 insertions(+), 31 deletions(-) diff --git a/midx-write.c b/midx-write.c index c93e1ae305..8b96c566e7 100644 --- a/midx-write.c +++ b/midx-write.c @@ -101,27 +101,13 @@ struct write_midx_context { }; static int should_include_pack(const struct write_midx_context *ctx, - const char *file_name) + const char *file_name, + int exclude_from_midx) { - /* - * Note that at most one of ctx->m and ctx->to_include are set, - * so we are testing midx_contains_pack() and - * string_list_has_string() independently (guarded by the - * appropriate NULL checks). - * - * We could support passing to_include while reusing an existing - * MIDX, but don't currently since the reuse process drags - * forward all packs from an existing MIDX (without checking - * whether or not they appear in the to_include list). - * - * If we added support for that, these next two conditional - * should be performed independently (likely checking - * to_include before the existing MIDX). - */ - if (ctx->m && midx_contains_pack(ctx->m, file_name)) + if (exclude_from_midx && ctx->m && midx_contains_pack(ctx->m, file_name)) return 0; - else if (ctx->to_include && - !string_list_has_string(ctx->to_include, file_name)) + if (ctx->to_include && !string_list_has_string(ctx->to_include, + file_name)) return 0; return 1; } @@ -135,7 +121,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, if (ends_with(file_name, ".idx")) { display_progress(ctx->progress, ++ctx->pack_paths_checked); - if (!should_include_pack(ctx, file_name)) + if (!should_include_pack(ctx, file_name, 1)) return; ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); @@ -894,6 +880,9 @@ static int fill_packs_from_midx(struct write_midx_context *ctx, uint32_t i; for (i = 0; i < ctx->m->num_packs; i++) { + if (!should_include_pack(ctx, ctx->m->pack_names[i], 0)) + continue; + ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); if (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) { @@ -948,15 +937,7 @@ static int write_midx_internal(const char *object_dir, die_errno(_("unable to create leading directories of %s"), midx_name.buf); - if (!packs_to_include) { - /* - * Only reference an existing MIDX when not filtering which - * packs to include, since all packs and objects are copied - * blindly from an existing MIDX if one is present. - */ - ctx.m = lookup_multi_pack_index(the_repository, object_dir); - } - + ctx.m = lookup_multi_pack_index(the_repository, object_dir); if (ctx.m && !midx_checksum_valid(ctx.m)) { warning(_("ignoring existing multi-pack-index; checksum mismatch")); ctx.m = NULL; @@ -965,6 +946,7 @@ static int write_midx_internal(const char *object_dir, ctx.nr = 0; ctx.alloc = ctx.m ? ctx.m->num_packs : 16; ctx.info = NULL; + ctx.to_include = packs_to_include; ALLOC_ARRAY(ctx.info, ctx.alloc); if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name, @@ -981,8 +963,6 @@ static int write_midx_internal(const char *object_dir, else ctx.progress = NULL; - ctx.to_include = packs_to_include; - for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx); stop_progress(&ctx.progress); From patchwork Wed May 29 22:55:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13679526 Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A5E061CB33A for ; Wed, 29 May 2024 22:55:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023347; cv=none; b=sGdAC68WerE7sY4zVDwg4AeapHvUul1b6x9K5beCXHPRKoLwvauiuLKiOXKLDJaHhDTWui+FoXDmzBNsd4RhfLLqXY8xZ6cG+FkXWQIGS8CICxPhwVquUCvZOBUvOaLdTqZ/kD8+jJHjtxT80xa1QpEvxCsiALxMDjk0NXLJP8A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023347; c=relaxed/simple; bh=ukeADAancgq8IGMI1ojGF+p3sxxbcBVYk43B3WJytwY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=HgLbIMrCsT5Ij1nd2YkvYijCXBALTUbimglM0i0taorq7mQxEs5oHoi2lbTeY0R1abfcLn6cKlxlWzXYv4DgxUUb/WBHilFaAjWZ9b/tQqkiY2z1w0qNeFFyrx9uGBxHY/Z+pmABkQ3b+fhwyLz5xXDcUxMLeGEb5+/6KdXlpfs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=lIPehZ55; arc=none smtp.client-ip=209.85.167.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="lIPehZ55" Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-3d04c170118so74827b6e.1 for ; Wed, 29 May 2024 15:55:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1717023344; x=1717628144; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=p4b4q/p62hnABvOLKsrxKFEtyh9sWfsD48Vnf+IZx3I=; b=lIPehZ55+nMuAMsP6uFEyBTfP30ZZlUnx6YtAchIZwAuGM30LwRmR6EVZ41tvCQ1ea RyWZ/mnyTDodpMxh46DzRLXJDiA7ZRoT62xlCk6HOSosNlku1o1S627IZtDqU9dkYTIG +FPauOOZEkB+mp+QL/JivJhQ50RJDW+/LnWQQepPnY/UDwdbi82rp2AJ7M5Ljj0AeJ3f V1AH8E0CAW0tZTC8n7ntAEeg03WNFdPSYeZRJFGNPIH3iBtKmVT9BwR0lVaMawcoeXNB wdFPF1qjaBX+laLJ57VrQe5e13Ovq0Rx7ZlYXFRLYzGscZljtQfBMEI7nNsTdW6/GyS0 LQyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717023344; x=1717628144; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=p4b4q/p62hnABvOLKsrxKFEtyh9sWfsD48Vnf+IZx3I=; b=RiZ6owjtn9pQkwgZvx0QSHCXodJI4/ZteScfTxfnSq+KmkHPu5euFhNrKxrmdFEbsj cckRe/dNX+a/AF7cbN00ePQ7unrdKZF1euNACjJu288WLtLcc/t9bCe3pxHA6Q1Rnj05 JHDmsJNda2GfL6R1FXl5mXbcfAtfDxW7UEj5w5cHRa/sFLGpCYFLvF5rbRo6NTH5YDWb yYvTQK1Q9dOqyRp4HjyTIxlfJwxEJxsCcAQwE72SNaIkOD10Si5hil6wu3eytwl7+8S5 VS8/kFt3viefAPQPZX8vKxPFv/7e9NH1uqeNxs0te0eVnCeJs4Q8gxj9fuuHoJlssirs MdOQ== X-Gm-Message-State: AOJu0Yw8cUAyQOFbIGzK3MNmZtawbt5V50LPKcUqvyvnzgcMbiy3Kj5a rGTD2cpdmWxN5vK4HlRdNhgYGaM9hVkJGHWAihH+lxqMR5LLBtA2WSrZHBPt42vCOyrOx+HfXBO zDZY= X-Google-Smtp-Source: AGHT+IGn1IZeiEBLx8HW50qItdMvFTE0yeEY/srLA0sdFEm9RljPimVklH63aaSss+yRAT7s1XHCaw== X-Received: by 2002:a05:6808:2014:b0:3c8:69b6:9b7e with SMTP id 5614622812f47-3d1dcc9a0b1mr513727b6e.7.1717023344271; Wed, 29 May 2024 15:55:44 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6ad7308a3ffsm43643606d6.106.2024.05.29.15.55.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 15:55:43 -0700 (PDT) Date: Wed, 29 May 2024 18:55:42 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v2 7/8] midx: replace `get_midx_rev_filename()` with a generic helper Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Commit f894081deae (pack-revindex: read multi-pack reverse indexes, 2021-03-30) introduced the `get_midx_rev_filename()` helper (later modified by commit 60980aed786 (midx.c: write MIDX filenames to strbuf, 2021-10-26)). This function returns the location of the classic ".rev" files we used to write for MIDXs (prior to 95e8383bac1 (midx.c: make changing the preferred pack safe, 2022-01-25)), which is always of the form: $GIT_DIR/objects/pack/multi-pack-index-$HASH.rev Replace this function with a generic helper that populates a strbuf with the above form, replacing the ".rev" extension with a caller-provided argument. This will allow us to remove a similarly-defined function in the pack-bitmap code (used to determine the location of a MIDX .bitmap file) by reimplementing it in terms of `get_midx_filename_ext()`. Signed-off-by: Taylor Blau --- midx.c | 14 ++++++++------ midx.h | 6 +++++- pack-revindex.c | 3 ++- 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/midx.c b/midx.c index 6f07de3688..bc4797196f 100644 --- a/midx.c +++ b/midx.c @@ -24,14 +24,16 @@ const unsigned char *get_midx_checksum(struct multi_pack_index *m) } void get_midx_filename(struct strbuf *out, const char *object_dir) +{ + get_midx_filename_ext(out, object_dir, NULL, NULL); +} + +void get_midx_filename_ext(struct strbuf *out, const char *object_dir, + const unsigned char *hash, const char *ext) { strbuf_addf(out, "%s/pack/multi-pack-index", object_dir); -} - -void get_midx_rev_filename(struct strbuf *out, struct multi_pack_index *m) -{ - get_midx_filename(out, m->object_dir); - strbuf_addf(out, "-%s.rev", hash_to_hex(get_midx_checksum(m))); + if (ext) + strbuf_addf(out, "-%s.%s", hash_to_hex(hash), ext); } static int midx_read_oid_fanout(const unsigned char *chunk_start, diff --git a/midx.h b/midx.h index dc477dff44..8554f2d616 100644 --- a/midx.h +++ b/midx.h @@ -74,9 +74,13 @@ struct multi_pack_index { #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3) #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4) +#define MIDX_EXT_REV "rev" +#define MIDX_EXT_BITMAP "bitmap" + const unsigned char *get_midx_checksum(struct multi_pack_index *m); void get_midx_filename(struct strbuf *out, const char *object_dir); -void get_midx_rev_filename(struct strbuf *out, struct multi_pack_index *m); +void get_midx_filename_ext(struct strbuf *out, const char *object_dir, + const unsigned char *hash, const char *ext); struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local); int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id); diff --git a/pack-revindex.c b/pack-revindex.c index a7624d8be8..fc63aa76a2 100644 --- a/pack-revindex.c +++ b/pack-revindex.c @@ -381,7 +381,8 @@ int load_midx_revindex(struct multi_pack_index *m) trace2_data_string("load_midx_revindex", the_repository, "source", "rev"); - get_midx_rev_filename(&revindex_name, m); + get_midx_filename_ext(&revindex_name, m->object_dir, + get_midx_checksum(m), MIDX_EXT_REV); ret = load_revindex_from_disk(revindex_name.buf, m->num_objects, From patchwork Wed May 29 22:55:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13679527 Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 14C1C1CB329 for ; Wed, 29 May 2024 22:55:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023350; cv=none; b=mIm92Cov+0hQbTh4eeTzUHTKWe3L1+y/KmXoIUOJSQ3MGAnlniBrAFoZIxwKAsz3CfVpqVmVDoSRSa2oo3GaZfKcG5MmsNnyy6T7OxzenQoQlmxvCofhkQJw7kD8nr3OKVbxHU19i8c5xbwCLEyD+DsxFEIqmDmiiiWQVtyHUSc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717023350; c=relaxed/simple; bh=BKHq+n632laculPR0CzkRvZOupS2+Ww6p3KuxVcr6Xc=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=MHVbOPJ0XqQDtLtWyfWthM5tMlgNoM86AHdzfSHT5MxuoZkh6ajeYqqY+Y/g+HePNnQJwsr4e4KI7oVB6MwDhZMfK1FPvQphwHUSOA1NJnEJZkL4nCeALfKW6v1ENqfZbWik3CotcOKEkGDu8Gx3Nij6h4eMbPTLKypBv7psAK4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=EQGJh3AL; arc=none smtp.client-ip=209.85.167.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="EQGJh3AL" Received: by mail-oi1-f182.google.com with SMTP id 5614622812f47-3d1dbd01529so117673b6e.2 for ; Wed, 29 May 2024 15:55:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1717023347; x=1717628147; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=uWawbqlIMKiW+bBSk3AyAis5ic0hwBHJQ6eheqV/IQI=; b=EQGJh3AL4p9sv87o+Pf1+h8GYqRGrRsV3776rHhif7ueDSnfGoE8vQVD6jh5yAPAUT UAPD6KXdOXX6DEs4dnwcR8b4yX81r/zth+8wP2sTitDMtq70wM9QiUSLKgC5BJLeBNYX oCbSB+v4408AoqG//T6g5u73TdJCKcaJwak7nkvb3+o7HXEhv+oAJD5PWdFcQADt45Al oRvO5dk2JH05/wQhMMK9CfZfd2pgrXmnQiTYoZx/WicO/NcArBs+utFiUOPFLmllQnMU P8GU7bsib95Ve5pRp0X/1G4krzoiqwEQy7vc7XHaZPwQSSkDW90P9Fyjg7yrgdXN9sZs h/vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717023347; x=1717628147; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=uWawbqlIMKiW+bBSk3AyAis5ic0hwBHJQ6eheqV/IQI=; b=idFQK9InsWth7RzCuT/HV3In5u3fEY43esSf7OeLT5wjB4XYdcKucCXUPtefvD/2fE KMvJ8EFUjTvbKI6Hlkrdj/cFjDUOR4rTf2kPxm86yjS4iuMKLstzmaKfQmpML+sSKXNW 0gFNsLty8ZiOO6liQzpgbXHD8No0Q1s9gZk//9th9RDJQEKwu0rx+rEFPI0rjD8kmZ4W 6be+cDfY/qczWXFE1OGiXkiC5ACmk9BV8nJcLZNHoXV+w+kv2S/59bP5KUWHd2OuBfJl VSn4/ZbCC9q1cNGDHGNp+m0ZP0ojHod3ugF1VTi9Lt7KuAfCd/kDwx2dIHHiGnpqYwix 8O8Q== X-Gm-Message-State: AOJu0Yw8EYA9cJkgSf/hsdVjbvveVS9AWAtoR1Febas4ndmFa8ZQwtqc myjNhdTQe6NyMztpsqqeMZhy+0yTid2JglwfSGdS3vNhpiR7xlpKd5phpMs3FQInD6iNGXe5SLh CxnI= X-Google-Smtp-Source: AGHT+IERdcaL53bcJMZ/+u7l/av+YcZOEFbxiZ1Irp88su4Ty3uCdfTJZEtQ/49NBkvTJXj8ALPYWg== X-Received: by 2002:a05:6808:1483:b0:3c8:3c10:adff with SMTP id 5614622812f47-3d1dcc91a0cmr515105b6e.9.1717023347506; Wed, 29 May 2024 15:55:47 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6ad92f558adsm20172346d6.21.2024.05.29.15.55.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 15:55:47 -0700 (PDT) Date: Wed, 29 May 2024 18:55:45 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Patrick Steinhardt , Junio C Hamano Subject: [PATCH v2 8/8] pack-bitmap.c: reimplement `midx_bitmap_filename()` with helper Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Now that we have the `get_midx_filename_ext()` helper, we can reimplement the `midx_bitmap_filename()` function in terms of it. Signed-off-by: Taylor Blau --- pack-bitmap.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index 35c5ef9d3c..fe8e8a51d3 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -309,9 +309,8 @@ static int load_bitmap_entries_v1(struct bitmap_index *index) char *midx_bitmap_filename(struct multi_pack_index *midx) { struct strbuf buf = STRBUF_INIT; - - get_midx_filename(&buf, midx->object_dir); - strbuf_addf(&buf, "-%s.bitmap", hash_to_hex(get_midx_checksum(midx))); + get_midx_filename_ext(&buf, midx->object_dir, get_midx_checksum(midx), + MIDX_EXT_BITMAP); return strbuf_detach(&buf, NULL); }