From patchwork Thu Feb 27 18:29:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13995006 Received: from mail-yw1-f176.google.com (mail-yw1-f176.google.com [209.85.128.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C786B20B1E1 for ; Thu, 27 Feb 2025 18:29:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740680972; cv=none; b=SL4m0CHI72QUuPYDWd8mOZ/0avB9Z5OsziErbxp69Oum6g37++FTZMob11os1zbTgAzdQ9iMKt6AX9Eilu86hCcAYutUYCAaGHTDY8atd7FRsRw9aABwFExae3NPjtfJad+mA4gW+Qyr6Dh1MLHuEe2dh9pyk81LJYHKSsB8qhU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740680972; c=relaxed/simple; bh=d4sabs5Ks2PUi+jFHExXZm9STW9zUGZSscwdIFNmBKQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=kq607Vz6e9OVAxOSSpq6NgjvM927hF1Cdt4qQtC3Ts2m3ka18Rf5C1BqgAnHkINPtIuQ4Ke64/9q4J0lonLT4yCdw5zPuVB0m+PLlzeecbuWPkqxDY+nu7at+4I0EIHeIBLsTiNpmoKbfF4v9B3wAoofsjaW31yD+5F7uCxYS9Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=ALpAxsw0; arc=none smtp.client-ip=209.85.128.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="ALpAxsw0" Received: by mail-yw1-f176.google.com with SMTP id 00721157ae682-6f88509dad2so12592037b3.3 for ; Thu, 27 Feb 2025 10:29:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1740680969; x=1741285769; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=LGaep0boiJCzl2bzVSWQZ6g+4TUx8NLYM8XPL8alPAw=; b=ALpAxsw0s3bsoVJK4k9FxX1sv6zDRgYeNwCmG474mBxeIg15sHQklqs8X5pAZwN+EF 8DLeXlWyC/P4n1WZAXAX8T1lZYltLW4Oy4XF3hGHVxmNsrQvpxftXAotv0yyW8PcaOdH tW3Qpv8Bwy/wvc8ecvN4Ds41Gr08WHMZ18mvYTaQuTEReyL8kQk4UfEQT/67m7quyFc2 EUmhzXze8aX84EUgWHUDrCslzel43rDPH1ElJ1oVCzMBQmyT6FkPvToi9NRNJl3FrvKv Hn0FnWUmOPlqMXgel55cFEtaPdmn5TgqcpDaX2KKgHlbBAIJbl4q+BrJLqUt5pOXEGS4 TXAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740680969; x=1741285769; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=LGaep0boiJCzl2bzVSWQZ6g+4TUx8NLYM8XPL8alPAw=; b=U05JciZfsOZVSMrX3nwPlntWc4G2f1ccR40TZdVsvozvNVH9sUDhYabvfAIHEAM8ic 9D3UrY1wRGuf5ASm33aJltywKLu0900lstMoyfFZTeEV4yOR+XtgBuZSYyEe8tXp/v1A 9gC4DCJ19YlLvUpedfIqK+gj/84FAXlCg+l+Vc+zqfB8XwXHpxAllwbKfDZx5NkwBIei JjHazu4Eu0CrpgOlTQJHKg925iN9gZ+ShhqoPl46OYNLWiHFdTMoO/j9UaLF54GA34Rq hFg0ToTANQIrpMVLTUBwnjVa1rgBD4NCgDc259xzSqpyFq9H7TAb0dQSx/ATIKaCgp35 0XZQ== X-Gm-Message-State: AOJu0YzLmdeYWNtsoygiJymjGM5eJREGLaexZocPog1LcZ8M17VomDKO iHlltZICPI1U+K6PHtY9KyqGozcrFJD3tmZAXxz27nGwMTe+rYMEA5K7ARxEputtZgkyfchwx+G D X-Gm-Gg: ASbGncsxAT5EdITCMi1ydgwrskhS8z+ivk7UnJoWsPdc15ce8h8GW3d1iS3DehU/ZYE +9w0GAzM2RI1/KolaDYp93SAbnIQDc65qPXBnpyGUtLDgBYeojk2Vv21OArAy19DRvPIRORSIoN 6iFHndbuLUiTZn4LFxN/E+Zzn3TSwwTxKLStKbuZ9tu3YHh9Z8V/V0tYWtRwQngBtr13N1fhZOm Pnc5i9vq7ZkA09D2lCu4JW1zl36UE+JjggljVsMJ8bCzvzXEios5zD9wnNzAyB8FaAOqDl2Iq8j sONZvtM0wxUbbpxkA4c2GlegoZ/bxU2DyF9DXQ/AE/sAn9Cd/mb6FOoFAAJMgWmGGHAOoYJzGw= = X-Google-Smtp-Source: AGHT+IGVbHjDJIttRmNWGG9XKcAqSoFlhk3k96Wr8Qce4C3suryLzMp/x5XrPxmZrf7tVymfVHBRhQ== X-Received: by 2002:a05:690c:7006:b0:6f9:e45d:8a32 with SMTP id 00721157ae682-6fd4a12e1afmr6861827b3.23.1740680969593; Thu, 27 Feb 2025 10:29:29 -0800 (PST) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id 00721157ae682-6fd3cb7e1f9sm3894007b3.88.2025.02.27.10.29.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Feb 2025 10:29:29 -0800 (PST) Date: Thu, 27 Feb 2025 13:29:28 -0500 From: Taylor Blau To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Elijah Newren Subject: [PATCH 1/2] builtin/repack.c: simplify cruft pack aggregation Message-ID: <8564f98259727225391edcb5ab3b47dd53f00e48.1740680964.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In 37dc6d8104 (builtin/repack.c: implement support for `--max-cruft-size`, 2023-10-02), 'git repack' built on support for multiple cruft packs in Git by instructing 'git pack-objects --cruft' how to aggregate smaller cruft packs up to the provided threshold. The implementation in 37dc6d8104 worked something like the following pseudo-code: total_size = 0; for (p in cruft packs) { if (p->pack_size + total_size < max_size) { total_size += p->pack_size; collapse(p) } else { retain(p); } } The original idea behind this approach was that smaller cruft packs would get combined together until the sum of their sizes was no larger than the given max pack size. There is a much simpler way to achieve this, however, which is to simply combine *all* cruft packs which are smaller than the threshold, regardless of what their sum is. With '--max-pack-size', 'pack-objects' will split out the resulting pack into individual pack(s) if necessary to ensure that the written pack(s) are each no larger than the provided threshold. This yields a slight behavior change, which is reflected in the removed test. Previous to this change, we would aggregate smaller cruft packs first, whereas now we will opportunistically combine as many cruft packs as possible. As as result, that test is no longer relevant, and can be deleted. Signed-off-by: Taylor Blau --- builtin/repack.c | 38 ++----------------------------------- t/t7704-repack-cruft.sh | 42 ----------------------------------------- 2 files changed, 2 insertions(+), 78 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index 75e3752353a..4d83d40f39f 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -1022,29 +1022,13 @@ static int write_filtered_pack(const struct pack_objects_args *args, return finish_pack_objects_cmd(&cmd, names, local); } -static int existing_cruft_pack_cmp(const void *va, const void *vb) -{ - struct packed_git *a = *(struct packed_git **)va; - struct packed_git *b = *(struct packed_git **)vb; - - if (a->pack_size < b->pack_size) - return -1; - if (a->pack_size > b->pack_size) - return 1; - return 0; -} - static void collapse_small_cruft_packs(FILE *in, size_t max_size, struct existing_packs *existing) { - struct packed_git **existing_cruft, *p; + struct packed_git *p; struct strbuf buf = STRBUF_INIT; - size_t total_size = 0; - size_t existing_cruft_nr = 0; size_t i; - ALLOC_ARRAY(existing_cruft, existing->cruft_packs.nr); - for (p = get_all_packs(the_repository); p; p = p->next) { if (!(p->is_cruft && p->pack_local)) continue; @@ -1056,24 +1040,7 @@ static void collapse_small_cruft_packs(FILE *in, size_t max_size, if (!string_list_has_string(&existing->cruft_packs, buf.buf)) continue; - if (existing_cruft_nr >= existing->cruft_packs.nr) - BUG("too many cruft packs (found %"PRIuMAX", but knew " - "of %"PRIuMAX")", - (uintmax_t)existing_cruft_nr + 1, - (uintmax_t)existing->cruft_packs.nr); - existing_cruft[existing_cruft_nr++] = p; - } - - QSORT(existing_cruft, existing_cruft_nr, existing_cruft_pack_cmp); - - for (i = 0; i < existing_cruft_nr; i++) { - size_t proposed; - - p = existing_cruft[i]; - proposed = st_add(total_size, p->pack_size); - - if (proposed <= max_size) { - total_size = proposed; + if (p->pack_size < max_size) { fprintf(in, "-%s\n", pack_basename(p)); } else { retain_cruft_pack(existing, p); @@ -1086,7 +1053,6 @@ static void collapse_small_cruft_packs(FILE *in, size_t max_size, existing->non_kept_packs.items[i].string); strbuf_release(&buf); - free(existing_cruft); } static int write_cruft_pack(const struct pack_objects_args *args, diff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh index 959e6e26488..5a76b541ddd 100755 --- a/t/t7704-repack-cruft.sh +++ b/t/t7704-repack-cruft.sh @@ -194,48 +194,6 @@ test_expect_success '--max-cruft-size combines existing packs when below thresho ) ' -test_expect_success '--max-cruft-size combines smaller packs first' ' - git init max-cruft-size-consume-small && - ( - cd max-cruft-size-consume-small && - - test_commit base && - git repack -ad && - - cruft_foo="$(generate_cruft_pack foo 524288)" && # 0.5 MiB - cruft_bar="$(generate_cruft_pack bar 524288)" && # 0.5 MiB - cruft_baz="$(generate_cruft_pack baz 1048576)" && # 1.0 MiB - cruft_quux="$(generate_cruft_pack quux 1572864)" && # 1.5 MiB - - test-tool pack-mtimes "$(basename $cruft_foo)" >expect.raw && - test-tool pack-mtimes "$(basename $cruft_bar)" >>expect.raw && - sort expect.raw >expect.objects && - - # repacking with `--max-cruft-size=2M` should combine - # both 0.5 MiB packs together, instead of, say, one of - # the 0.5 MiB packs with the 1.0 MiB pack - ls $packdir/pack-*.mtimes | sort >cruft.before && - git repack -d --cruft --max-cruft-size=2M && - ls $packdir/pack-*.mtimes | sort >cruft.after && - - comm -13 cruft.before cruft.after >cruft.new && - comm -23 cruft.before cruft.after >cruft.removed && - - test_line_count = 1 cruft.new && - test_line_count = 2 cruft.removed && - - # the two smaller packs should be rolled up first - printf "%s\n" $cruft_foo $cruft_bar | sort >expect.removed && - test_cmp expect.removed cruft.removed && - - # ...and contain the set of objects rolled up - test-tool pack-mtimes "$(basename $(cat cruft.new))" >actual.raw && - sort actual.raw >actual.objects && - - test_cmp expect.objects actual.objects - ) -' - test_expect_success 'setup --max-cruft-size with freshened objects' ' git init max-cruft-size-freshen && ( From patchwork Thu Feb 27 18:29:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13995007 Received: from mail-yw1-f176.google.com (mail-yw1-f176.google.com [209.85.128.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3ED011E833D for ; Thu, 27 Feb 2025 18:29:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740680976; cv=none; b=SFAAtyKk09D7AGdhJuL4ymTvRaikdgP8miD+fQ0UIowigh+MkkoKWxWm6MNvkP8vzc0B1cj43Vx2fTRlJ3QO3EZJWWD1DjFm6N3Gb19Biygj3oGuOrI8W5pF+q0W2Vc8C9acVwF3s+9b/fe6ZCksGruT4R0KRQRG6/sxdhL4b/k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740680976; c=relaxed/simple; bh=fe27Y2z6dqGnTk4/kj7b3KvIuMllgCCk5dZrO3RUYPg=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Q3TW1MThZIaWfMJIwcq5h53ESmwmoQ88xI85dhePJzeHIRI8Fwap9jHUNzFhbm10HFEfsDUn1blP7X+TOmPgZ94ZntaEsZzO865WgURaz/FP/yQkBIxAtlqJdynU16RjOTjJguE01hya+DET/ZoDHSiYF7Rc9cczlTcQ6TtKqXY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=gKIZcHqw; arc=none smtp.client-ip=209.85.128.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="gKIZcHqw" Received: by mail-yw1-f176.google.com with SMTP id 00721157ae682-6ef60e500d7so11187967b3.0 for ; Thu, 27 Feb 2025 10:29:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1740680973; x=1741285773; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=qiCyUUp8hYeKuLGz8c65JdRkoYi5T2z0mOnSvybiDVI=; b=gKIZcHqw3vA7Cnak3YnLgrC5slCagyCmlhsbAoOeuncmauO/aWTY6QGFnsJvfmZL3Y XD83nmuXSXnmCDtWey80sgD1WECYEyiQ2jSsGYRUxY/PKNMr3wbDI9sklgUzo2qocGX8 5Ius85RwruWHw4oCqatBcA3syNhnbhuvPu225npbhBR4bZbTFdnQDgLHIrGBvcbkWGvT 1N1Do8+OFN8JMPgiQFhK9tKkWUpHCOHFs+mDGvdZWb51PwFsRM3zhHFfbBaBF7HyhMhJ f8RLApbnAwdl8EHEJbys1jYpjG3MCmrTZ8jHGCOL9voggvg21gTvKAo8UUOKaxIp7wfz gBVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740680973; x=1741285773; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=qiCyUUp8hYeKuLGz8c65JdRkoYi5T2z0mOnSvybiDVI=; b=jrVWRPHn4XWa63223soJDuWXd70vhFM7gkrmOccwt3eYOtuyey8A9bQXMX3SwEhJz7 khuCHZ4zzyOwMBZXWHENnN+NbbUyIBg8GKbLgLJItCL/xzizi9SGmACl1nSG+uxN2+gn 6mir1TqeokB1YiQpEzNMZzwNv+FKvJ0wZPxy/P6TYkkHk1lrNlDuloueY8/r6ZA6p2CS UhnvgHIylicELIc/QMiXiD7a4jHdKFuAJuJw+P5M5oyocd0kPloawjzwavCtHlR9F+q9 XgtvGtg0yReB4Rs6N16wC4DuUzmNHrcinPXfRh1+ZOdZcZ86AdZL6RKBMDWr/NF8vum9 33IQ== X-Gm-Message-State: AOJu0YwiYxt6H/9OzzhiURDqpy/71ziJBjL5hN8GhgFay868+g4bL7we z+b1FcDDhIHd9VJqzervJYNLBoHhWUnTmHdgQk/IAtTTNEDrWvEtzZhs5Q/fUEy6TDFmAmGcEEV m X-Gm-Gg: ASbGncvES5fSDGPTHkKSih6oFYTtjEBsd90mLXHwrXFfqkTX+P3Q8il1T0/Kj9Bql+i bBFpkelzLPyNdkq2EWs8WUrNwHlV/k9l81+EAdzRgQLrNfXausxdS8xkBXZOpa1IPD2kYtUA9sr IZ2GMEq5vcN2caoYdgJ+AUdl2gf4cBZ9h+eaibxmFzOAr+MjL6rzxG21xLsWBLDYsY8qf/eSR8G gohYtEskn7KECCcDVy76RMqgOBz+DQ2IXXog+PyV1m5YqbQO5BqvZenkYS+8PIBesA2jJwm3foj Idyqnpy6oAb19kUl8raETo1qemt/bURaZ+mUBycM4gbUtCDphm0KZhbPUiIbPj7eooUYoyS36Q= = X-Google-Smtp-Source: AGHT+IGVXjUMmshOebV5lWxMpp43h+R9oL4a7xg3iA1XzfsC+xZNMFyxXcliPb4SbrAZEKlmW3nSLw== X-Received: by 2002:a05:690c:648a:b0:6fd:47aa:c6ba with SMTP id 00721157ae682-6fd4a090311mr8586807b3.19.1740680972557; Thu, 27 Feb 2025 10:29:32 -0800 (PST) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id 00721157ae682-6fd3cb9dac4sm3911827b3.102.2025.02.27.10.29.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Feb 2025 10:29:32 -0800 (PST) Date: Thu, 27 Feb 2025 13:29:31 -0500 From: Taylor Blau To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Elijah Newren Subject: [PATCH 2/2] builtin/pack-objects.c: freshen objects from existing cruft packs Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Once an object is written into a cruft pack, we can only freshen it by writing a new loose or packed copy of that object with a more recent mtime. Prior to 61568efa95 (builtin/pack-objects.c: support `--max-pack-size` with `--cruft`, 2023-08-28), we typically had at most one cruft pack in a repository at any given time. So freshening unreachable objects was straightforward when already rewriting the cruft pack (and its *.mtimes file). But 61568efa95 changes things: 'pack-objects' now supports writing multiple cruft packs when invoked with `--cruft` and the `--max-pack-size` flag. Cruft packs are rewritten until they reach some size threshold, at which point they are considered "frozen", and will only be modified in a pruning GC, or if the threshold itself is adjusted. However, this process breaks down when we attempt to freshen an object packed in an earlier cruft pack that is larger than the threshold and thus will survive the repack. When this is the case, it is impossible to freshen objects in cruft pack(s) which are larger than the threshold. This is because we avoid writing them in the new cruft pack entirely, for a couple of reasons. 1. When enumerating packed objects via 'add_objects_in_unpacked_packs()' we pass the SKIP_IN_CORE_KEPT_PACKS, which is used to avoid looping over the packs we're going to retain (which are marked as kept in-core by 'read_cruft_objects()'). This means that we will avoid enumerating additional packed copies of objects found in any cruft packs which are larger than the given size threshold. Thus there is no opportunity to call 'create_object_entry()' whatsoever. 2. We likewise will discard the loose copy (if one exists) of any unreachable object packed in a cruft pack that is larger than the threshold. Here our call path is 'add_unreachable_loose_objects()', which uses the 'add_loose_object()' callback. That function will eventually land us in 'want_object_in_pack()' (via 'add_cruft_object_entry()'), and we'll discard the object as it appears in one of the packs which we marked as kept in-core. This means in effect that it is impossible to freshen an unreachable object once it appears in a cruft pack larger than the given threshold. Instead, we should pack an additional copy of an unreachable object we want to freshen even if it appears in a cruft pack, provided that the cruft copy has an mtime which is before the mtime of the copy we are trying to pack/freshen. This is sub-optimal in the sense that it requires keeping an additional copy of unreachable objects upon freshening, but we don't have a better alternative without the ability to make in-place modifications to existing *.mtimes files. In order to implement this, we have to adjust the behavior of 'want_found_object()'. When 'pack-objects' is told that we're *not* going to retain any cruft packs (i.e. the set of packs marked as kept in-core does not contain a cruft pack), the behavior is unchanged. But when there *is* at least one cruft pack that we're holding onto, it is no longer sufficient to reject a copy of an object found in that cruft pack for that reason alone. In this case, we only want to reject a candidate object when copies of that object either: - exists in a non-cruft pack that we are retaining, regardless of that pack's mtime, or - exists in a cruft pack with an mtime more recent than the copy we are debating whether or not to pack, in which case freshening would be redundant. To do this, keep track of whether or not we have any cruft packs in our in-core kept list with a new 'ignore_packed_keep_in_core_has_cruft' flag. When we end up in this new special case, we replace a call to 'has_object_kept_pack()' to 'want_cruft_object_mtime()', and only reject objects when we have a copy in an existing cruft pack with a more recent mtime (in which case "freshening" would be redundant). Signed-off-by: Taylor Blau --- builtin/pack-objects.c | 118 ++++++++++++++++++++++++++++++++++------ packfile.c | 3 +- packfile.h | 2 + t/t7704-repack-cruft.sh | 64 ++++++++++++++++++++++ 4 files changed, 169 insertions(+), 18 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 58a9b161262..79e1e6fb52b 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -206,6 +206,7 @@ static int have_non_local_packs; static int incremental; static int ignore_packed_keep_on_disk; static int ignore_packed_keep_in_core; +static int ignore_packed_keep_in_core_has_cruft; static int allow_ofs_delta; static struct pack_idx_option pack_idx_opts; static const char *base_name; @@ -1502,8 +1503,60 @@ static int have_duplicate_entry(const struct object_id *oid, return 1; } +static int want_cruft_object_mtime(struct repository *r, + const struct object_id *oid, + unsigned flags, uint32_t mtime) +{ + struct packed_git **cache; + + for (cache = kept_pack_cache(r, flags); *cache; cache++) { + struct packed_git *p = *cache; + off_t ofs; + uint32_t candidate_mtime; + + ofs = find_pack_entry_one(oid, p); + if (!ofs) + continue; + + /* + * We have a copy of the object 'oid' in a non-cruft + * pack. We can avoid packing an additional copy + * regardless of what the existing copy's mtime is since + * it is outside of a cruft pack. + */ + if (!p->is_cruft) + return 0; + + /* + * If we have a copy of the object 'oid' in a cruft + * pack, then either read the cruft pack's mtime for + * that object, or, if that can't be loaded, assume the + * pack's mtime itself. + */ + if (!load_pack_mtimes(p)) { + uint32_t pos; + if (offset_to_pack_pos(p, ofs, &pos) < 0) + continue; + candidate_mtime = nth_packed_mtime(p, pos); + } else { + candidate_mtime = p->mtime; + } + + /* + * We have a surviving copy of the object in a cruft + * pack whose mtime is greater than or equal to the one + * we are considering. We can thus avoid packing an + * additional copy of that object. + */ + if (mtime <= candidate_mtime) + return 0; + } + + return -1; +} + static int want_found_object(const struct object_id *oid, int exclude, - struct packed_git *p) + struct packed_git *p, uint32_t mtime) { if (exclude) return 1; @@ -1553,12 +1606,29 @@ static int want_found_object(const struct object_id *oid, int exclude, if (ignore_packed_keep_in_core) flags |= IN_CORE_KEEP_PACKS; - if (ignore_packed_keep_on_disk && p->pack_keep) - return 0; - if (ignore_packed_keep_in_core && p->pack_keep_in_core) - return 0; - if (has_object_kept_pack(p->repo, oid, flags)) - return 0; + /* + * If the object is in a pack that we want to ignore, *and* we + * don't have any cruft packs that are being retained, we can + * abort quickly. + */ + if (!ignore_packed_keep_in_core_has_cruft) { + if (ignore_packed_keep_on_disk && p->pack_keep) + return 0; + if (ignore_packed_keep_in_core && p->pack_keep_in_core) + return 0; + if (has_object_kept_pack(p->repo, oid, flags)) + return 0; + } else { + /* + * But if there is at least one cruft pack which + * is being kept, we only want to include the + * provided object if it has a strictly greater + * mtime than any existing cruft copy. + */ + if (!want_cruft_object_mtime(p->repo, oid, flags, + mtime)) + return 0; + } } /* @@ -1577,7 +1647,8 @@ static int want_object_in_pack_one(struct packed_git *p, const struct object_id *oid, int exclude, struct packed_git **found_pack, - off_t *found_offset) + off_t *found_offset, + uint32_t found_mtime) { off_t offset; @@ -1593,7 +1664,7 @@ static int want_object_in_pack_one(struct packed_git *p, *found_offset = offset; *found_pack = p; } - return want_found_object(oid, exclude, p); + return want_found_object(oid, exclude, p, found_mtime); } return -1; } @@ -1607,10 +1678,11 @@ static int want_object_in_pack_one(struct packed_git *p, * function finds if there is any pack that has the object and returns the pack * and its offset in these variables. */ -static int want_object_in_pack(const struct object_id *oid, - int exclude, - struct packed_git **found_pack, - off_t *found_offset) +static int want_object_in_pack_mtime(const struct object_id *oid, + int exclude, + struct packed_git **found_pack, + off_t *found_offset, + uint32_t found_mtime) { int want; struct list_head *pos; @@ -1625,7 +1697,8 @@ static int want_object_in_pack(const struct object_id *oid, * are present we will determine the answer right now. */ if (*found_pack) { - want = want_found_object(oid, exclude, *found_pack); + want = want_found_object(oid, exclude, *found_pack, + found_mtime); if (want != -1) return want; @@ -1636,7 +1709,7 @@ static int want_object_in_pack(const struct object_id *oid, for (m = get_multi_pack_index(the_repository); m; m = m->next) { struct pack_entry e; if (fill_midx_entry(the_repository, oid, &e, m)) { - want = want_object_in_pack_one(e.p, oid, exclude, found_pack, found_offset); + want = want_object_in_pack_one(e.p, oid, exclude, found_pack, found_offset, found_mtime); if (want != -1) return want; } @@ -1644,7 +1717,7 @@ static int want_object_in_pack(const struct object_id *oid, list_for_each(pos, get_packed_git_mru(the_repository)) { struct packed_git *p = list_entry(pos, struct packed_git, mru); - want = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset); + want = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime); if (!exclude && want > 0) list_move(&p->mru, get_packed_git_mru(the_repository)); @@ -1674,6 +1747,15 @@ static int want_object_in_pack(const struct object_id *oid, return 1; } +static inline int want_object_in_pack(const struct object_id *oid, + int exclude, + struct packed_git **found_pack, + off_t *found_offset) +{ + return want_object_in_pack_mtime(oid, exclude, found_pack, found_offset, + 0); +} + static struct object_entry *create_object_entry(const struct object_id *oid, enum object_type type, uint32_t hash, @@ -3606,7 +3688,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type entry->no_try_delta = no_try_delta(name); } } else { - if (!want_object_in_pack(oid, 0, &pack, &offset)) + if (!want_object_in_pack_mtime(oid, 0, &pack, &offset, mtime)) return; if (!pack && type == OBJ_BLOB && !has_loose_object(oid)) { /* @@ -3680,6 +3762,8 @@ static void mark_pack_kept_in_core(struct string_list *packs, unsigned keep) struct packed_git *p = item->util; if (!p) die(_("could not find pack '%s'"), item->string); + if (p->is_cruft && keep) + ignore_packed_keep_in_core_has_cruft = 1; p->pack_keep_in_core = keep; } } diff --git a/packfile.c b/packfile.c index 2d80d80cb38..9d09f8bc726 100644 --- a/packfile.c +++ b/packfile.c @@ -24,6 +24,7 @@ #include "commit-graph.h" #include "pack-revindex.h" #include "promisor-remote.h" +#include "pack-mtimes.h" char *odb_pack_name(struct repository *r, struct strbuf *buf, const unsigned char *hash, const char *ext) @@ -2107,7 +2108,7 @@ static void maybe_invalidate_kept_pack_cache(struct repository *r, r->objects->kept_pack_cache.flags = 0; } -static struct packed_git **kept_pack_cache(struct repository *r, unsigned flags) +struct packed_git **kept_pack_cache(struct repository *r, unsigned flags) { maybe_invalidate_kept_pack_cache(r, flags); diff --git a/packfile.h b/packfile.h index 00ada7a938f..25097213d06 100644 --- a/packfile.h +++ b/packfile.h @@ -197,6 +197,8 @@ int has_object_pack(struct repository *r, const struct object_id *oid); int has_object_kept_pack(struct repository *r, const struct object_id *oid, unsigned flags); +struct packed_git **kept_pack_cache(struct repository *r, unsigned flags); + /* * Return 1 if an object in a promisor packfile is or refers to the given * object, 0 otherwise. diff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh index 5a76b541ddd..2b20d66333c 100755 --- a/t/t7704-repack-cruft.sh +++ b/t/t7704-repack-cruft.sh @@ -262,6 +262,70 @@ test_expect_success '--max-cruft-size with freshened objects (packed)' ' ) ' +test_expect_success '--max-cruft-size with freshened objects (previously cruft)' ' + git init max-cruft-size-threshold && + ( + cd max-cruft-size-threshold && + + test_commit base && + foo="$(generate_random_blob foo $((2*1024*1024)))" && + bar="$(generate_random_blob bar $((2*1024*1024)))" && + baz="$(generate_random_blob baz $((2*1024*1024)))" && + + test-tool chmtime --get -100000 \ + "$objdir/$(test_oid_to_path "$foo")" >foo.old && + test-tool chmtime --get -100000 \ + "$objdir/$(test_oid_to_path "$bar")" >bar.old && + test-tool chmtime --get -100000 \ + "$objdir/$(test_oid_to_path "$baz")" >baz.old && + + git repack --cruft -d && + + # Make a packed copy of object $foo with a more recent + # mtime. + foo="$(generate_random_blob foo $((2*1024*1024)))" && + foo_pack="$(echo "$foo" | git pack-objects $packdir/pack)" && + test-tool chmtime --get -100 \ + "$packdir/pack-$foo_pack.pack" >foo.new && + git prune-packed && + + # Make a loose copy of object $bar with a more recent + # mtime. + bar="$(generate_random_blob bar $((2*1024*1024)))" && + test-tool chmtime --get -100 \ + "$objdir/$(test_oid_to_path "$bar")" >bar.new && + + # Make a new cruft object $quux to ensure we do not + # generate an identical pack to the existing cruft + # pack. + quux="$(generate_random_blob quux $((1024)))" && + test-tool chmtime --get -100 \ + "$objdir/$(test_oid_to_path "$quux")" >quux.new && + + git repack --cruft --max-cruft-size=3M -d && + + for p in $packdir/pack-*.mtimes + do + test-tool pack-mtimes "$(basename "$p")" || return 1 + done >actual.raw && + sort actual.raw >actual && + + # Among the set of all cruft packs, we should see both + # mtimes for object $foo and $bar, as well as the + # single new copy of $baz. + sort >expect <<-EOF && + $foo $(cat foo.old) + $foo $(cat foo.new) + $bar $(cat bar.old) + $bar $(cat bar.new) + $baz $(cat baz.old) + $quux $(cat quux.new) + EOF + + test_cmp expect actual + ) +' + test_expect_success '--max-cruft-size with pruning' ' git init max-cruft-size-prune && (