From patchwork Thu Apr 17 21:12:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 14056187 Received: from mail-qt1-f178.google.com (mail-qt1-f178.google.com [209.85.160.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E59AF20A5E5 for ; Thu, 17 Apr 2025 21:12:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744924338; cv=none; b=Gfs1kHMiK1FPbRggBP+0vultqVPrHG8YTnV7m4K70P1lKPAqgEUv8R/Ic46bNQm95w4ueB9dP+VVb5I/YOAohnJ2gt2eXduoyLuX8LU/b+ntzC+/jQ8WVtlGhjerjJ39VJN4jeDSlTmCXfiNnLqbMgcsF6zg5F/Nyi7nlkq1LDg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744924338; c=relaxed/simple; bh=TBtfsBb0rpZ3p5zr6DC5bCU8oYGTHoxduAOhsz+7hGk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=J3avKUv1rX7y3J6AL7WuO20XYUt3ozYNccud3SuRn1Gy8dt99hW3bq2+5uK+NSWk3952Yxv9Ezdf3XwhKTmpPhAHH4QOBB5gA7SoSjew+3de4eEsqzrVEzHzf53Jowxz6zaYf7FOGMcOayPn9zwGsbxD2RebqsV4uL0oSDCgK74= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=y69yVHUE; arc=none smtp.client-ip=209.85.160.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="y69yVHUE" Received: by mail-qt1-f178.google.com with SMTP id d75a77b69052e-476b89782c3so13736891cf.1 for ; Thu, 17 Apr 2025 14:12:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1744924335; x=1745529135; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=cFPX1yxFSFqxQPyW4yacuwuhvQbulx/Xx1RTo7ASFps=; b=y69yVHUEXQvCOphL5/+0cW8/zaEqLtDY0HMDxSmXnEFe2qgrQP7HWJpP+y5ej2l9lX v4kHFB4RePjGtNBdUuUa6JsJpYtVAiz0BmI1Hs+awYU3hT1AIomA4pS1DoTSBwvSNGQm yQEWrlCC2L+X6TRno1E42S+qibumKhmSss5AUkXGOVBJ5V25rzYIs0ciAuulhVOsD5HA q6SXz84M0ToCGz4lyh+4i3WRYBxwo1ZW/9+ww7qfRBtdj973a+1j1GPI8GXmtjzWSNwG qTVqM5Nteg4i0U+ctP7l+mDuSO6ache1WHENcaU9+dxN2v1FOHa2RdlYQwNUrUyw0yCg oWow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744924335; x=1745529135; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=cFPX1yxFSFqxQPyW4yacuwuhvQbulx/Xx1RTo7ASFps=; b=Q/5aPujmjGA0dceGLiB8Xf3gGXVSeVzxOSCQOTqci2+j7LEds5cyiNXl+t4w6grZw8 zl19OjLRV972YnbTWhV6eY5pbRKbFtQcPX0zhtyL0M/RGV5ZFdMTXvbM7r9qc/L0/srL ceaW3M2+3O+jMokLjmOhTbEme/hod+/e8GxmO/xRBwdHMp6Nwfd99CAptem0JiOKPaAu D8FSKHzituRsbCKMY3JvrChtabn8xhxTQLw6kf12tl1KISSkG5umLfcpp43PdWjzSHuM GP3yGzrOkOOL+GfvIp9aV44K7IJ2hPDyt8G+kBem39VCgeDK/Q3gj4HMwyQ5DAdAyzPS dzMQ== X-Gm-Message-State: AOJu0Yxzlf2Ava3zRKJWUDxNdYJ1bwkhraAXYRVVem05jyc4pPvJHVfX VmwTDZfTDJ1wg82Vn/Xg7se44IeZu0w0rlsqy1pGuaEyVnz4FJ7d1Y+MgQ8QA6mboUawTjtBy/K A0O8= X-Gm-Gg: ASbGncufeCr+uSUujp0C4SUwbNdrV3IlAuijz7NF+EN8Ncjhfvwvg46autjs9PwSIVC bKx6B0wdh0X0PnGerwQao2fz34tYGnZPHe3ha8gW8tic03u6oDEX5aXQv1oxuA+q4D5vlSizt1z g6jU+lIImo0MDnayjQ7djrh0rLT3uEAsdSKO1k9Afl1sv0bvJAm6h09xeYKp/ccOSsDOfO7zLts XBm5ifgVkuy/J2hfDSdBkvHyZAL6wasY626rwLgTjCieNp/x+xlZgLHLBHDiWiEl186eHaqPmaR gWznEylcFRjSX+OErzVVf4f78PPLGoPpDt+Wmv0LRiPJkqOPVuRu6Zxk9ArBNOOSADwQVOo3IUP AMk+6pUL8GLt4 X-Google-Smtp-Source: AGHT+IEeWI3GAm80r8yTt/X75jG2OsPLOk1m0zBxTXar2VvatDX6d7ytYawTVEq0H33VuKEa5cU3WA== X-Received: by 2002:ac8:5e14:0:b0:476:a3c8:c78d with SMTP id d75a77b69052e-47aec3cf9cfmr5205081cf.29.1744924335522; Thu, 17 Apr 2025 14:12:15 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id d75a77b69052e-47ae9cf9a8dsm3178951cf.70.2025.04.17.14.12.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Apr 2025 14:12:15 -0700 (PDT) Date: Thu, 17 Apr 2025 17:12:14 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Elijah Newren , Jeff King , Junio C Hamano Subject: [PATCH 1/4] pack-bitmap: write lookup table extension by default Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The lookup table extension for reachability bitmaps was first introduced via 3fe0121479 (Merge branch 'ac/bitmap-lookup-table', 2022-09-05). For each bitmapped commit, the lookup table encodes three unsigned integers: - The pack position of the commit. - An offset within the *.bitmap itself where that commit's bitmap lives. - An index into the same table where information on that commit's XOR pair can be found. Lookup tables make bitmap operations faster because they no longer have to read the entire *.bitmap file to discover what commits have corresponding reachability bitmaps. When bitmap lookup tables were first introduced, we established a baseline level of performance in p5310 with and without lookup tables. Here is the baseline without: Test this tree ----------------------------------------------------------------------- 5310.6: simulated clone 14.04(5.78+1.79) 5310.7: simulated fetch 1.95(3.05+0.20) 5310.8: pack to file (bitmap) 44.73(20.55+7.45) 5310.9: rev-list (commits) 0.78(0.46+0.10) 5310.10: rev-list (objects) 4.07(3.97+0.08) 5310.11: rev-list with tag negated via --not 0.06(0.02+0.03) --all (objects) 5310.12: rev-list with negative tag (objects) 0.21(0.15+0.05) 5310.13: rev-list count with blob:none 0.24(0.17+0.06) 5310.14: rev-list count with blob:limit=1k 7.07(5.92+0.48) 5310.15: rev-list count with tree:0 0.25(0.17+0.07) 5310.16: simulated partial clone 5.67(3.28+0.64) 5310.18: clone (partial bitmap) 16.05(8.34+1.86) 5310.19: pack to file (partial bitmap) 59.76(27.22+7.43) 5310.20: rev-list with tree filter (partial bitmap) 0.90(0.18+0.16) , and here is the same set of tests, this time with the lookup table enabled: Test this tree ----------------------------------------------------------------------- 5310.26: simulated clone 13.69(5.72+1.78) 5310.27: simulated fetch 1.84(3.02+0.16) 5310.28: pack to file (bitmap) 45.63(20.67+7.50) 5310.29: rev-list (commits) 0.56(0.39+0.8) 5310.30: rev-list (objects) 3.77(3.74+0.08) 5310.31: rev-list with tag negated via --not 0.05(0.02+0.03) --all (objects) 5310.32: rev-list with negative tag (objects) 0.21(0.15+0.05) 5310.33: rev-list count with blob:none 0.23(0.17+0.05) 5310.34: rev-list count with blob:limit=1k 6.65(5.72+0.40) 5310.35: rev-list count with tree:0 0.23(0.16+0.06) 5310.36: simulated partial clone 5.57(3.26+0.59) 5310.38: clone (partial bitmap) 15.89(8.39+1.84) 5310.39: pack to file (partial bitmap) 58.32(27.55+7.47) 5310.40: rev-list with tree filter (partial bitmap) 0.73(0.18+0.15) (All numbers here come from a ~2022-era copy of the kernel, via Abhradeep Chakraborty who implemented the lookup table extension). In the almost three years since lookup tables were introduced, GitHub has used them in production without issue, taking advantage of the above performance benefits along the way. Since this feature has had sufficient time to flush out any bugs and/or performance regressions, let's enable it by default so that all bitmap users can reap similar performance benefits. [1]: https://lore.kernel.org/git/pull.1266.git.1655728395.gitgitgadget@gmail.com/ Signed-off-by: Taylor Blau --- Documentation/config/pack.adoc | 2 +- builtin/multi-pack-index.c | 1 + builtin/pack-objects.c | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Documentation/config/pack.adoc b/Documentation/config/pack.adoc index da527377fa..ba538e3d9c 100644 --- a/Documentation/config/pack.adoc +++ b/Documentation/config/pack.adoc @@ -191,7 +191,7 @@ pack.writeBitmapLookupTable:: bitmap index (if one is written). This table is used to defer loading individual bitmaps as late as possible. This can be beneficial in repositories that have relatively large bitmap - indexes. Defaults to false. + indexes. Defaults to true. pack.readReverseIndex:: When true, git will read any .rev file(s) that may be available diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index 2a938466f5..6ad6f814e3 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -142,6 +142,7 @@ static int cmd_multi_pack_index_write(int argc, const char **argv, int ret; opts.flags |= MIDX_WRITE_BITMAP_HASH_CACHE; + opts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE; git_config(git_multi_pack_index_write_config, NULL); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 3973267e9e..384fefbb1d 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -239,7 +239,7 @@ static enum { WRITE_BITMAP_QUIET, WRITE_BITMAP_TRUE, } write_bitmap_index; -static uint16_t write_bitmap_options = BITMAP_OPT_HASH_CACHE; +static uint16_t write_bitmap_options = BITMAP_OPT_HASH_CACHE | BITMAP_OPT_LOOKUP_TABLE; static int exclude_promisor_objects; static int exclude_promisor_objects_best_effort; From patchwork Thu Apr 17 21:12:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 14056188 Received: from mail-qk1-f182.google.com (mail-qk1-f182.google.com [209.85.222.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA73A20C48D for ; Thu, 17 Apr 2025 21:12:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744924341; cv=none; b=OlLVo16+bwc0b/MiwzySnGW+zbfmvYGFlQ5nX8HpWK5J2i6Kzeh1FSsj64WLedLDSyQXkrwrmgS1XpbS9btgGR0ChYEz4Zs4oaLL5pT94H7GxDFZQRZG/BG/Dx3XhJBL2OpVFa6xtXuVFqCS8msURf1MOEZkb3i7X2c/oZ5BLVc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744924341; c=relaxed/simple; bh=WLg1EMTkCxZMOY0JfGrpXDLhNlQqF6x+XF+yjOKy2bM=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=P/20jxU9jA4XsHfHHev5ulHenQ6poJtb556qwEWNprwtFyUq1EicyheQYDPS+9zWK9DEAuls8SXbMuUveCZKlYVKz0VRv1GOjmW75xRtO1Ydz45GLVzCAlZtXGrgm55hsWQRh0EsmmJ/I7WuneZXhT7U2FOTT2y25cMY8yOb9Lg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=tGCyAS1n; arc=none smtp.client-ip=209.85.222.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="tGCyAS1n" Received: by mail-qk1-f182.google.com with SMTP id af79cd13be357-7be49f6b331so129010985a.1 for ; Thu, 17 Apr 2025 14:12:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1744924338; x=1745529138; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=bz8APon2NyHjpYlSUdGNNSSkl4iAVMW3VqT85kMBhS4=; b=tGCyAS1ntWxna56NDQGyugoDwFPraoOkQTdddgFwnR0IUEdUSt7hCFFsRpPDeWC50d 39soLy2pUfAboufXl9hKBK8Wb41Mx+irlF++65kvHqjRlbHIoJ8cfUf87ac8evqmj1QG fhZ4osesCLt1uNuDaypPMSfijrdOyt9DFGtHfCj1oZDEBICHAqq3t9PQLCMmHeQP+oAv WvrVen5btjdps6Vh+jAPi1SUIMEgpy1e8rFmJpnqxN/SxtejkFxuOEm2CKV0SRyD1nOJ emWtgtQCpf8iKmBIlvkXxOo1I8sHe0V1GfeulWjAztQFJjbiTxdvmXyPPKDWxwSZgr6V WtkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744924338; x=1745529138; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=bz8APon2NyHjpYlSUdGNNSSkl4iAVMW3VqT85kMBhS4=; b=K4/AeeOBcuvyv6uWAuLVftPj6wJgxwBU1O1Lwxs95BXspem1l8MP81NEAIUPknxPkj RLzEVUVljQ5E2iAxULlIeGjWeYTV2wL5NU2uimbN9t7a3zkC/g6RKDb9Fm6nkWR5Vqlv kmcssEDYhQCEoZ++Pdfmn4juooukjmpR8BpG3pnuo4rlZA5okUveX2iBMeII4/t6QxLq 1NOwd5y3Y1FVfkD3V5dgzWTE5XHBikMTJI1qjFMYRvkoGzdQLUwCCAsdPFiYMaqBWO8s Dzam9cenFaaE26yQJVxLZPJvVMctn2UCYW/Rd42HeaIgw7m3d6+DNKcbj3Vx2VTXPJhp syzw== X-Gm-Message-State: AOJu0YxFjm8381uuFRFMoxJi/SkPZj3Sp0WJ+MxqJh5gs2lIj27bQ7GQ vENvXD0GgKyT8URq3o3moHDyZfwYy4IadcWzTJ0OfElH2fUAKYWdgX+TsUFlQeg+K4RD2pIMjT+ Q+jM= X-Gm-Gg: ASbGnctMukrA42uQ7Mz+ye/m3kAu3iD11u3G01cFgy90O6Go19BOGBRpDIVPbexsYMB 8S0bmmcHdUMKtFSXjDedMoMh6iNfYNcuj1BMgzwaFrxN82SKbyo69WPFI3PwZPQ/0Dei+eDEQQP xTaXYnwUlDCjnmTl3rnDuF8ULFWmhYlDhVGlcIODuRJ3IeCDc/Wjc0JsH6g/gyo8YzLBgCcImf2 e+Jd0F+RvVQFBJU57K8nS7CzJQBhmZR727uGUrdS+EfnfaJWYMbjPtwH3x5qATYdCEgmWjVePEZ e754MnSYvQKwnpKBNAJtPNW5EM+e6a7g1D05P+7ej/mHmMHH4T4wkaJOhpvhGbrrZJoEMt4Yrxi qE+i6zUkTfK9Y X-Google-Smtp-Source: AGHT+IEZ3sa3fqW90f7H3SOZRP6EzOc8F0w6hVPNh7YKjXsnQVGyX60I4Yr28zpU6vUMNh35XIYrXg== X-Received: by 2002:a05:620a:4112:b0:7c5:59e1:f0d with SMTP id af79cd13be357-7c92800924amr55039585a.39.1744924338532; Thu, 17 Apr 2025 14:12:18 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id af79cd13be357-7c925a8fcdcsm33826485a.42.2025.04.17.14.12.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Apr 2025 14:12:18 -0700 (PDT) Date: Thu, 17 Apr 2025 17:12:17 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Elijah Newren , Jeff King , Junio C Hamano Subject: [PATCH 2/4] p5312: removed duplicate performance test script Message-ID: <51c4604e16c886d888138f2b513e4d3407b10728.1744924321.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When the reachability bitmap format learned to read and write a lookup table containing the set of commits which received reachability bitmaps, commit 761416ef91 (bitmap-lookup-table: add performance tests for lookup table, 2022-08-14) added that mirrored p5310 but with reverse indexes enabled. Later on in a8dd7e05b1 (config: enable `pack.writeReverseIndex` by default, 2023-04-12), we enabled reverse indexes by default, which made these two tests indistinguishable from one another. Commit a8dd7e05b1 should have removed p5312 as a duplicate, but didn't do so. Correct that by removing p5312 as a functional duplicate of p5310. Signed-off-by: Taylor Blau --- t/perf/p5312-pack-bitmaps-revs.sh | 34 ------------------------------- 1 file changed, 34 deletions(-) delete mode 100755 t/perf/p5312-pack-bitmaps-revs.sh diff --git a/t/perf/p5312-pack-bitmaps-revs.sh b/t/perf/p5312-pack-bitmaps-revs.sh deleted file mode 100755 index ceec60656b..0000000000 --- a/t/perf/p5312-pack-bitmaps-revs.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/bin/sh - -test_description='Tests pack performance using bitmaps (rev index enabled)' -. ./perf-lib.sh -. "${TEST_DIRECTORY}/perf/lib-bitmap.sh" - -test_lookup_pack_bitmap () { - test_expect_success 'start the test from scratch' ' - rm -rf * .git - ' - - test_perf_large_repo - - test_expect_success 'setup bitmap config' ' - git config pack.writebitmaps true - ' - - # we need to create the tag up front such that it is covered by the repack and - # thus by generated bitmaps. - test_expect_success 'create tags' ' - git tag --message="tag pointing to HEAD" perf-tag HEAD - ' - - test_perf "enable lookup table: $1" ' - git config pack.writeBitmapLookupTable '"$1"' - ' - - test_pack_bitmap -} - -test_lookup_pack_bitmap false -test_lookup_pack_bitmap true - -test_done From patchwork Thu Apr 17 21:12:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 14056189 Received: from mail-qk1-f182.google.com (mail-qk1-f182.google.com [209.85.222.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E5E4120E021 for ; Thu, 17 Apr 2025 21:12:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744924345; cv=none; b=joBBAoq2CU+HP+Uh1d6CyPWRW4EKvv2+2aUSp48GtzL+K4vUDLX35EunWZ6kdQ3RzQ/e1g9H3yHP2wEjDTINKkRDs2Burgp11C25aCClJ2h+oSurCBgBY5AAR9jOL79aKn9ueN2nEodcxoDThZWGOYcuQ+3DZl4JErBZuHiOcvA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744924345; c=relaxed/simple; bh=fj8vU7FR9F/qZtClnWRn7JepXSWYwf3YLmTuAoScBuc=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=UpmVXktjykhZEmk+Okqlkx7SGD4+7/jAlx+e109z3fwm/Gun5p8KwfBYIVFMFJ5vHCA4F4aiqTLJlWdyqt2OFR8l2/1jLdAdxBi6aGzIG7WO3+TBHUeeEo4Fjxn7IEc50ay3iyUHjxnUPKUpZkhOVYOjlwMGAhHUrd/spNDRUU4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=fLht5hdG; arc=none smtp.client-ip=209.85.222.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="fLht5hdG" Received: by mail-qk1-f182.google.com with SMTP id af79cd13be357-7c5b2472969so122367985a.1 for ; Thu, 17 Apr 2025 14:12:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1744924341; x=1745529141; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=Z2phTotV+oShzmiwAUvA5ur83GsRNUF6Cts+dq3LavQ=; b=fLht5hdGWdX6SYpn2RgTaQW9C1Us9q9wfUQr0wZbKCKxx/H07EL/CTeaCF0Zjw6Ta8 GUxD2qmXg36rR1YN/hqzrZCf2/IoIEE7ElPVMBjpc7ba8A7QflrmwbIH1bHPMUdhr/HH QKsHNNvfpxkSHCyVwn6A6g4JSBHALJxl9IyP4gbEujwxBjg3EZxUVKAXRpC6aqSZ2Z0d K+vsegPzM4otlT1NtEKlcA/MbaIlRxQTmpWCmk8jPSQ4CsTvjiOplHFbNaQL2q9rt9SY lTD4pUEQU7cnQUMbPua/GV3OnoY2eHjKRe/FPNYFWkaljMfjEOszcJojNEyNffPrUyj2 lFog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744924341; x=1745529141; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Z2phTotV+oShzmiwAUvA5ur83GsRNUF6Cts+dq3LavQ=; b=rdDjGxMm9tChZfXfx3rRE/ohSUrP1I+q6ZeN3tBdjDKl1pv8k6f27BoByJaMU7MqGY d4+wEZ7Kt0QuEIM3vV0QNMvA1A8joVkPfJNbQHhIsOhro2POINlFGdmVyIsphc6Sxcf2 fOQNIs95lyBPEgNkuJZaXZV5C8OMcBDxNXc9RPtR6axdCMnGHo2Wpvp59E+sxoAzU+wN aO3Y0TrkcLqlKg2sR6Iju6hVpY33hQxdmiDqI5LjzLpxDGQ7mgzNdtvaHklh6NjDsaUo IFOvtMmqEtDsiTNSzrKyrzBynkXfjEngG6kA9PpTyIKYHhQqJaR+whp5dd2o33SO54SO AZfA== X-Gm-Message-State: AOJu0YwRLCDZeX8GEUmM8NKEDj6pEKMh+YgIpYDTxPygwkPB1/yS4bcm bREUssXgNL94zvzyVYcFlhsA9BsOs4EIjdMPV2TXOJdGYprJ4lMzPLiFjOWj33ORetq8nc78z/c 3kKU= X-Gm-Gg: ASbGncs2dfwbUnT5Wmj5017j8j3St+npEoW4BSVFXL5n1ZXIelwlcj89CO601wFyx19 7RENgzZ3yE4aHfNA3+B4N0mP2p/WhvAo4D24wo1WgrDThn5JgpN77T8qULLR/YFSHDyfBVJg9Hd Hw9G5gngSkkdU5ZILhsYcUKgVxA3szxfxbyjC81JSBdlrc4Z+o16Ur7H3cq36s7c/c5BVOu4XA3 721bEjvq0eVCZEQnP+mA/lyYiQeyobXEpz5npb9mWEMb5B0dmIn2Dz426YSy32bEpBZvFFS47ap TNT1ywwqwLSgR8CJ4L0FasnMZT4vZOe6YYT9s8mWoHqshHDcNLmWIg/0f6qTO0DHVLAJbVl07uk 03DRgLbX2mEDs X-Google-Smtp-Source: AGHT+IGsKepdrQq6qUhQ6FQ//IrDiZa4vY/dU46ggS4IZIspfTvoTP4NoOeyTZjPMkeGW4w/oFfg8A== X-Received: by 2002:a05:620a:bc4:b0:7c5:562d:ccf8 with SMTP id af79cd13be357-7c927f76f3cmr68662385a.8.1744924341493; Thu, 17 Apr 2025 14:12:21 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id af79cd13be357-7c925b75c8dsm32806185a.111.2025.04.17.14.12.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Apr 2025 14:12:21 -0700 (PDT) Date: Thu, 17 Apr 2025 17:12:20 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Elijah Newren , Jeff King , Junio C Hamano Subject: [PATCH 3/4] t/perf: avoid testing bitmaps without lookup table Message-ID: <8cc5952e594b78ffb2ba4bcaabd62a8e5b8fe72a.1744924321.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In a previous commit, the setting which controls whether or not the pack- and MIDX-bitmap machinery writes a lookup table, 'pack.writeBitmapLookupTable' was enabled by default. As a result, we can clean up many of our bitmap-related performance tests. Many of the relevant performance tests look something like: test_it () { test_expect_success 'setup pack.writeBitmapLookupTable' ' git config pack.writeBitmapLookupTable '"$1"' ' # ... } test_it true test_it false , where the two invocations of 'test_it' run the tests with and without bitmap lookup tables enabled. But now that lookup tables are enabled by default and have proven to be a performance win, let's avoid benchmarking what is now an uncommon and non-default scenario. Signed-off-by: Taylor Blau --- t/perf/p5310-pack-bitmaps.sh | 47 +++++------- t/perf/p5311-pack-bitmaps-fetch.sh | 76 +++++++++---------- t/perf/p5326-multi-pack-bitmaps.sh | 107 ++++++++++++--------------- t/perf/p5333-pseudo-merge-bitmaps.sh | 1 - 4 files changed, 102 insertions(+), 129 deletions(-) diff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh index b1399f1007..52f9fca14b 100755 --- a/t/perf/p5310-pack-bitmaps.sh +++ b/t/perf/p5310-pack-bitmaps.sh @@ -4,37 +4,28 @@ test_description='Tests pack performance using bitmaps' . ./perf-lib.sh . "${TEST_DIRECTORY}/perf/lib-bitmap.sh" -test_lookup_pack_bitmap () { - test_expect_success 'start the test from scratch' ' - rm -rf * .git - ' +test_expect_success 'start the test from scratch' ' + rm -rf * .git +' - test_perf_large_repo +test_perf_large_repo - # note that we do everything through config, - # since we want to be able to compare bitmap-aware - # git versus non-bitmap git - # - # We intentionally use the deprecated pack.writebitmaps - # config so that we can test against older versions of git. - test_expect_success 'setup bitmap config' ' - git config pack.writebitmaps true - ' +# note that we do everything through config, +# since we want to be able to compare bitmap-aware +# git versus non-bitmap git +# +# We intentionally use the deprecated pack.writebitmaps +# config so that we can test against older versions of git. +test_expect_success 'setup bitmap config' ' + git config pack.writebitmaps true +' - # we need to create the tag up front such that it is covered by the repack and - # thus by generated bitmaps. - test_expect_success 'create tags' ' - git tag --message="tag pointing to HEAD" perf-tag HEAD - ' +# we need to create the tag up front such that it is covered by the repack and +# thus by generated bitmaps. +test_expect_success 'create tags' ' + git tag --message="tag pointing to HEAD" perf-tag HEAD +' - test_perf "enable lookup table: $1" ' - git config pack.writeBitmapLookupTable '"$1"' - ' - - test_pack_bitmap -} - -test_lookup_pack_bitmap false -test_lookup_pack_bitmap true +test_pack_bitmap test_done diff --git a/t/perf/p5311-pack-bitmaps-fetch.sh b/t/perf/p5311-pack-bitmaps-fetch.sh index 047efb995d..75b05a600e 100755 --- a/t/perf/p5311-pack-bitmaps-fetch.sh +++ b/t/perf/p5311-pack-bitmaps-fetch.sh @@ -3,52 +3,46 @@ test_description='performance of fetches from bitmapped packs' . ./perf-lib.sh -test_fetch_bitmaps () { - test_expect_success 'setup test directory' ' - rm -fr * .git - ' - - test_perf_default_repo +test_expect_success 'setup test directory' ' + rm -fr * .git +' - test_expect_success 'create bitmapped server repo' ' - git config pack.writebitmaps true && - git config pack.writeBitmapLookupTable '"$1"' && - git repack -ad - ' +test_perf_default_repo - # simulate a fetch from a repository that last fetched N days ago, for - # various values of N. We do so by following the first-parent chain, - # and assume the first entry in the chain that is N days older than the current - # HEAD is where the HEAD would have been then. - for days in 1 2 4 8 16 32 64 128; do - title=$(printf '%10s' "($days days)") - test_expect_success "setup revs from $days days ago" ' - now=$(git log -1 --format=%ct HEAD) && - then=$(($now - ($days * 86400))) && - tip=$(git rev-list -1 --first-parent --until=$then HEAD) && - { - echo HEAD && - echo ^$tip - } >revs - ' +test_expect_success 'create bitmapped server repo' ' + git config pack.writebitmaps true && + git repack -ad +' - test_perf "server $title (lookup=$1)" ' - git pack-objects --stdout --revs \ - --thin --delta-base-offset \ - tmp.pack - ' +# simulate a fetch from a repository that last fetched N days ago, for +# various values of N. We do so by following the first-parent chain, +# and assume the first entry in the chain that is N days older than the current +# HEAD is where the HEAD would have been then. +for days in 1 2 4 8 16 32 64 128; do + title=$(printf '%10s' "($days days)") + test_expect_success "setup revs from $days days ago" ' + now=$(git log -1 --format=%ct HEAD) && + then=$(($now - ($days * 86400))) && + tip=$(git rev-list -1 --first-parent --until=$then HEAD) && + { + echo HEAD && + echo ^$tip + } >revs + ' - test_size "size $title" ' - test_file_size tmp.pack - ' + test_perf "server $title" ' + git pack-objects --stdout --revs \ + --thin --delta-base-offset \ + tmp.pack + ' - test_perf "client $title (lookup=$1)" ' - git index-pack --stdin --fix-thin X-Patchwork-Id: 14056190 Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA95820C48D for ; Thu, 17 Apr 2025 21:12:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744924347; cv=none; b=E24E9vdDgUYos1X9qMrp5zaxLjDoTPXVWvRNjskFXAMNvhekeuaD9l6jztBhKCmc1ryZ//GcCmks0SSt+cpuOy59bjcvlPsJ0PG/J1CJKPQQd/R3yMNdeI25Z0iR/ZxJUcxASKQZDPGcnCbGIERVGSxLt6NWepOMkQXARL8FlI4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744924347; c=relaxed/simple; bh=5hxTwHr/osfqYdor1U8XDWtSy10bRFtgjLpEJ9DnHBo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=gAhTzKDNd+1kTtrxLC3KiT/ZFRoF49bCb5HBlcemq+wyDqe3NZuWdD2UL69i6d9n46GJqrTiTbCiYPOf+NVJTyZ0pGJgWUpOti7U4Zo1ef64iASkreTy1QINq0biMhq/oHRSYiyMdjf8a2Ii47Xciftgdite1x5Aa4OWQZwda7U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=emSnXL1E; arc=none smtp.client-ip=209.85.222.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="emSnXL1E" Received: by mail-qk1-f172.google.com with SMTP id af79cd13be357-7c5e1b40f68so126424585a.1 for ; Thu, 17 Apr 2025 14:12:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1744924344; x=1745529144; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=LTORUGvVIfHeQvTJ4ksMsk1qox99eoiYs6suXQ6MmQw=; b=emSnXL1EY9fgPWTzaSRjouBhUzLBu3YfCqTHJfbPQub5jC0FS7gJVTUEmdmeTZKbIG DzTm3Im4TPrlIVgRj/gb00Zasmrina/8UrQDEpjt8rWWx5QeCcD/fy+U/FZ2ma5Xd6HT qrt8IjkUcyVZL0D4/steEPcGpzxXWgITbuA3rYJnmDyDo/pE/bRnrriH9SuM/aDe9ol9 tsS2FUpOQfNBOhqU9PpZS/2KAQocYF4xdOMphzlSw0k8bwdn1RJFiU6bAyvhJJ5O12l9 8ksCNdbOVZ53dKyL6rBkrApb2355Tej0NBhXfXAkwQl7UyNa1EbXkZ3tu/5Aage26+E4 ZNkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744924344; x=1745529144; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=LTORUGvVIfHeQvTJ4ksMsk1qox99eoiYs6suXQ6MmQw=; b=k+RmE0dBwFJpzTCbTt0Izw6VpCzud1i2cFJ7xbIujND+S4ZBsxYai//YEM7gHrdnvn w4ImCuHgRnk0N/4/855pVUmfBvC97wFsI95Vh+In4M0LLmzZ8s6O11wGdMBij/rNEjKX 1q0DvMV2ynK7uxyc/0j9RB7Sy2OEqxSEDAtoqb/kqUFzlFx+SX8nxy9TdiwiA4zR4Rqw +c4jv9zQ7ZZMGZebbM60wzieKKwP2jTktNsWKoHOMhM5bVFaCihbxCm7NrWvBtWzPhMX 7d7uAJMTDbTqiXvtLKPd0ayUDBPRoWqKyNjY2Iz8OYiEDrtyw4W/U+OMp0GUNj44Vub8 ZXug== X-Gm-Message-State: AOJu0Yx03TF4KZWyOp3r3Y0J9lgOpSrzTDhFOILh5LrgzgLEMNKqEKd4 m2EX8E1pNVANXahhK3rHjiqr9sSGsrLP0xb7w3t6THnNMuCSfPh5v4UrU7siJIf/aE0m43n98P2 MJuE= X-Gm-Gg: ASbGncv/i4UthJ9XT6jhcBO6Ut1J+AdxlHXj9QK5IWaexCLB91b+1gbGR2PKk9Ih1js gJC5bWebrlDWrQ3if82pKJ3/xUtVbtsUTkpdYgu4jg3uZpSHVcSVhHZS1JyRfYkGQr5mYlsOUmq +MbBwLpivFV866N25pr1y089HWy24dZrzF3xd0tCS7QMqmjJjllVctg/aLX2qgmHOYtny0lH7/I PQKz+ZrdT52Gb30QXEmiHy9KTiE5ssQ1AY5bloSnxj/IiuD9uN3w+Rq60ZtmQH/jB9Qk9nB8zMr v0cIcUSVhZ0WzAVVDyUwHbOZ5+gpmGRpB8j5VPvDnpm/p3Ex7WDYmNYL0s7WxceCyeeuOPPfaR5 VJBd8p1CX3X3B X-Google-Smtp-Source: AGHT+IFEsYJCPhoRHw778ubM+JydMqagCodR+b0Mvtd81w+6p+fjobzITU2IhGhoyWpITUJCWhQgzA== X-Received: by 2002:a05:620a:3953:b0:7c5:468b:5658 with SMTP id af79cd13be357-7c927fb904cmr63193685a.28.1744924344462; Thu, 17 Apr 2025 14:12:24 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id af79cd13be357-7c925b778b5sm32662785a.102.2025.04.17.14.12.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Apr 2025 14:12:24 -0700 (PDT) Date: Thu, 17 Apr 2025 17:12:23 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Elijah Newren , Jeff King , Junio C Hamano Subject: [PATCH 4/4] t/perf/lib-bitmap.sh: avoid test_perf during setup Message-ID: <0906e14c0e55b52573c7e0b632c7c639850700ec.1744924321.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In the test_pack_bitmap() helper function, we first repack the repository under test for consistency and to eliminate any effects from different distributions of objects among packs. This step is performed with test_perf, so it is repeated $GIT_PERF_REPEAT_COUNT number of times. But we do not care about timing this portion of the setup phase, and repeating the process does not change the outcome. Use test_expect_success to avoid spending time repeating an idempotent portion of the setup for performance tests that use test_pack_bitmap(). Signed-off-by: Taylor Blau --- t/perf/lib-bitmap.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/perf/lib-bitmap.sh b/t/perf/lib-bitmap.sh index 55a8feb1dc..fdf5f35f1b 100644 --- a/t/perf/lib-bitmap.sh +++ b/t/perf/lib-bitmap.sh @@ -69,7 +69,7 @@ test_partial_bitmap () { } test_pack_bitmap () { - test_perf "repack to disk" ' + test_expect_success "repack to disk" ' git repack -ad '