From patchwork Wed Jul 20 18:38:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Abhradeep Chakraborty X-Patchwork-Id: 12924392 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34ED0C433EF for ; Wed, 20 Jul 2022 18:38:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231937AbiGTSil (ORCPT ); Wed, 20 Jul 2022 14:38:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229648AbiGTSih (ORCPT ); Wed, 20 Jul 2022 14:38:37 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B88057391B for ; Wed, 20 Jul 2022 11:38:29 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id b21-20020a05600c4e1500b003a32bc8612fso1317800wmq.3 for ; Wed, 20 Jul 2022 11:38:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=+/xO9ll8JpeGK0I4H7EYcynDHbRKCdQAMzzulpwBe7Q=; b=mnKOAFm6xETuUv6A+d0BRzPHJY+QA7/7YcrqhZOrsIxloMdsfFHfNkf/6E1lV33WTe e2Sp8mE53YAef8XOcq0cU2SWQP8DNMYvYxBFjyFdTIud2VO7VslD7Gn+WQ26Xwv6mRCj G9AdGAbfcb2o1h08PZvoCSNYBm9TGoRRA4FA8AgEgocnubQyZuQuU4nNrGjklKSbXEsL RLppWFhgzuNIMJ2j7H5AV1dCcaYXTSaIX5RPYzg81eSRVxkpypsqkZVi1y319zmI02bj +Xn1e/EzmEXyR3AUZ+K6a10OKlRV6uLveSG5R9832mNq1g8si5Lpi1FUWI5rB9YNYo6z 2deA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=+/xO9ll8JpeGK0I4H7EYcynDHbRKCdQAMzzulpwBe7Q=; b=6NX7dAk5PWlMt/gVEdhSw3MtQ0NdgO0ajI7fiS6rNVLGBHWq2BoOjCVNLA2cQmbn7a XeKuFohTyoaNXZ5xFW7MpWdKlKY2Wt7xz3XCsn/NPA41/C39OPHM4FqcviZSGRROPuJJ H+kUvYI4UHmNWruapPsgrEkuGOT+GcgHv+nRISuw7t+kaiP/Tjy5U0D39LAwbAkxY/u9 svvP7kS0MAQcK+/w8bX5/Km30QIykYb2hJx4zJJbhUGitRrq1YlERaum9nLBH8b2xRXH MFbiWEzpfk4QRoC+T5H/NlQedcQwbazB8uHkn/pS7FRBZCXthwDeLKQTQo5Xl2JaQHu/ exEQ== X-Gm-Message-State: AJIora+aDCdyG54cL8kXY9fVGMU8mHoHTNql29V5694L/ANILJr9Uc/J dDmIa880ETeOVRgt4xTNhn3BGKF9ZuE= X-Google-Smtp-Source: AGRyM1u7sDm9D+sTy8XcWGETL2jnr2wZFBucgW4vOtWm9t9f/6fpqrGdiciwBkgSz5tvCzRZ7044tA== X-Received: by 2002:a05:600c:348f:b0:3a2:e369:9cc2 with SMTP id a15-20020a05600c348f00b003a2e3699cc2mr5173315wmq.166.1658342307493; Wed, 20 Jul 2022 11:38:27 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s2-20020a7bc382000000b003a03185231bsm3356035wmj.31.2022.07.20.11.38.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Jul 2022 11:38:26 -0700 (PDT) Message-Id: <33aca8f3dc8b12773f187fcb8e40060c2be4aabc.1658342304.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Jul 2022 18:38:19 +0000 Subject: [PATCH v5 1/6] Documentation/technical: describe bitmap lookup table extension Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Taylor Blau , Kaartic Sivaram , Derrick Stolee , Philip Oakley , Martin =?utf-8?b?w4VncmVu?= , Abhradeep Chakraborty , Abhradeep Chakraborty Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Abhradeep Chakraborty From: Abhradeep Chakraborty When reading bitmap file, Git loads each and every bitmap one by one even if all the bitmaps are not required. A "bitmap lookup table" extension to the bitmap format can reduce the overhead of loading bitmaps which stores a list of bitmapped commit id pos (in the midx or pack, along with their offset and xor offset. This way Git can load only the necessary bitmaps without loading the previous bitmaps. Older versions of Git ignore the lookup table extension and don't throw any kind of warning or error while parsing the bitmap file. Add some information for the new "bitmap lookup table" extension in the bitmap-format documentation. Mentored-by: Taylor Blau Co-Mentored-by: Kaartic Sivaraam Co-Authored-by: Taylor Blau Signed-off-by: Abhradeep Chakraborty --- Documentation/technical/bitmap-format.txt | 39 +++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt index 04b3ec21785..c30dc177643 100644 --- a/Documentation/technical/bitmap-format.txt +++ b/Documentation/technical/bitmap-format.txt @@ -67,6 +67,17 @@ MIDXs, both the bit-cache and rev-cache extensions are required. pack/MIDX. The format and meaning of the name-hash is described below. + ** {empty} + BITMAP_OPT_LOOKUP_TABLE (0x10): ::: + If present, the end of the bitmap file contains a table + containing a list of `N` + triplets. The format and meaning of the table is described + below. ++ +NOTE: Unlike the xor_offset used to compress an individual bitmap, +`xor_row` stores an *absolute* index into the lookup table, not a location +relative to the current entry. + 4-byte entry count (network byte order) The total count of entries (bitmapped commits) in this bitmap index. @@ -205,3 +216,31 @@ Note that this hashing scheme is tied to the BITMAP_OPT_HASH_CACHE flag. If implementations want to choose a different hashing scheme, they are free to do so, but MUST allocate a new header flag (because comparing hashes made under two different schemes would be pointless). + +Commit lookup table +------------------- + +If the BITMAP_OPT_LOOKUP_TABLE flag is set, the last `N * (4 + 8 + 4)` +bytes (preceding the name-hash cache and trailing hash) of the `.bitmap` +file contains a lookup table specifying the information needed to get +the desired bitmap from the entries without parsing previous unnecessary +bitmaps. + +For a `.bitmap` containing `nr_entries` reachability bitmaps, the table +contains a list of `nr_entries` triplets +(sorted in the ascending order of `commit_pos`). The content of i'th +triplet is - + + * {empty} + commit_pos (4 byte integer, network byte order): :: + It stores the object position of a commit (in the midx or pack + index). + + * {empty} + offset (8 byte integer, network byte order): :: + The offset from which that commit's bitmap can be read. + + * {empty} + xor_row (4 byte integer, network byte order): :: + The position of the triplet whose bitmap is used to compress + this one, or `0xffffffff` if no such bitmap exists. From patchwork Wed Jul 20 18:38:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Abhradeep Chakraborty X-Patchwork-Id: 12924393 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 767FEC433EF for ; Wed, 20 Jul 2022 18:38:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233300AbiGTSis (ORCPT ); Wed, 20 Jul 2022 14:38:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232246AbiGTSiq (ORCPT ); Wed, 20 Jul 2022 14:38:46 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B10273923 for ; Wed, 20 Jul 2022 11:38:31 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id w8-20020a05600c014800b003a32e89bc4eso185025wmm.5 for ; Wed, 20 Jul 2022 11:38:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JWMGsY31T3LEGQ6YK4QNg917rcyopL0Wjhs3P4Ya9Dc=; b=ju0fJzo+UiNTSoxNx3bT738yJv2F0B/7FQ1M2ZzzKF70tIF1By4/ZqDNqbUVPTh7z5 Za+XtqzVqAO9OULE8oaLsbbkIUQXxpLWbXpRLjwIc7r0lQFK9vAu1meruXiUuMqzp3Au 7pFfzsoHMnPNYEE4pz2fGMmSuv/nk+z1wvKkQKsV7r1xXMLMM+G3iWUQXNBx86QDd59s 77VbdsQSG7Kr1/qdn+fjtpVht2k0M3DKoGJnIMGuzcl9fWMhTvNSz7h8QA+hPnGY7kn0 /T9gXPuCHDG7KdKufTq/xzGSyHzwfvu/ULEkMLT+LkHC8uVb7NZkUqGEcNw76gEllVLU VljA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JWMGsY31T3LEGQ6YK4QNg917rcyopL0Wjhs3P4Ya9Dc=; b=pzQ+8fo197s4vs965aU7PxsrMfaKy8a2qCK7sB9RgGQjbUpntDxIgmyjYJFMu4l//T bBiLMNPbcdd0RFFQEsY2cV4THnLsLmHOkVJmCXo99+Baiy8+4SdQzhF7RZffTVo+fXK5 IuW+sBGX7bwxyZ4zv3WQHR3tshOvZweMRxlF/h0JXXPXiCqRB58eKd4c60kPcoQxkD7q z0D4ljDtR11DsrNt+HXQMgapM8d31v68krgri9L54ZvDCLFsoLf1UCp4DzjmMgRsmfxN 6/5o6L4NQjnCMSgQXFciKlww/7En6elacC3i61G6fRZ86omjXDjLM9Unay24ZPTzCwCm QkEQ== X-Gm-Message-State: AJIora/hyB1ZYqtuEUsU3so/I9Nq/4Gptmg4Zgy20aJN6r7dz/E68+5m VBbFCZ8R2vm6HCcONq2U8KGyNMzMUso= X-Google-Smtp-Source: AGRyM1sUd+04Cjc57L4jtIJtrhLnL/y1T3tBAn90nhG3PPPdPdDWYKvU2Z+sSbK/1x0yFG94GJOYSg== X-Received: by 2002:a05:600c:3ac3:b0:3a0:45b6:7efb with SMTP id d3-20020a05600c3ac300b003a045b67efbmr5050904wms.183.1658342309060; Wed, 20 Jul 2022 11:38:29 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o4-20020adfcf04000000b0021d7fa77710sm16189350wrj.92.2022.07.20.11.38.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Jul 2022 11:38:28 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 20 Jul 2022 18:38:20 +0000 Subject: [PATCH v5 2/6] pack-bitmap-write.c: write lookup table extension Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Taylor Blau , Kaartic Sivaram , Derrick Stolee , Philip Oakley , Martin =?utf-8?b?w4VncmVu?= , Abhradeep Chakraborty , Abhradeep Chakraborty Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Abhradeep Chakraborty From: Abhradeep Chakraborty The bitmap lookup table extension was documented by an earlier change, but Git does not yet know how to write that extension. Teach Git to write bitmap lookup table extension. The table contains the list of `N` ` triplets. These triplets are sorted according to their commit pos (ascending order). The meaning of each data in the i'th triplet is given below: - commit_pos stores commit position (in the pack-index or midx). It is a 4 byte network byte order unsigned integer. - offset is the position (in the bitmap file) from which that commit's bitmap can be read. - xor_row is the position of the triplet in the lookup table whose bitmap is used to compress this bitmap, or `0xffffffff` if no such bitmap exists. Mentored-by: Taylor Blau Co-mentored-by: Kaartic Sivaraam Co-authored-by: Taylor Blau Signed-off-by: Abhradeep Chakraborty --- pack-bitmap-write.c | 112 ++++++++++++++++++++++++++++++++++++++++---- pack-bitmap.h | 5 +- 2 files changed, 107 insertions(+), 10 deletions(-) diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index c43375bd344..9843790cb60 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -650,20 +650,19 @@ static const struct object_id *oid_access(size_t pos, const void *table) static void write_selected_commits_v1(struct hashfile *f, struct pack_idx_entry **index, - uint32_t index_nr) + uint32_t index_nr, + off_t *offsets, + uint32_t *commit_positions) { int i; for (i = 0; i < writer.selected_nr; ++i) { struct bitmapped_commit *stored = &writer.selected[i]; - int commit_pos = - oid_pos(&stored->commit->object.oid, index, index_nr, oid_access); + if (offsets) + offsets[i] = hashfile_total(f); - if (commit_pos < 0) - BUG("trying to write commit not in index"); - - hashwrite_be32(f, commit_pos); + hashwrite_be32(f, commit_positions[i]); hashwrite_u8(f, stored->xor_offset); hashwrite_u8(f, stored->flags); @@ -671,6 +670,81 @@ static void write_selected_commits_v1(struct hashfile *f, } } +static int table_cmp(const void *_va, const void *_vb, void *_data) +{ + uint32_t *commit_positions = _data; + uint32_t a = commit_positions[*(uint32_t *)_va]; + uint32_t b = commit_positions[*(uint32_t *)_vb]; + + if (a > b) + return 1; + else if (a < b) + return -1; + + return 0; +} + +static void write_lookup_table(struct hashfile *f, + struct pack_idx_entry **index, + uint32_t index_nr, + off_t *offsets, + uint32_t *commit_positions) +{ + uint32_t i; + uint32_t *table, *table_inv; + + ALLOC_ARRAY(table, writer.selected_nr); + ALLOC_ARRAY(table_inv, writer.selected_nr); + + for (i = 0; i < writer.selected_nr; i++) + table[i] = i; + + /* + * At the end of this sort table[j] = i means that the i'th + * bitmap corresponds to j'th bitmapped commit (among the selected + * commits) in lex order of OIDs. + */ + QSORT_S(table, writer.selected_nr, table_cmp, commit_positions); + + /* table_inv helps us discover that relationship (i'th bitmap + * to j'th commit by j = table_inv[i]) + */ + for (i = 0; i < writer.selected_nr; i++) + table_inv[table[i]] = i; + + trace2_region_enter("pack-bitmap-write", "writing_lookup_table", the_repository); + for (i = 0; i < writer.selected_nr; i++) { + struct bitmapped_commit *selected = &writer.selected[table[i]]; + uint32_t xor_offset = selected->xor_offset; + uint32_t xor_row; + + if (xor_offset) { + /* + * xor_index stores the index (in the bitmap entries) + * of the corresponding xor bitmap. But we need to convert + * this index into lookup table's index. So, table_inv[xor_index] + * gives us the index position w.r.t. the lookup table. + * + * If "k = table[i] - xor_offset" then the xor base is the k'th + * bitmap. `table_inv[k]` gives us the position of that bitmap + * in the lookup table. + */ + uint32_t xor_index = table[i] - xor_offset; + xor_row = table_inv[xor_index]; + } else { + xor_row = 0xffffffff; + } + + hashwrite_be32(f, commit_positions[table[i]]); + hashwrite_be64(f, (uint64_t)offsets[table[i]]); + hashwrite_be32(f, xor_row); + } + trace2_region_leave("pack-bitmap-write", "writing_lookup_table", the_repository); + + free(table); + free(table_inv); +} + static void write_hash_cache(struct hashfile *f, struct pack_idx_entry **index, uint32_t index_nr) @@ -695,8 +769,10 @@ void bitmap_writer_finish(struct pack_idx_entry **index, { static uint16_t default_version = 1; static uint16_t flags = BITMAP_OPT_FULL_DAG; + off_t *offsets = NULL; struct strbuf tmp_file = STRBUF_INIT; struct hashfile *f; + uint32_t *commit_positions = NULL; struct bitmap_disk_header header; @@ -715,7 +791,25 @@ void bitmap_writer_finish(struct pack_idx_entry **index, dump_bitmap(f, writer.trees); dump_bitmap(f, writer.blobs); dump_bitmap(f, writer.tags); - write_selected_commits_v1(f, index, index_nr); + + ALLOC_ARRAY(commit_positions, writer.selected_nr); + for (uint32_t i = 0; i < writer.selected_nr; ++i) { + struct bitmapped_commit *stored = &writer.selected[i]; + int commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access); + + if (commit_pos < 0) + BUG(_("trying to write commit not in index")); + + commit_positions[i] = commit_pos; + } + + if (options & BITMAP_OPT_LOOKUP_TABLE) + CALLOC_ARRAY(offsets, index_nr); + + write_selected_commits_v1(f, index, index_nr, offsets, commit_positions); + + if (options & BITMAP_OPT_LOOKUP_TABLE) + write_lookup_table(f, index, index_nr, offsets, commit_positions); if (options & BITMAP_OPT_HASH_CACHE) write_hash_cache(f, index, index_nr); @@ -730,4 +824,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index, die_errno("unable to rename temporary bitmap file to '%s'", filename); strbuf_release(&tmp_file); + free(offsets); + free(commit_positions); } diff --git a/pack-bitmap.h b/pack-bitmap.h index 3d3ddd77345..67a9d0fc303 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -24,8 +24,9 @@ struct bitmap_disk_header { #define NEEDS_BITMAP (1u<<22) enum pack_bitmap_opts { - BITMAP_OPT_FULL_DAG = 1, - BITMAP_OPT_HASH_CACHE = 4, + BITMAP_OPT_FULL_DAG = 0x1, + BITMAP_OPT_HASH_CACHE = 0x4, + BITMAP_OPT_LOOKUP_TABLE = 0x10, }; enum pack_bitmap_flags { From patchwork Wed Jul 20 18:38:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Abhradeep Chakraborty X-Patchwork-Id: 12924397 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4DBFC433EF for ; Wed, 20 Jul 2022 18:39:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232748AbiGTSjC (ORCPT ); Wed, 20 Jul 2022 14:39:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231320AbiGTSit (ORCPT ); Wed, 20 Jul 2022 14:38:49 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D93F7392E for ; Wed, 20 Jul 2022 11:38:32 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id bu1so27342320wrb.9 for ; Wed, 20 Jul 2022 11:38:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=HP8e6+r2gOIEwMZTiyu3oAYFuF6lHU29ObNQnsjbIWk=; b=KpSCKrgTILUT8ZORatJmJwxVbjN8EwaesNOr3Kji20Bi61wQdloTgizzAsZpJdkvY3 kc3cTq2HMghmPaVxmfXAw+fWfOX5uNk4+1uAKyNENC5K30mbVa/ZsjB8ZK4SpGiSontD MKUfCG4Vg92lSutPTUtw+wqdt4rMXrmk9VvInQkQKiKVz8q/mn96khPlgEBDnYaKdzr6 BaQi9Im31nIR8pQBuwqOGJV08rgF0gHbb21dSITQdJXkWC8mCcZmYM0+h+gRGn/0gZuY TNoPRTXCPqi0puXsbbVJ9x2YCr17r2w1gp1IHMJrgVayXn3d1srYwzSlM7B46tzUB1oX 98XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=HP8e6+r2gOIEwMZTiyu3oAYFuF6lHU29ObNQnsjbIWk=; b=fd/L6ThLmKlmTM6LZwAH2fWkH5+iGi9TYNSFLqfHwVEeEGPj4PWR2dnRp9iP38kXEl 7JaWCuBU9gXnxQKgoyrT8aQ8lXLlpoGHyK+c/tqmpK6LoDAAzBfR0laE19uzK+tBCCfd s4BI8EyBjwpA/0fA2HpUgyKYmFxJ0EIfFetvqi5LdSP4Yhso+lq4hq7TVbJ+6W9G9lil NtjCFYiJ/H0Mq6rHiJCiRatz7hiqkoiJJOsaJQWrccR1wY9otBTb6e7MBc+BGEjJLYw5 593e+EHidPHUK8RQKVtfI/4Iz3guiDlzX5IrMmAhVCRdSDrG6eieFQWxq8FtR5vygLck NzXg== X-Gm-Message-State: AJIora8VdkEG7KXXJrMZWSlv6JUeKliFuQHC7ESUpJIUbqCEDLmQeLQx tJ0I+UOjSCXqKfFED9ZrHIEKZYqIsNE= X-Google-Smtp-Source: AGRyM1tPuG9ahMwDILumVBMXUpk78q+T0P9wn7qWeVLjD0aVXSlERCw1v0a0S8NrtDlpibeRRvjP9A== X-Received: by 2002:adf:ce88:0:b0:21e:492f:b806 with SMTP id r8-20020adfce88000000b0021e492fb806mr3377757wrn.635.1658342310305; Wed, 20 Jul 2022 11:38:30 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z17-20020a7bc7d1000000b003a2e27fc275sm3045527wmk.12.2022.07.20.11.38.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Jul 2022 11:38:29 -0700 (PDT) Message-Id: <59b465e5a7817c145172f25e73ad807c7ba67e84.1658342304.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Jul 2022 18:38:21 +0000 Subject: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Taylor Blau , Kaartic Sivaram , Derrick Stolee , Philip Oakley , Martin =?utf-8?b?w4VncmVu?= , Abhradeep Chakraborty , Abhradeep Chakraborty Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Abhradeep Chakraborty From: Abhradeep Chakraborty Teach Git to provide a way for users to enable/disable bitmap lookup table extension by providing a config option named 'writeBitmapLookupTable'. Default is false. Also add test to verify writting of lookup table. Mentored-by: Taylor Blau Co-Mentored-by: Kaartic Sivaraam Co-Authored-by: Taylor Blau Signed-off-by: Abhradeep Chakraborty --- Documentation/config/pack.txt | 7 + builtin/multi-pack-index.c | 7 + builtin/pack-objects.c | 8 + midx.c | 3 + midx.h | 1 + t/t5310-pack-bitmaps.sh | 792 ++++++++++++++++-------------- t/t5311-pack-bitmaps-shallow.sh | 53 +- t/t5326-multi-pack-bitmaps.sh | 421 +++++++++------- t/t5327-multi-pack-bitmaps-rev.sh | 24 +- 9 files changed, 733 insertions(+), 583 deletions(-) diff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt index ad7f73a1ead..b955ca572ec 100644 --- a/Documentation/config/pack.txt +++ b/Documentation/config/pack.txt @@ -164,6 +164,13 @@ When writing a multi-pack reachability bitmap, no new namehashes are computed; instead, any namehashes stored in an existing bitmap are permuted into their appropriate location when writing a new bitmap. +pack.writeBitmapLookupTable:: + When true, Git will include a "lookup table" section in the + bitmap index (if one is written). This table is used to defer + loading individual bitmaps as late as possible. This can be + beneficial in repositories that have relatively large bitmap + indexes. Defaults to false. + pack.writeReverseIndex:: When true, git will write a corresponding .rev file (see: link:../technical/pack-format.html[Documentation/technical/pack-format.txt]) diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index 5edbb7fe86e..55402b46f41 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -87,6 +87,13 @@ static int git_multi_pack_index_write_config(const char *var, const char *value, opts.flags &= ~MIDX_WRITE_BITMAP_HASH_CACHE; } + if (!strcmp(var, "pack.writebitmaplookuptable")) { + if (git_config_bool(var, value)) + opts.flags |= MIDX_WRITE_BITMAP_LOOKUP_TABLE; + else + opts.flags &= ~MIDX_WRITE_BITMAP_LOOKUP_TABLE; + } + /* * We should never make a fall-back call to 'git_default_config', since * this was already called in 'cmd_multi_pack_index()'. diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 39e28cfcafc..46e26774963 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3148,6 +3148,14 @@ static int git_pack_config(const char *k, const char *v, void *cb) else write_bitmap_options &= ~BITMAP_OPT_HASH_CACHE; } + + if (!strcmp(k, "pack.writebitmaplookuptable")) { + if (git_config_bool(k, v)) + write_bitmap_options |= BITMAP_OPT_LOOKUP_TABLE; + else + write_bitmap_options &= ~BITMAP_OPT_LOOKUP_TABLE; + } + if (!strcmp(k, "pack.usebitmaps")) { use_bitmap_index_default = git_config_bool(k, v); return 0; diff --git a/midx.c b/midx.c index 5f0dd386b02..9c26d04bfde 100644 --- a/midx.c +++ b/midx.c @@ -1072,6 +1072,9 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash, if (flags & MIDX_WRITE_BITMAP_HASH_CACHE) options |= BITMAP_OPT_HASH_CACHE; + if (flags & MIDX_WRITE_BITMAP_LOOKUP_TABLE) + options |= BITMAP_OPT_LOOKUP_TABLE; + prepare_midx_packing_data(&pdata, ctx); commits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, ctx); diff --git a/midx.h b/midx.h index 22e8e53288e..5578cd7b835 100644 --- a/midx.h +++ b/midx.h @@ -47,6 +47,7 @@ struct multi_pack_index { #define MIDX_WRITE_REV_INDEX (1 << 1) #define MIDX_WRITE_BITMAP (1 << 2) #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3) +#define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4) const unsigned char *get_midx_checksum(struct multi_pack_index *m); void get_midx_filename(struct strbuf *out, const char *object_dir); diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh index f775fc1ce69..c0607172827 100755 --- a/t/t5310-pack-bitmaps.sh +++ b/t/t5310-pack-bitmaps.sh @@ -26,22 +26,413 @@ has_any () { grep -Ff "$1" "$2" } -setup_bitmap_history - -test_expect_success 'setup writing bitmaps during repack' ' - git config repack.writeBitmaps true -' - -test_expect_success 'full repack creates bitmaps' ' - GIT_TRACE2_EVENT="$(pwd)/trace" \ +test_bitmap_cases () { + writeLookupTable=false + for i in "$@" + do + case "$i" in + "pack.writeBitmapLookupTable") writeLookupTable=true;; + esac + done + + test_expect_success 'setup test repository' ' + rm -fr * .git && + git init && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' + ' + setup_bitmap_history + + test_expect_success 'setup writing bitmaps during repack' ' + git config repack.writeBitmaps true + ' + + test_expect_success 'full repack creates bitmaps' ' + GIT_TRACE2_EVENT="$(pwd)/trace" \ + git repack -ad && + ls .git/objects/pack/ | grep bitmap >output && + test_line_count = 1 output && + grep "\"key\":\"num_selected_commits\",\"value\":\"106\"" trace && + grep "\"key\":\"num_maximal_commits\",\"value\":\"107\"" trace + ' + + basic_bitmap_tests + + test_expect_success 'pack-objects respects --local (non-local loose)' ' + git init --bare alt.git && + echo $(pwd)/alt.git/objects >.git/objects/info/alternates && + echo content1 >file1 && + # non-local loose object which is not present in bitmapped pack + altblob=$(GIT_DIR=alt.git git hash-object -w file1) && + # non-local loose object which is also present in bitmapped pack + git cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin && + git add file1 && + test_tick && + git commit -m commit_file1 && + echo HEAD | git pack-objects --local --stdout --revs >1.pack && + git index-pack 1.pack && + list_packed_objects 1.idx >1.objects && + printf "%s\n" "$altblob" "$blob" >nonlocal-loose && + ! has_any nonlocal-loose 1.objects + ' + + test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' ' + echo content2 >file2 && + blob2=$(git hash-object -w file2) && + git add file2 && + test_tick && + git commit -m commit_file2 && + printf "%s\n" "$blob2" "$bitmaptip" >keepobjects && + pack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep && + rm $(objpath $blob2) && + echo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack && + git index-pack 2a.pack && + list_packed_objects 2a.idx >2a.objects && + ! has_any keepobjects 2a.objects + ' + + test_expect_success 'pack-objects respects --local (non-local pack)' ' + mv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ && + echo HEAD | git pack-objects --local --stdout --revs >2b.pack && + git index-pack 2b.pack && + list_packed_objects 2b.idx >2b.objects && + ! has_any keepobjects 2b.objects + ' + + test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' ' + ls .git/objects/pack/ | grep bitmap >output && + test_line_count = 1 output && + packbitmap=$(basename $(cat output) .bitmap) && + list_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects && + test_when_finished "rm -f .git/objects/pack/$packbitmap.keep" && + >.git/objects/pack/$packbitmap.keep && + echo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack && + git index-pack 3a.pack && + list_packed_objects 3a.idx >3a.objects && + ! has_any packbitmap.objects 3a.objects + ' + + test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' ' + mv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ && + rm -f .git/objects/pack/multi-pack-index && + test_when_finished "mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/" && + echo HEAD | git pack-objects --local --stdout --revs >3b.pack && + git index-pack 3b.pack && + list_packed_objects 3b.idx >3b.objects && + ! has_any packbitmap.objects 3b.objects + ' + + test_expect_success 'pack-objects to file can use bitmap' ' + # make sure we still have 1 bitmap index from previous tests + ls .git/objects/pack/ | grep bitmap >output && + test_line_count = 1 output && + # verify equivalent packs are generated with/without using bitmap index + packasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects && + list_packed_objects packb-$packbsha1.idx >packb.objects && + test_cmp packa.objects packb.objects + ' + + test_expect_success 'full repack, reusing previous bitmaps' ' git repack -ad && - ls .git/objects/pack/ | grep bitmap >output && - test_line_count = 1 output && - grep "\"key\":\"num_selected_commits\",\"value\":\"106\"" trace && - grep "\"key\":\"num_maximal_commits\",\"value\":\"107\"" trace -' + ls .git/objects/pack/ | grep bitmap >output && + test_line_count = 1 output + ' + + test_expect_success 'fetch (full bitmap)' ' + git --git-dir=clone.git fetch origin second:second && + git rev-parse HEAD >expect && + git --git-dir=clone.git rev-parse HEAD >actual && + test_cmp expect actual + ' + + test_expect_success 'create objects for missing-HAVE tests' ' + blob=$(echo "missing have" | git hash-object -w --stdin) && + tree=$(printf "100644 blob $blob\tfile\n" | git mktree) && + parent=$(echo parent | git commit-tree $tree) && + commit=$(echo commit | git commit-tree $tree -p $parent) && + cat >revs <<-EOF + HEAD + ^HEAD^ + ^$commit + EOF + ' + + test_expect_success 'pack-objects respects --incremental' ' + cat >revs2 <<-EOF && + HEAD + $commit + EOF + git pack-objects --incremental --stdout --revs 4.pack && + git index-pack 4.pack && + list_packed_objects 4.idx >4.objects && + test_line_count = 4 4.objects && + git rev-list --objects $commit >revlist && + cut -d" " -f1 revlist |sort >objects && + test_cmp 4.objects objects + ' + + test_expect_success 'pack with missing blob' ' + rm $(objpath $blob) && + git pack-objects --stdout --revs /dev/null + ' + + test_expect_success 'pack with missing tree' ' + rm $(objpath $tree) && + git pack-objects --stdout --revs /dev/null + ' + + test_expect_success 'pack with missing parent' ' + rm $(objpath $parent) && + git pack-objects --stdout --revs /dev/null + ' + + test_expect_success JGIT,SHA1 'we can read jgit bitmaps' ' + git clone --bare . compat-jgit.git && + ( + cd compat-jgit.git && + rm -f objects/pack/*.bitmap && + jgit gc && + git rev-list --test-bitmap HEAD + ) + ' + + test_expect_success JGIT,SHA1 'jgit can read our bitmaps' ' + git clone --bare . compat-us.git && + ( + cd compat-us.git && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && + git repack -adb && + # jgit gc will barf if it does not like our bitmaps + jgit gc + ) + ' + + test_expect_success 'splitting packs does not generate bogus bitmaps' ' + test-tool genrandom foo $((1024 * 1024)) >rand && + git add rand && + git commit -m "commit with big file" && + git -c pack.packSizeLimit=500k repack -adb && + git init --bare no-bitmaps.git && + git -C no-bitmaps.git fetch .. HEAD + ' + + test_expect_success 'set up reusable pack' ' + rm -f .git/objects/pack/*.keep && + git repack -adb && + reusable_pack () { + git for-each-ref --format="%(objectname)" | + git pack-objects --delta-base-offset --revs --stdout "$@" + } + ' + + test_expect_success 'pack reuse respects --honor-pack-keep' ' + test_when_finished "rm -f .git/objects/pack/*.keep" && + for i in .git/objects/pack/*.pack + do + >${i%.pack}.keep || return 1 + done && + reusable_pack --honor-pack-keep >empty.pack && + git index-pack empty.pack && + git show-index actual && + test_must_be_empty actual + ' + + test_expect_success 'pack reuse respects --local' ' + mv .git/objects/pack/* alt.git/objects/pack/ && + test_when_finished "mv alt.git/objects/pack/* .git/objects/pack/" && + reusable_pack --local >empty.pack && + git index-pack empty.pack && + git show-index actual && + test_must_be_empty actual + ' + + test_expect_success 'pack reuse respects --incremental' ' + reusable_pack --incremental >empty.pack && + git index-pack empty.pack && + git show-index actual && + test_must_be_empty actual + ' + + test_expect_success 'truncated bitmap fails gracefully (ewah)' ' + test_config pack.writebitmaphashcache false && + git repack -ad && + git rev-list --use-bitmap-index --count --all >expect && + bitmap=$(ls .git/objects/pack/*.bitmap) && + test_when_finished "rm -f $bitmap" && + test_copy_bytes 256 <$bitmap >$bitmap.tmp && + mv -f $bitmap.tmp $bitmap && + git rev-list --use-bitmap-index --count --all >actual 2>stderr && + test_cmp expect actual && + test_i18ngrep corrupt.ewah.bitmap stderr + ' + + test_expect_success 'truncated bitmap fails gracefully (cache)' ' + git repack -ad && + git rev-list --use-bitmap-index --count --all >expect && + bitmap=$(ls .git/objects/pack/*.bitmap) && + test_when_finished "rm -f $bitmap" && + test_copy_bytes 512 <$bitmap >$bitmap.tmp && + mv -f $bitmap.tmp $bitmap && + git rev-list --use-bitmap-index --count --all >actual 2>stderr && + test_cmp expect actual && + test_i18ngrep corrupted.bitmap.index stderr + ' + + # Create a state of history with these properties: + # + # - refs that allow a client to fetch some new history, while sharing some old + # history with the server; we use branches delta-reuse-old and + # delta-reuse-new here + # + # - the new history contains an object that is stored on the server as a delta + # against a base that is in the old history + # + # - the base object is not immediately reachable from the tip of the old + # history; finding it would involve digging down through history we know the + # other side has + # + # This should result in a state where fetching from old->new would not + # traditionally reuse the on-disk delta (because we'd have to dig to realize + # that the client has it), but we will do so if bitmaps can tell us cheaply + # that the other side has it. + test_expect_success 'set up thin delta-reuse parent' ' + # This first commit contains the buried base object. + test-tool genrandom delta 16384 >file && + git add file && + git commit -m "delta base" && + base=$(git rev-parse --verify HEAD:file) && + + # These intermediate commits bury the base back in history. + # This becomes the "old" state. + for i in 1 2 3 4 5 + do + echo $i >file && + git commit -am "intermediate $i" || return 1 + done && + git branch delta-reuse-old && + + # And now our new history has a delta against the buried base. Note + # that this must be smaller than the original file, since pack-objects + # prefers to create deltas from smaller objects to larger. + test-tool genrandom delta 16300 >file && + git commit -am "delta result" && + delta=$(git rev-parse --verify HEAD:file) && + git branch delta-reuse-new && + + # Repack with bitmaps and double check that we have the expected delta + # relationship. + git repack -adb && + have_delta $delta $base + ' + + # Now we can sanity-check the non-bitmap behavior (that the server is not able + # to reuse the delta). This isn't strictly something we care about, so this + # test could be scrapped in the future. But it makes sure that the next test is + # actually triggering the feature we want. + # + # Note that our tools for working with on-the-wire "thin" packs are limited. So + # we actually perform the fetch, retain the resulting pack, and inspect the + # result. + test_expect_success 'fetch without bitmaps ignores delta against old base' ' + test_config pack.usebitmaps false && + test_when_finished "rm -rf client.git" && + git init --bare client.git && + ( + cd client.git && + git config transfer.unpackLimit 1 && + git fetch .. delta-reuse-old:delta-reuse-old && + git fetch .. delta-reuse-new:delta-reuse-new && + have_delta $delta $ZERO_OID + ) + ' + + # And do the same for the bitmap case, where we do expect to find the delta. + test_expect_success 'fetch with bitmaps can reuse old base' ' + test_config pack.usebitmaps true && + test_when_finished "rm -rf client.git" && + git init --bare client.git && + ( + cd client.git && + git config transfer.unpackLimit 1 && + git fetch .. delta-reuse-old:delta-reuse-old && + git fetch .. delta-reuse-new:delta-reuse-new && + have_delta $delta $base + ) + ' + + test_expect_success 'pack.preferBitmapTips' ' + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && + + # create enough commits that not all are receive bitmap + # coverage even if they are all at the tip of some reference. + test_commit_bulk --message="%s" 103 && + + git rev-list HEAD >commits.raw && + sort commits && + + git log --format="create refs/tags/%s %H" HEAD >refs && + git update-ref --stdin bitmaps && + + # remember which commits did not receive bitmaps + comm -13 bitmaps commits >before && + test_file_not_empty before && + + # mark the commits which did not receive bitmaps as preferred, + # and generate the bitmap again + perl -pe "s{^}{create refs/tags/include/$. }" bitmaps && + comm -13 bitmaps commits >after && + + ! test_cmp before after + ) + ' + + test_expect_success 'complains about multiple pack bitmaps' ' + rm -fr repo && + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && + + test_commit base && + + git repack -adb && + bitmap="$(ls .git/objects/pack/pack-*.bitmap)" && + mv "$bitmap" "$bitmap.bak" && + + test_commit other && + git repack -ab && + + mv "$bitmap.bak" "$bitmap" && + + find .git/objects/pack -type f -name "*.pack" >packs && + find .git/objects/pack -type f -name "*.bitmap" >bitmaps && + test_line_count = 2 packs && + test_line_count = 2 bitmaps && + + git rev-list --use-bitmap-index HEAD 2>err && + grep "ignoring extra bitmap file" err + ) + ' +} -basic_bitmap_tests +test_bitmap_cases test_expect_success 'incremental repack fails when bitmaps are requested' ' test_commit more-1 && @@ -54,375 +445,12 @@ test_expect_success 'incremental repack can disable bitmaps' ' git repack -d --no-write-bitmap-index ' -test_expect_success 'pack-objects respects --local (non-local loose)' ' - git init --bare alt.git && - echo $(pwd)/alt.git/objects >.git/objects/info/alternates && - echo content1 >file1 && - # non-local loose object which is not present in bitmapped pack - altblob=$(GIT_DIR=alt.git git hash-object -w file1) && - # non-local loose object which is also present in bitmapped pack - git cat-file blob $blob | GIT_DIR=alt.git git hash-object -w --stdin && - git add file1 && - test_tick && - git commit -m commit_file1 && - echo HEAD | git pack-objects --local --stdout --revs >1.pack && - git index-pack 1.pack && - list_packed_objects 1.idx >1.objects && - printf "%s\n" "$altblob" "$blob" >nonlocal-loose && - ! has_any nonlocal-loose 1.objects -' - -test_expect_success 'pack-objects respects --honor-pack-keep (local non-bitmapped pack)' ' - echo content2 >file2 && - blob2=$(git hash-object -w file2) && - git add file2 && - test_tick && - git commit -m commit_file2 && - printf "%s\n" "$blob2" "$bitmaptip" >keepobjects && - pack2=$(git pack-objects pack2 .git/objects/pack/pack2-$pack2.keep && - rm $(objpath $blob2) && - echo HEAD | git pack-objects --honor-pack-keep --stdout --revs >2a.pack && - git index-pack 2a.pack && - list_packed_objects 2a.idx >2a.objects && - ! has_any keepobjects 2a.objects -' - -test_expect_success 'pack-objects respects --local (non-local pack)' ' - mv .git/objects/pack/pack2-$pack2.* alt.git/objects/pack/ && - echo HEAD | git pack-objects --local --stdout --revs >2b.pack && - git index-pack 2b.pack && - list_packed_objects 2b.idx >2b.objects && - ! has_any keepobjects 2b.objects -' - -test_expect_success 'pack-objects respects --honor-pack-keep (local bitmapped pack)' ' - ls .git/objects/pack/ | grep bitmap >output && - test_line_count = 1 output && - packbitmap=$(basename $(cat output) .bitmap) && - list_packed_objects .git/objects/pack/$packbitmap.idx >packbitmap.objects && - test_when_finished "rm -f .git/objects/pack/$packbitmap.keep" && - >.git/objects/pack/$packbitmap.keep && - echo HEAD | git pack-objects --honor-pack-keep --stdout --revs >3a.pack && - git index-pack 3a.pack && - list_packed_objects 3a.idx >3a.objects && - ! has_any packbitmap.objects 3a.objects -' - -test_expect_success 'pack-objects respects --local (non-local bitmapped pack)' ' - mv .git/objects/pack/$packbitmap.* alt.git/objects/pack/ && - rm -f .git/objects/pack/multi-pack-index && - test_when_finished "mv alt.git/objects/pack/$packbitmap.* .git/objects/pack/" && - echo HEAD | git pack-objects --local --stdout --revs >3b.pack && - git index-pack 3b.pack && - list_packed_objects 3b.idx >3b.objects && - ! has_any packbitmap.objects 3b.objects -' - -test_expect_success 'pack-objects to file can use bitmap' ' - # make sure we still have 1 bitmap index from previous tests - ls .git/objects/pack/ | grep bitmap >output && - test_line_count = 1 output && - # verify equivalent packs are generated with/without using bitmap index - packasha1=$(git pack-objects --no-use-bitmap-index --all packa packa.objects && - list_packed_objects packb-$packbsha1.idx >packb.objects && - test_cmp packa.objects packb.objects -' - -test_expect_success 'full repack, reusing previous bitmaps' ' - git repack -ad && - ls .git/objects/pack/ | grep bitmap >output && - test_line_count = 1 output -' - -test_expect_success 'fetch (full bitmap)' ' - git --git-dir=clone.git fetch origin second:second && - git rev-parse HEAD >expect && - git --git-dir=clone.git rev-parse HEAD >actual && - test_cmp expect actual -' - -test_expect_success 'create objects for missing-HAVE tests' ' - blob=$(echo "missing have" | git hash-object -w --stdin) && - tree=$(printf "100644 blob $blob\tfile\n" | git mktree) && - parent=$(echo parent | git commit-tree $tree) && - commit=$(echo commit | git commit-tree $tree -p $parent) && - cat >revs <<-EOF - HEAD - ^HEAD^ - ^$commit - EOF -' - -test_expect_success 'pack-objects respects --incremental' ' - cat >revs2 <<-EOF && - HEAD - $commit - EOF - git pack-objects --incremental --stdout --revs 4.pack && - git index-pack 4.pack && - list_packed_objects 4.idx >4.objects && - test_line_count = 4 4.objects && - git rev-list --objects $commit >revlist && - cut -d" " -f1 revlist |sort >objects && - test_cmp 4.objects objects -' - -test_expect_success 'pack with missing blob' ' - rm $(objpath $blob) && - git pack-objects --stdout --revs /dev/null -' +test_bitmap_cases "pack.writeBitmapLookupTable" -test_expect_success 'pack with missing tree' ' - rm $(objpath $tree) && - git pack-objects --stdout --revs /dev/null -' - -test_expect_success 'pack with missing parent' ' - rm $(objpath $parent) && - git pack-objects --stdout --revs /dev/null -' - -test_expect_success JGIT,SHA1 'we can read jgit bitmaps' ' - git clone --bare . compat-jgit.git && - ( - cd compat-jgit.git && - rm -f objects/pack/*.bitmap && - jgit gc && - git rev-list --test-bitmap HEAD - ) -' - -test_expect_success JGIT,SHA1 'jgit can read our bitmaps' ' - git clone --bare . compat-us.git && - ( - cd compat-us.git && - git repack -adb && - # jgit gc will barf if it does not like our bitmaps - jgit gc - ) -' - -test_expect_success 'splitting packs does not generate bogus bitmaps' ' - test-tool genrandom foo $((1024 * 1024)) >rand && - git add rand && - git commit -m "commit with big file" && - git -c pack.packSizeLimit=500k repack -adb && - git init --bare no-bitmaps.git && - git -C no-bitmaps.git fetch .. HEAD -' - -test_expect_success 'set up reusable pack' ' - rm -f .git/objects/pack/*.keep && - git repack -adb && - reusable_pack () { - git for-each-ref --format="%(objectname)" | - git pack-objects --delta-base-offset --revs --stdout "$@" - } -' - -test_expect_success 'pack reuse respects --honor-pack-keep' ' - test_when_finished "rm -f .git/objects/pack/*.keep" && - for i in .git/objects/pack/*.pack - do - >${i%.pack}.keep || return 1 - done && - reusable_pack --honor-pack-keep >empty.pack && - git index-pack empty.pack && - git show-index actual && - test_must_be_empty actual -' - -test_expect_success 'pack reuse respects --local' ' - mv .git/objects/pack/* alt.git/objects/pack/ && - test_when_finished "mv alt.git/objects/pack/* .git/objects/pack/" && - reusable_pack --local >empty.pack && - git index-pack empty.pack && - git show-index actual && - test_must_be_empty actual -' - -test_expect_success 'pack reuse respects --incremental' ' - reusable_pack --incremental >empty.pack && - git index-pack empty.pack && - git show-index actual && - test_must_be_empty actual -' - -test_expect_success 'truncated bitmap fails gracefully (ewah)' ' - test_config pack.writebitmaphashcache false && - git repack -ad && - git rev-list --use-bitmap-index --count --all >expect && - bitmap=$(ls .git/objects/pack/*.bitmap) && - test_when_finished "rm -f $bitmap" && - test_copy_bytes 256 <$bitmap >$bitmap.tmp && - mv -f $bitmap.tmp $bitmap && - git rev-list --use-bitmap-index --count --all >actual 2>stderr && - test_cmp expect actual && - test_i18ngrep corrupt.ewah.bitmap stderr -' - -test_expect_success 'truncated bitmap fails gracefully (cache)' ' - git repack -ad && - git rev-list --use-bitmap-index --count --all >expect && - bitmap=$(ls .git/objects/pack/*.bitmap) && - test_when_finished "rm -f $bitmap" && - test_copy_bytes 512 <$bitmap >$bitmap.tmp && - mv -f $bitmap.tmp $bitmap && - git rev-list --use-bitmap-index --count --all >actual 2>stderr && - test_cmp expect actual && - test_i18ngrep corrupted.bitmap.index stderr -' - -# Create a state of history with these properties: -# -# - refs that allow a client to fetch some new history, while sharing some old -# history with the server; we use branches delta-reuse-old and -# delta-reuse-new here -# -# - the new history contains an object that is stored on the server as a delta -# against a base that is in the old history -# -# - the base object is not immediately reachable from the tip of the old -# history; finding it would involve digging down through history we know the -# other side has -# -# This should result in a state where fetching from old->new would not -# traditionally reuse the on-disk delta (because we'd have to dig to realize -# that the client has it), but we will do so if bitmaps can tell us cheaply -# that the other side has it. -test_expect_success 'set up thin delta-reuse parent' ' - # This first commit contains the buried base object. - test-tool genrandom delta 16384 >file && - git add file && - git commit -m "delta base" && - base=$(git rev-parse --verify HEAD:file) && - - # These intermediate commits bury the base back in history. - # This becomes the "old" state. - for i in 1 2 3 4 5 - do - echo $i >file && - git commit -am "intermediate $i" || return 1 - done && - git branch delta-reuse-old && - - # And now our new history has a delta against the buried base. Note - # that this must be smaller than the original file, since pack-objects - # prefers to create deltas from smaller objects to larger. - test-tool genrandom delta 16300 >file && - git commit -am "delta result" && - delta=$(git rev-parse --verify HEAD:file) && - git branch delta-reuse-new && - - # Repack with bitmaps and double check that we have the expected delta - # relationship. - git repack -adb && - have_delta $delta $base -' - -# Now we can sanity-check the non-bitmap behavior (that the server is not able -# to reuse the delta). This isn't strictly something we care about, so this -# test could be scrapped in the future. But it makes sure that the next test is -# actually triggering the feature we want. -# -# Note that our tools for working with on-the-wire "thin" packs are limited. So -# we actually perform the fetch, retain the resulting pack, and inspect the -# result. -test_expect_success 'fetch without bitmaps ignores delta against old base' ' - test_config pack.usebitmaps false && - test_when_finished "rm -rf client.git" && - git init --bare client.git && - ( - cd client.git && - git config transfer.unpackLimit 1 && - git fetch .. delta-reuse-old:delta-reuse-old && - git fetch .. delta-reuse-new:delta-reuse-new && - have_delta $delta $ZERO_OID - ) -' - -# And do the same for the bitmap case, where we do expect to find the delta. -test_expect_success 'fetch with bitmaps can reuse old base' ' - test_config pack.usebitmaps true && - test_when_finished "rm -rf client.git" && - git init --bare client.git && - ( - cd client.git && - git config transfer.unpackLimit 1 && - git fetch .. delta-reuse-old:delta-reuse-old && - git fetch .. delta-reuse-new:delta-reuse-new && - have_delta $delta $base - ) -' - -test_expect_success 'pack.preferBitmapTips' ' - git init repo && - test_when_finished "rm -fr repo" && - ( - cd repo && - - # create enough commits that not all are receive bitmap - # coverage even if they are all at the tip of some reference. - test_commit_bulk --message="%s" 103 && - - git rev-list HEAD >commits.raw && - sort commits && - - git log --format="create refs/tags/%s %H" HEAD >refs && - git update-ref --stdin bitmaps && - - # remember which commits did not receive bitmaps - comm -13 bitmaps commits >before && - test_file_not_empty before && - - # mark the commits which did not receive bitmaps as preferred, - # and generate the bitmap again - perl -pe "s{^}{create refs/tags/include/$. }" bitmaps && - comm -13 bitmaps commits >after && - - ! test_cmp before after - ) -' - -test_expect_success 'complains about multiple pack bitmaps' ' - rm -fr repo && - git init repo && - test_when_finished "rm -fr repo" && - ( - cd repo && - - test_commit base && - - git repack -adb && - bitmap="$(ls .git/objects/pack/pack-*.bitmap)" && - mv "$bitmap" "$bitmap.bak" && - - test_commit other && - git repack -ab && - - mv "$bitmap.bak" "$bitmap" && - - find .git/objects/pack -type f -name "*.pack" >packs && - find .git/objects/pack -type f -name "*.bitmap" >bitmaps && - test_line_count = 2 packs && - test_line_count = 2 bitmaps && - - git rev-list --use-bitmap-index HEAD 2>err && - grep "ignoring extra bitmap file" err - ) +test_expect_success 'verify writing bitmap lookup table when enabled' ' + GIT_TRACE2_EVENT="$(pwd)/trace2" \ + git repack -ad && + grep "\"label\":\"writing_lookup_table\"" trace2 ' test_done diff --git a/t/t5311-pack-bitmaps-shallow.sh b/t/t5311-pack-bitmaps-shallow.sh index 872a95df338..9dae60f73e3 100755 --- a/t/t5311-pack-bitmaps-shallow.sh +++ b/t/t5311-pack-bitmaps-shallow.sh @@ -17,23 +17,40 @@ test_description='check bitmap operation with shallow repositories' # the tree for A. But in a shallow one, we've grafted away # A, and fetching A to B requires that the other side send # us the tree for file=1. -test_expect_success 'setup shallow repo' ' - echo 1 >file && - git add file && - git commit -m orig && - echo 2 >file && - git commit -a -m update && - git clone --no-local --bare --depth=1 . shallow.git && - echo 1 >file && - git commit -a -m repeat -' - -test_expect_success 'turn on bitmaps in the parent' ' - git repack -adb -' - -test_expect_success 'shallow fetch from bitmapped repo' ' - (cd shallow.git && git fetch) -' +test_shallow_bitmaps () { + writeLookupTable=false + + for i in "$@" + do + case $i in + "pack.writeBitmapLookupTable") writeLookupTable=true;; + esac + done + + test_expect_success 'setup shallow repo' ' + rm -rf * .git && + git init && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && + echo 1 >file && + git add file && + git commit -m orig && + echo 2 >file && + git commit -a -m update && + git clone --no-local --bare --depth=1 . shallow.git && + echo 1 >file && + git commit -a -m repeat + ' + + test_expect_success 'turn on bitmaps in the parent' ' + git repack -adb + ' + + test_expect_success 'shallow fetch from bitmapped repo' ' + (cd shallow.git && git fetch) + ' +} + +test_shallow_bitmaps +test_shallow_bitmaps "pack.writeBitmapLookupTable" test_done diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh index 4fe57414c13..3b206adcee6 100755 --- a/t/t5326-multi-pack-bitmaps.sh +++ b/t/t5326-multi-pack-bitmaps.sh @@ -15,17 +15,24 @@ GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 sane_unset GIT_TEST_MIDX_WRITE_REV sane_unset GIT_TEST_MIDX_READ_RIDX -midx_bitmap_core - bitmap_reuse_tests() { from=$1 to=$2 + writeLookupTable=false + + for i in $3-${$#} + do + case $i in + "pack.writeBitmapLookupTable") writeLookupTable=true;; + esac + done test_expect_success "setup pack reuse tests ($from -> $to)" ' rm -fr repo && git init repo && ( cd repo && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && test_commit_bulk 16 && git tag old-tip && @@ -43,6 +50,7 @@ bitmap_reuse_tests() { test_expect_success "build bitmap from existing ($from -> $to)" ' ( cd repo && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && test_commit_bulk --id=further 16 && git tag new-tip && @@ -59,6 +67,7 @@ bitmap_reuse_tests() { test_expect_success "verify resulting bitmaps ($from -> $to)" ' ( cd repo && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && git for-each-ref && git rev-list --test-bitmap refs/tags/old-tip && git rev-list --test-bitmap refs/tags/new-tip @@ -66,244 +75,294 @@ bitmap_reuse_tests() { ' } -bitmap_reuse_tests 'pack' 'MIDX' -bitmap_reuse_tests 'MIDX' 'pack' -bitmap_reuse_tests 'MIDX' 'MIDX' +test_midx_bitmap_cases () { + writeLookupTable=false + writeBitmapLookupTable= + + for i in "$@" + do + case $i in + "pack.writeBitmapLookupTable") + writeLookupTable=true + writeBitmapLookupTable="$i" + ;; + esac + done + + test_expect_success 'setup test_repository' ' + rm -rf * .git && + git init && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' + ' -test_expect_success 'missing object closure fails gracefully' ' - rm -fr repo && - git init repo && - test_when_finished "rm -fr repo" && - ( - cd repo && + midx_bitmap_core - test_commit loose && - test_commit packed && + bitmap_reuse_tests 'pack' 'MIDX' "$writeBitmapLookupTable" + bitmap_reuse_tests 'MIDX' 'pack' "$writeBitmapLookupTable" + bitmap_reuse_tests 'MIDX' 'MIDX' "$writeBitmapLookupTable" - # Do not pass "--revs"; we want a pack without the "loose" - # commit. - git pack-objects $objdir/pack/pack <<-EOF && - $(git rev-parse packed) - EOF + test_expect_success 'missing object closure fails gracefully' ' + rm -fr repo && + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && - test_must_fail git multi-pack-index write --bitmap 2>err && - grep "doesn.t have full closure" err && - test_path_is_missing $midx - ) -' + test_commit loose && + test_commit packed && -midx_bitmap_partial_tests + # Do not pass "--revs"; we want a pack without the "loose" + # commit. + git pack-objects $objdir/pack/pack <<-EOF && + $(git rev-parse packed) + EOF -test_expect_success 'removing a MIDX clears stale bitmaps' ' - rm -fr repo && - git init repo && - test_when_finished "rm -fr repo" && - ( - cd repo && - test_commit base && - git repack && - git multi-pack-index write --bitmap && + test_must_fail git multi-pack-index write --bitmap 2>err && + grep "doesn.t have full closure" err && + test_path_is_missing $midx + ) + ' - # Write a MIDX and bitmap; remove the MIDX but leave the bitmap. - stale_bitmap=$midx-$(midx_checksum $objdir).bitmap && - rm $midx && + midx_bitmap_partial_tests - # Then write a new MIDX. - test_commit new && - git repack && - git multi-pack-index write --bitmap && + test_expect_success 'removing a MIDX clears stale bitmaps' ' + rm -fr repo && + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && + test_commit base && + git repack && + git multi-pack-index write --bitmap && + + # Write a MIDX and bitmap; remove the MIDX but leave the bitmap. + stale_bitmap=$midx-$(midx_checksum $objdir).bitmap && + rm $midx && + + # Then write a new MIDX. + test_commit new && + git repack && + git multi-pack-index write --bitmap && + + test_path_is_file $midx && + test_path_is_file $midx-$(midx_checksum $objdir).bitmap && + test_path_is_missing $stale_bitmap + ) + ' - test_path_is_file $midx && - test_path_is_file $midx-$(midx_checksum $objdir).bitmap && - test_path_is_missing $stale_bitmap - ) -' + test_expect_success 'pack.preferBitmapTips' ' + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && -test_expect_success 'pack.preferBitmapTips' ' - git init repo && - test_when_finished "rm -fr repo" && - ( - cd repo && + test_commit_bulk --message="%s" 103 && - test_commit_bulk --message="%s" 103 && + git log --format="%H" >commits.raw && + sort commits && - git log --format="%H" >commits.raw && - sort commits && + git log --format="create refs/tags/%s %H" HEAD >refs && + git update-ref --stdin refs && - git update-ref --stdin bitmaps && + comm -13 bitmaps commits >before && + test_line_count = 1 before && - test-tool bitmap list-commits | sort >bitmaps && - comm -13 bitmaps commits >before && - test_line_count = 1 before && + perl -ne "printf(\"create refs/tags/include/%d \", $.); print" \ + bitmaps && + comm -13 bitmaps commits >after && - git -c pack.preferBitmapTips=refs/tags/include \ - multi-pack-index write --bitmap && - test-tool bitmap list-commits | sort >bitmaps && - comm -13 bitmaps commits >after && + ! test_cmp before after + ) + ' - ! test_cmp before after - ) -' + test_expect_success 'writing a bitmap with --refs-snapshot' ' + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && -test_expect_success 'writing a bitmap with --refs-snapshot' ' - git init repo && - test_when_finished "rm -fr repo" && - ( - cd repo && + test_commit one && + test_commit two && - test_commit one && - test_commit two && + git rev-parse one >snapshot && - git rev-parse one >snapshot && + git repack -ad && - git repack -ad && + # First, write a MIDX which see both refs/tags/one and + # refs/tags/two (causing both of those commits to receive + # bitmaps). + git multi-pack-index write --bitmap && - # First, write a MIDX which see both refs/tags/one and - # refs/tags/two (causing both of those commits to receive - # bitmaps). - git multi-pack-index write --bitmap && + test_path_is_file $midx && + test_path_is_file $midx-$(midx_checksum $objdir).bitmap && - test_path_is_file $midx && - test_path_is_file $midx-$(midx_checksum $objdir).bitmap && + test-tool bitmap list-commits | sort >bitmaps && + grep "$(git rev-parse one)" bitmaps && + grep "$(git rev-parse two)" bitmaps && - test-tool bitmap list-commits | sort >bitmaps && - grep "$(git rev-parse one)" bitmaps && - grep "$(git rev-parse two)" bitmaps && + rm -fr $midx-$(midx_checksum $objdir).bitmap && + rm -fr $midx && - rm -fr $midx-$(midx_checksum $objdir).bitmap && - rm -fr $midx && + # Then again, but with a refs snapshot which only sees + # refs/tags/one. + git multi-pack-index write --bitmap --refs-snapshot=snapshot && - # Then again, but with a refs snapshot which only sees - # refs/tags/one. - git multi-pack-index write --bitmap --refs-snapshot=snapshot && + test_path_is_file $midx && + test_path_is_file $midx-$(midx_checksum $objdir).bitmap && - test_path_is_file $midx && - test_path_is_file $midx-$(midx_checksum $objdir).bitmap && + test-tool bitmap list-commits | sort >bitmaps && + grep "$(git rev-parse one)" bitmaps && + ! grep "$(git rev-parse two)" bitmaps + ) + ' - test-tool bitmap list-commits | sort >bitmaps && - grep "$(git rev-parse one)" bitmaps && - ! grep "$(git rev-parse two)" bitmaps - ) -' + test_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' ' + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && -test_expect_success 'write a bitmap with --refs-snapshot (preferred tips)' ' - git init repo && - test_when_finished "rm -fr repo" && - ( - cd repo && + test_commit_bulk --message="%s" 103 && - test_commit_bulk --message="%s" 103 && + git log --format="%H" >commits.raw && + sort commits && - git log --format="%H" >commits.raw && - sort commits && + git log --format="create refs/tags/%s %H" HEAD >refs && + git update-ref --stdin refs && - git update-ref --stdin bitmaps && + comm -13 bitmaps commits >before && + test_line_count = 1 before && - test-tool bitmap list-commits | sort >bitmaps && - comm -13 bitmaps commits >before && - test_line_count = 1 before && + ( + grep -vf before commits.raw && + # mark missing commits as preferred + sed "s/^/+/" before + ) >snapshot && + rm -fr $midx-$(midx_checksum $objdir).bitmap && + rm -fr $midx && + + git multi-pack-index write --bitmap --refs-snapshot=snapshot && + test-tool bitmap list-commits | sort >bitmaps && + comm -13 bitmaps commits >after && + + ! test_cmp before after + ) + ' + + test_expect_success 'hash-cache values are propagated from pack bitmaps' ' + rm -fr repo && + git init repo && + test_when_finished "rm -fr repo" && ( - grep -vf before commits.raw && - # mark missing commits as preferred - sed "s/^/+/" before - ) >snapshot && + cd repo && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && - rm -fr $midx-$(midx_checksum $objdir).bitmap && - rm -fr $midx && + test_commit base && + test_commit base2 && + git repack -adb && - git multi-pack-index write --bitmap --refs-snapshot=snapshot && - test-tool bitmap list-commits | sort >bitmaps && - comm -13 bitmaps commits >after && + test-tool bitmap dump-hashes >pack.raw && + test_file_not_empty pack.raw && + sort pack.raw >pack.hashes && - ! test_cmp before after - ) -' + test_commit new && + git repack && + git multi-pack-index write --bitmap && -test_expect_success 'hash-cache values are propagated from pack bitmaps' ' - rm -fr repo && - git init repo && - test_when_finished "rm -fr repo" && - ( - cd repo && + test-tool bitmap dump-hashes >midx.raw && + sort midx.raw >midx.hashes && - test_commit base && - test_commit base2 && - git repack -adb && + # ensure that every namehash in the pack bitmap can be found in + # the midx bitmap (i.e., that there are no oid-namehash pairs + # unique to the pack bitmap). + comm -23 pack.hashes midx.hashes >dropped.hashes && + test_must_be_empty dropped.hashes + ) + ' - test-tool bitmap dump-hashes >pack.raw && - test_file_not_empty pack.raw && - sort pack.raw >pack.hashes && + test_expect_success 'no .bitmap is written without any objects' ' + rm -fr repo && + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && - test_commit new && - git repack && - git multi-pack-index write --bitmap && + empty="$(git pack-objects $objdir/pack/pack packs <<-EOF && + pack-$empty.idx + EOF - test-tool bitmap dump-hashes >midx.raw && - sort midx.raw >midx.hashes && + git multi-pack-index write --bitmap --stdin-packs \ + err && - # ensure that every namehash in the pack bitmap can be found in - # the midx bitmap (i.e., that there are no oid-namehash pairs - # unique to the pack bitmap). - comm -23 pack.hashes midx.hashes >dropped.hashes && - test_must_be_empty dropped.hashes - ) -' + grep "bitmap without any objects" err && -test_expect_success 'no .bitmap is written without any objects' ' - rm -fr repo && - git init repo && - test_when_finished "rm -fr repo" && - ( - cd repo && + test_path_is_file $midx && + test_path_is_missing $midx-$(midx_checksum $objdir).bitmap + ) + ' + + test_expect_success 'graceful fallback when missing reverse index' ' + rm -fr repo && + git init repo && + test_when_finished "rm -fr repo" && + ( + cd repo && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && - empty="$(git pack-objects $objdir/pack/pack packs <<-EOF && - pack-$empty.idx - EOF + test_commit base && - git multi-pack-index write --bitmap --stdin-packs \ - err && + # write a pack and MIDX bitmap containing base + git repack -adb && + git multi-pack-index write --bitmap && - grep "bitmap without any objects" err && + GIT_TEST_MIDX_READ_RIDX=0 \ + git rev-list --use-bitmap-index HEAD 2>err && + ! grep "ignoring extra bitmap file" err + ) + ' +} - test_path_is_file $midx && - test_path_is_missing $midx-$(midx_checksum $objdir).bitmap - ) -' +test_midx_bitmap_cases + +test_midx_bitmap_cases "pack.writeBitmapLookupTable" -test_expect_success 'graceful fallback when missing reverse index' ' +test_expect_success 'multi-pack-index write writes lookup table if enabled' ' rm -fr repo && git init repo && test_when_finished "rm -fr repo" && ( cd repo && - test_commit base && - - # write a pack and MIDX bitmap containing base - git repack -adb && - git multi-pack-index write --bitmap && - - GIT_TEST_MIDX_READ_RIDX=0 \ - git rev-list --use-bitmap-index HEAD 2>err && - ! grep "ignoring extra bitmap file" err + git config pack.writeBitmapLookupTable true && + git repack -ad && + GIT_TRACE2_EVENT="$(pwd)/trace" \ + git multi-pack-index write --bitmap && + grep "\"label\":\"writing_lookup_table\"" trace ) ' diff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh index d30ba632c87..5ed16a820d1 100755 --- a/t/t5327-multi-pack-bitmaps-rev.sh +++ b/t/t5327-multi-pack-bitmaps-rev.sh @@ -17,7 +17,27 @@ GIT_TEST_MIDX_READ_RIDX=0 export GIT_TEST_MIDX_WRITE_REV export GIT_TEST_MIDX_READ_RIDX -midx_bitmap_core rev -midx_bitmap_partial_tests rev +test_midx_bitmap_rev () { + writeLookupTable=false + + for i in "$@" + do + case $i in + "pack.writeBitmapLookupTable") writeLookupTable=true;; + esac + done + + test_expect_success 'setup bitmap config' ' + rm -rf * .git && + git init && + git config pack.writeBitmapLookupTable '"$writeLookupTable"' + ' + + midx_bitmap_core rev + midx_bitmap_partial_tests rev + } + + test_midx_bitmap_rev + test_midx_bitmap_rev "pack.writeBitmapLookupTable" test_done From patchwork Wed Jul 20 18:38:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Abhradeep Chakraborty X-Patchwork-Id: 12924394 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13F71C43334 for ; Wed, 20 Jul 2022 18:39:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229972AbiGTSi6 (ORCPT ); Wed, 20 Jul 2022 14:38:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55890 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230213AbiGTSis (ORCPT ); Wed, 20 Jul 2022 14:38:48 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ABA0173932 for ; Wed, 20 Jul 2022 11:38:33 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id u14-20020a05600c00ce00b003a323062569so1816732wmm.4 for ; Wed, 20 Jul 2022 11:38:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=yTC6lT9rn/NZmBzJEjQfzVXNMtSSIQT4Lwvu3QNXRGA=; b=MUEJRpS3Z04cNnwm8jKCH1cpg0tXj5H8kN7ciUHlYeXyWuSLW/gkuKj2gAavS+0VWM f07rJIhbiBqvaqYklM0QHGSTQcqaJuirUc4XcgxKK3I327Yu8Bh2C6HSIxtikaKIw/9A aJIBsguoqtUX2oiDsqzCXEbseMZsduF6QgSCu0jVGY3iZNDa79kHQaOCPLSTODCTHInv 2L3G6Zf7Pj/yAtnY2eixSPU75+vqPFKwebig9MbD4hV2ZK4Ff1ekcTtoH7z4r1ZTC9Qa usG1Q8KbzLFXcg2OSjfvxdv19Ug8wUoi5YM8B8X+FA8QPVzJKNzjLuRnaOUxWg8D1FEu WiWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=yTC6lT9rn/NZmBzJEjQfzVXNMtSSIQT4Lwvu3QNXRGA=; b=msxar19mqvuI5JK4rKiAQD3RK2++u9OXwxZYbtvFKMhdDRyIFqSLPJs7XBtE1KAsx1 6blTpc43b2wu8L80z5nzn+EBYj11xfbZmY2QkD+020HS/VYCdHHbwwN76bzkVyO9tEXT 1LzhaG/KoIyCB1NCsYBbtjfJNOVWVqB69Q8wIUPu1o1GildWkFWor9lulWTQ/Z6rrt6J 19QNosoc+YgoxP9CBtXbtGlMeaPoIVEEVBiDjUF/pO9uFVTMUxShjp94y6nwQGsXnoqT 3RkKf8uI6/a9RWshK55UZeA8/N0SOljbfihr0+NUbnCgB4MKW2sXZA7ElXa26+0qJi0r euzw== X-Gm-Message-State: AJIora81WVeQhJmnDVSnrLmwTINavkG4MXAA1cIrdXrjqjhw7H5/fRSr dwgWknFBrp4tcMaOMr0ozt4TIcffvq4= X-Google-Smtp-Source: AGRyM1vN9Hbhf+iARh2rrHkWgNzb+VsCsu8ci+KudtLe4i4o8WFk8OZKFU9Q+zKEfUSrriQ3Gdg6MA== X-Received: by 2002:a05:600c:892:b0:3a2:fb4c:2381 with SMTP id l18-20020a05600c089200b003a2fb4c2381mr5211353wmp.89.1658342311288; Wed, 20 Jul 2022 11:38:31 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n3-20020adff083000000b0021d928d2388sm10886191wro.85.2022.07.20.11.38.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Jul 2022 11:38:30 -0700 (PDT) Message-Id: <6918f0860adbea1156fb7a9f1a5aedd211872481.1658342304.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Jul 2022 18:38:22 +0000 Subject: [PATCH v5 4/6] pack-bitmap: prepare to read lookup table extension Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Taylor Blau , Kaartic Sivaram , Derrick Stolee , Philip Oakley , Martin =?utf-8?b?w4VncmVu?= , Abhradeep Chakraborty , Abhradeep Chakraborty Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Abhradeep Chakraborty From: Abhradeep Chakraborty Earlier change teaches Git to write bitmap lookup table. But Git does not know how to parse them. Teach Git to parse the existing bitmap lookup table. The older versions of Git are not affected by it. Those versions ignore the lookup table. Mentored-by: Taylor Blau Co-Mentored-by: Kaartic Sivaraam Signed-off-by: Abhradeep Chakraborty --- pack-bitmap.c | 275 ++++++++++++++++++++++++++++++++++++++-- pack-bitmap.h | 9 ++ t/t5310-pack-bitmaps.sh | 22 ++++ 3 files changed, 296 insertions(+), 10 deletions(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index 36134222d7a..c7336397717 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -82,6 +82,12 @@ struct bitmap_index { /* The checksum of the packfile or MIDX; points into map. */ const unsigned char *checksum; + /* + * If not NULL, this point into the commit table extension + * (within the memory mapped region `map`). + */ + unsigned char *table_lookup; + /* * Extended index. * @@ -185,6 +191,16 @@ static int load_bitmap_header(struct bitmap_index *index) index->hashes = (void *)(index_end - cache_size); index_end -= cache_size; } + + if (flags & BITMAP_OPT_LOOKUP_TABLE) { + size_t table_size = st_mult(ntohl(header->entry_count), + BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH); + if (table_size > index_end - index->map - header_size) + return error(_("corrupted bitmap index file (too short to fit lookup table)")); + if (git_env_bool("GIT_TEST_READ_COMMIT_TABLE", 1)) + index->table_lookup = (void *)(index_end - table_size); + index_end -= table_size; + } } index->entry_count = ntohl(header->entry_count); @@ -211,11 +227,13 @@ static struct stored_bitmap *store_bitmap(struct bitmap_index *index, hash_pos = kh_put_oid_map(index->bitmaps, stored->oid, &ret); - /* a 0 return code means the insertion succeeded with no changes, - * because the SHA1 already existed on the map. this is bad, there - * shouldn't be duplicated commits in the index */ + /* + * A 0 return code means the insertion succeeded with no changes, + * because the SHA1 already existed on the map. This is bad, there + * shouldn't be duplicated commits in the index. + */ if (ret == 0) { - error("Duplicate entry in bitmap index: %s", oid_to_hex(oid)); + error(_("duplicate entry in bitmap index: %s"), oid_to_hex(oid)); return NULL; } @@ -470,7 +488,7 @@ static int load_bitmap(struct bitmap_index *bitmap_git) !(bitmap_git->tags = read_bitmap_1(bitmap_git))) goto failed; - if (load_bitmap_entries_v1(bitmap_git) < 0) + if (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0) goto failed; return 0; @@ -557,13 +575,238 @@ struct include_data { struct bitmap *seen; }; +struct bitmap_lookup_table_triplet { + uint32_t commit_pos; + uint64_t offset; + uint32_t xor_row; +}; + +struct bitmap_lookup_table_xor_item { + struct object_id oid; + uint64_t offset; +}; + +/* + * Given a `triplet` struct pointer and pointer `p`, this + * function reads the triplet beginning at `p` into the struct. + * Note that this function assumes that there is enough memory + * left for filling the `triplet` struct from `p`. + */ +static int lookup_table_get_triplet_by_pointer(struct bitmap_lookup_table_triplet *triplet, + const unsigned char *p) +{ + if (!triplet) + return -1; + + triplet->commit_pos = get_be32(p); + p += sizeof(uint32_t); + triplet->offset = get_be64(p); + p += sizeof(uint64_t); + triplet->xor_row = get_be32(p); + return 0; +} + +/* + * This function gets the raw triplet from `row`'th row in the + * lookup table and fills that data to the `triplet`. + */ +static int lookup_table_get_triplet(struct bitmap_index *bitmap_git, + uint32_t pos, + struct bitmap_lookup_table_triplet *triplet) +{ + unsigned char *p = NULL; + if (pos >= bitmap_git->entry_count) + return error(_("corrupt bitmap lookup table: triplet position out of index")); + + p = bitmap_git->table_lookup + st_mult(pos, BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH); + + return lookup_table_get_triplet_by_pointer(triplet, p); +} + +/* + * Searches for a matching triplet. `commit_pos` is a pointer + * to the wanted commit position value. `table_entry` points to + * a triplet in lookup table. The first 4 bytes of each + * triplet (pointed by `table_entry`) are compared with `*commit_pos`. + */ +static int triplet_cmp(const void *commit_pos, const void *table_entry) +{ + + uint32_t a = *(uint32_t *)commit_pos; + uint32_t b = get_be32(table_entry); + if (a > b) + return 1; + else if (a < b) + return -1; + + return 0; +} + +static uint32_t bsearch_pos(struct bitmap_index *bitmap_git, + struct object_id *oid, + uint32_t *result) +{ + int found; + + if (bitmap_is_midx(bitmap_git)) + found = bsearch_midx(oid, bitmap_git->midx, result); + else + found = bsearch_pack(oid, bitmap_git->pack, result); + + return found; +} + +/* + * `bsearch_triplet_by_pos` function searches for the raw triplet + * having commit position same as `commit_pos` and fills `triplet` + * object from the raw triplet. Returns 1 on success and 0 on + * failure. + */ +static int bsearch_triplet_by_pos(uint32_t commit_pos, + struct bitmap_index *bitmap_git, + struct bitmap_lookup_table_triplet *triplet) +{ + unsigned char *p = bsearch(&commit_pos, bitmap_git->table_lookup, bitmap_git->entry_count, + BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH, triplet_cmp); + + if (!p) + return -1; + + return lookup_table_get_triplet_by_pointer(triplet, p); +} + +static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_git, + struct commit *commit) +{ + uint32_t commit_pos, xor_row; + uint64_t offset; + int flags, found; + struct bitmap_lookup_table_triplet triplet; + struct object_id *oid = &commit->object.oid; + struct ewah_bitmap *bitmap; + struct stored_bitmap *xor_bitmap = NULL; + const int bitmap_header_size = 6; + static struct bitmap_lookup_table_xor_item *xor_items = NULL; + static size_t xor_items_nr = 0, xor_items_alloc = 0; + static int is_corrupt = 0; + + if (is_corrupt) + return NULL; + + found = bsearch_pos(bitmap_git, oid, &commit_pos); + + if (!found) + return NULL; + + if (bsearch_triplet_by_pos(commit_pos, bitmap_git, &triplet) < 0) + return NULL; + + xor_items_nr = 0; + offset = triplet.offset; + xor_row = triplet.xor_row; + + if (xor_row != 0xffffffff) { + int xor_flags; + khiter_t hash_pos; + struct bitmap_lookup_table_xor_item *xor_item; + + while (xor_row != 0xffffffff) { + ALLOC_GROW(xor_items, xor_items_nr + 1, xor_items_alloc); + + if (xor_items_nr + 1 >= bitmap_git->entry_count) { + error(_("corrupt bitmap lookup table: xor chain exceed entry count")); + goto corrupt; + } + + if (lookup_table_get_triplet(bitmap_git, xor_row, &triplet) < 0) + goto corrupt; + + xor_item = &xor_items[xor_items_nr]; + xor_item->offset = triplet.offset; + + if (nth_bitmap_object_oid(bitmap_git, &xor_item->oid, triplet.commit_pos) < 0) { + error(_("corrupt bitmap lookup table: commit index %u out of range"), + triplet.commit_pos); + goto corrupt; + } + + hash_pos = kh_get_oid_map(bitmap_git->bitmaps, xor_item->oid); + + /* + * If desired bitmap is already stored, we don't need + * to iterate further. Because we know that bitmaps + * that are needed to be parsed to parse this bitmap + * has already been stored. So, assign this stored bitmap + * to the xor_bitmap. + */ + if (hash_pos < kh_end(bitmap_git->bitmaps) && + (xor_bitmap = kh_value(bitmap_git->bitmaps, hash_pos))) + break; + xor_items_nr++; + xor_row = triplet.xor_row; + } + + while (xor_items_nr) { + xor_item = &xor_items[xor_items_nr - 1]; + bitmap_git->map_pos = xor_item->offset; + if (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) { + error(_("corrupt ewah bitmap: truncated header for bitmap of commit \"%s\""), + oid_to_hex(&xor_item->oid)); + goto corrupt; + } + + bitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t); + xor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos); + bitmap = read_bitmap_1(bitmap_git); + + if (!bitmap) + goto corrupt; + + xor_bitmap = store_bitmap(bitmap_git, bitmap, &xor_item->oid, xor_bitmap, xor_flags); + xor_items_nr--; + } + } + + bitmap_git->map_pos = offset; + if (bitmap_git->map_size - bitmap_git->map_pos < bitmap_header_size) { + error(_("corrupt ewah bitmap: truncated header for bitmap of commit \"%s\""), + oid_to_hex(oid)); + goto corrupt; + } + + bitmap_git->map_pos = bitmap_git->map_pos + sizeof(uint32_t) + sizeof(uint8_t); + flags = read_u8(bitmap_git->map, &bitmap_git->map_pos); + bitmap = read_bitmap_1(bitmap_git); + + if (!bitmap) + goto corrupt; + + return store_bitmap(bitmap_git, bitmap, oid, xor_bitmap, flags); + +corrupt: + free(xor_items); + is_corrupt = 1; + return NULL; +} + struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git, struct commit *commit) { khiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps, commit->object.oid); - if (hash_pos >= kh_end(bitmap_git->bitmaps)) - return NULL; + if (hash_pos >= kh_end(bitmap_git->bitmaps)) { + struct stored_bitmap *bitmap = NULL; + if (!bitmap_git->table_lookup) + return NULL; + + trace2_region_enter("pack-bitmap", "reading_lookup_table", the_repository); + /* NEEDSWORK: cache misses aren't recorded */ + bitmap = lazy_bitmap_for_commit(bitmap_git, commit); + trace2_region_leave("pack-bitmap", "reading_lookup_table", the_repository); + if (!bitmap) + return NULL; + return lookup_stored_bitmap(bitmap); + } return lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos)); } @@ -1699,8 +1942,10 @@ void test_bitmap_walk(struct rev_info *revs) if (revs->pending.nr != 1) die("you must specify exactly one commit to test"); - fprintf(stderr, "Bitmap v%d test (%d entries loaded)\n", - bitmap_git->version, bitmap_git->entry_count); + fprintf(stderr, "Bitmap v%d test (%d entries%s)", + bitmap_git->version, + bitmap_git->entry_count, + bitmap_git->table_lookup ? "" : " loaded"); root = revs->pending.objects[0].item; bm = bitmap_for_commit(bitmap_git, (struct commit *)root); @@ -1753,13 +1998,23 @@ void test_bitmap_walk(struct rev_info *revs) int test_bitmap_commits(struct repository *r) { - struct bitmap_index *bitmap_git = prepare_bitmap_git(r); struct object_id oid; MAYBE_UNUSED void *value; + struct bitmap_index *bitmap_git = prepare_bitmap_git(r); + + /* + * As this function is only used to print bitmap selected + * commits, we don't have to read the commit table. + */ if (!bitmap_git) die("failed to load bitmap indexes"); + if (bitmap_git->table_lookup) { + if (load_bitmap_entries_v1(bitmap_git) < 0) + die(_("failed to load bitmap indexes")); + } + kh_foreach(bitmap_git->bitmaps, oid, value, { printf("%s\n", oid_to_hex(&oid)); }); diff --git a/pack-bitmap.h b/pack-bitmap.h index 67a9d0fc303..9278f71ac91 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -23,6 +23,15 @@ struct bitmap_disk_header { #define NEEDS_BITMAP (1u<<22) +/* + * The width in bytes of a single triplet in the lookup table + * extension: + * (commit_pos, offset, xor_row) + * + * whose fields ar 32-, 64-, 32- bits wide, respectively. + */ +#define BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH (16) + enum pack_bitmap_opts { BITMAP_OPT_FULL_DAG = 0x1, BITMAP_OPT_HASH_CACHE = 0x4, diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh index c0607172827..7e50f8e7653 100755 --- a/t/t5310-pack-bitmaps.sh +++ b/t/t5310-pack-bitmaps.sh @@ -258,6 +258,7 @@ test_bitmap_cases () { test_expect_success 'truncated bitmap fails gracefully (ewah)' ' test_config pack.writebitmaphashcache false && + test_config pack.writebitmaplookuptable false && git repack -ad && git rev-list --use-bitmap-index --count --all >expect && bitmap=$(ls .git/objects/pack/*.bitmap) && @@ -270,6 +271,7 @@ test_bitmap_cases () { ' test_expect_success 'truncated bitmap fails gracefully (cache)' ' + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && git repack -ad && git rev-list --use-bitmap-index --count --all >expect && bitmap=$(ls .git/objects/pack/*.bitmap) && @@ -453,4 +455,24 @@ test_expect_success 'verify writing bitmap lookup table when enabled' ' grep "\"label\":\"writing_lookup_table\"" trace2 ' +test_expect_success 'lookup table is actually used to traverse objects' ' + git repack -adb && + GIT_TRACE2_EVENT="$(pwd)/trace3" \ + git rev-list --use-bitmap-index --count --all && + grep "\"label\":\"reading_lookup_table\"" trace3 +' + +test_expect_success 'truncated bitmap fails gracefully (lookup table)' ' + test_config pack.writebitmaphashcache false && + git repack -adb && + git rev-list --use-bitmap-index --count --all >expect && + bitmap=$(ls .git/objects/pack/*.bitmap) && + test_when_finished "rm -f $bitmap" && + test_copy_bytes 512 <$bitmap >$bitmap.tmp && + mv -f $bitmap.tmp $bitmap && + git rev-list --use-bitmap-index --count --all >actual 2>stderr && + test_cmp expect actual && + test_i18ngrep corrupted.bitmap.index stderr +' + test_done From patchwork Wed Jul 20 18:38:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Abhradeep Chakraborty X-Patchwork-Id: 12924395 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFEBBC433EF for ; Wed, 20 Jul 2022 18:39:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229850AbiGTSjA (ORCPT ); Wed, 20 Jul 2022 14:39:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232246AbiGTSit (ORCPT ); Wed, 20 Jul 2022 14:38:49 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6EDC97393A for ; Wed, 20 Jul 2022 11:38:34 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id u5so4305268wrm.4 for ; Wed, 20 Jul 2022 11:38:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=dRxvaKKIUZWa1PC2tLR6WPHJBu061R3VyLFIM7v5ZQs=; b=TarGACvFKf4yON7MnpCPHD5Uz/U0zbuExtrSqS8W++oxPZM/wKmDlRWw9DSCC8ULGv wvrbRz+Sgg4YBVDzrLGpb+NzEiirNANr8RgNaertioAlGucUGaJhy5dqepsoKYA4XLJN Dh6fk0vwsFHIuIWoU86TUJj45Uhc3SWFYDZNf0Zoy6k7qQcG7CvJhLI7boKLvKWFKl4U 6bwpbdOPMY9U2cG7kwChqa0K4qHV/8bzie43NtQquYrpWLr0GHEFDPmTIT5pkkprWjqD eWMzSmhtcEy+jxSYKgoSa8aCbV798NBEj9j8RCug4W1+5PjzDmu7L/IbO1XK0HtZswuL bGlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=dRxvaKKIUZWa1PC2tLR6WPHJBu061R3VyLFIM7v5ZQs=; b=2yr48EGhkhdDlSBl58hGKrc+EnaM5VE2X9M/bXGuOTIt/nVJWgQVoBYPJFoWyF5pLO TdIDJsePyJVCufJFSmUNtxByd33wXK7SghOXra2rDkr/t4fZDHDrZCFe/atNduZoWRZS X4tWFhqcmGLjgaggCLqF6COnFzqrYVC1ucMepKhdac2ZLf0OOgewIf9oPQEG2XNvTz8o 2r8e5QZ6Lf+kVCpCnpB9Ng/XrRTOQeSc+oIaX8KDIxb2dDRT4DB4jyfUqsQ6jnObC0Pn Ogt8CxHSTFuTtHTFKerFWXiqtT8FL1QKR4ERBJb+CIdO0btcWncAxbphL5FFUOu8uMec y4Rg== X-Gm-Message-State: AJIora/kt4jvAErL9BGyd3JOzTodhnQUStBDDyFSkZo+JMzQRqSaB5/F Z/eU/VZ0bC1qzfuNjmssPHG3ynT7DwI= X-Google-Smtp-Source: AGRyM1vQ8AerUdZxVjx3hNnaUulEW+2k5f92Vl6MNZXsKk1GdnMigEchmJIIhmWehJHzTrInNPihdg== X-Received: by 2002:adf:fbc6:0:b0:21d:3fc3:99e with SMTP id d6-20020adffbc6000000b0021d3fc3099emr31541722wrs.550.1658342312475; Wed, 20 Jul 2022 11:38:32 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r9-20020a05600c2c4900b003a02cbf862esm3264300wmg.13.2022.07.20.11.38.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Jul 2022 11:38:31 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 20 Jul 2022 18:38:23 +0000 Subject: [PATCH v5 5/6] p5310-pack-bitmaps.sh: enable `pack.writeReverseIndex` Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Taylor Blau , Kaartic Sivaram , Derrick Stolee , Philip Oakley , Martin =?utf-8?b?w4VncmVu?= , Abhradeep Chakraborty , Abhradeep Chakraborty Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Abhradeep Chakraborty From: Abhradeep Chakraborty Enable `pack.writeReverseIndex` before running pack-bitmap related performance tests. The performance difference with `pack.writeReverseIndex` enabled and with disabled are given below - With `pack.writeReverseIndex` ------------------------------- Test this tree ------------------------------------------------------------------------- 5310.3: repack to disk 296.55(256.53+14.52) 5310.4: simulated clone 15.64(8.88+1.39) 5310.5: simulated fetch 1.65(2.75+0.20) 5310.6: pack to file (bitmap) 48.71(30.20+7.58) 5310.7: rev-list (commits) 0.61(0.41+0.08) 5310.8: rev-list (objects) 4.38(4.26+0.09) 5310.9: rev-list with tag negated via --not 0.07(0.02+0.04) --all (objects) 5310.10: rev-list with negative tag (objects) 0.05(0.01+0.03) 5310.11: rev-list count with blob:none 0.08(0.03+0.04) 5310.12: rev-list count with blob:limit=1k 7.29(6.92+0.30) 5310.13: rev-list count with tree:0 0.08(0.03+0.04) 5310.14: simulated partial clone 9.45(8.12+0.41) 5310.16: clone (partial bitmap) 17.02(10.61+2.67) 5310.17: pack to file (partial bitmap) 51.91(28.57+7.48) 5310.18: rev-list with tree filter (partial bitmap) 1.00(0.22+0.24) Without `pack.writeReverseIndex`: ----------------------------- Test this tree ------------------------------------------------------------------------ 5310.3: repack to disk 293.80(251.30+14.30) 5310.4: simulated clone 12.50(5.15+1.36) 5310.5: simulated fetch 1.83(2.90+0.23) 5310.6: pack to file (bitmap) 39.70(20.25+7.14) 5310.7: rev-list (commits) 1.00(0.60+0.13) 5310.8: rev-list (objects) 4.11(4.00+0.10) 5310.9: rev-list with tag negated via --not 0.07(0.02+0.05) --all (objects) 5310.10: rev-list with negative tag (objects) 0.23(0.16+0.06) 5310.11: rev-list count with blob:none 0.27(0.18+0.08) 5310.12: rev-list count with blob:limit=1k 6.41(5.98+0.41) 5310.13: rev-list count with tree:0 0.26(0.18+0.07) 5310.14: simulated partial clone 4.34(3.29+0.37) 5310.16: clone (partial bitmap) 21.48(15.12+2.42) 5310.17: pack to file (partial bitmap) 47.35(37.80+4.84) 5310.18: rev-list with tree filter (partial bitmap) 0.73(0.07+0.21) Signed-off-by: Abhradeep Chakraborty --- t/perf/p5310-pack-bitmaps.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh index 7ad4f237bc3..6e8abcd5b21 100755 --- a/t/perf/p5310-pack-bitmaps.sh +++ b/t/perf/p5310-pack-bitmaps.sh @@ -13,7 +13,8 @@ test_perf_large_repo # We intentionally use the deprecated pack.writebitmaps # config so that we can test against older versions of git. test_expect_success 'setup bitmap config' ' - git config pack.writebitmaps true + git config pack.writebitmaps true && + git config pack.writeReverseIndex true ' # we need to create the tag up front such that it is covered by the repack and From patchwork Wed Jul 20 18:38:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Abhradeep Chakraborty X-Patchwork-Id: 12924396 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5591C43334 for ; Wed, 20 Jul 2022 18:39:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234047AbiGTSjG (ORCPT ); Wed, 20 Jul 2022 14:39:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231130AbiGTSiu (ORCPT ); Wed, 20 Jul 2022 14:38:50 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 466AE735B9 for ; Wed, 20 Jul 2022 11:38:36 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id 8-20020a05600c024800b003a2fe343db1so1817126wmj.1 for ; Wed, 20 Jul 2022 11:38:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=XjUJzE9yaJxw0bRYj+H2dvhx/HHTOWok2rfeh+O2nmc=; b=CkM8PyJWQYRd7FU91JRtzcmrAVEbmWMsIEELx4m+Jp1inK8R549aGnPKPrbC+1trOa nGaU6+TTLxBFqMFb0C+501s1Ux3XFE6dHLqNhhzTzDNyZ7GPb1ZC2KBeEHpKlUQaP1P1 hn5MhQd+W3xwZaJRsA2q1zIVqnGRnhOY43NmXk5Vx3XyvA1xouk19ypQL13/8XtZ+CI4 k2KcZ5HYI0+trrDBFrvGx2tx8KqYsl2FvE0FKNzY5b+xzjx2gc1/2tDd2jcU0eBb4QXc Pgc1gRDRpWeepJYzl6415MqWu82RQko2kwlOSaIkIqbA8uhORy2pTxPz8hlWKzs8morm bgjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=XjUJzE9yaJxw0bRYj+H2dvhx/HHTOWok2rfeh+O2nmc=; b=C9pKSSIYFrJgdwzFiJ7SMSXsy7Q3m38sk/Ku8Y5G2a8VDia1CRYNYCkT1F8vGtT07j ADwivy33ZCPt4KJdOMtYrWhCsjwr2TaEkvlhXXZU2aPI6I2pcCUnHeUKkYpvBETzvqru vxsutyKO2K26Jp0GAEN5A5PIdqRaHhq4RFV0seIFxwPMEZFZpL6ZNbNnyOPKEFytLsg4 uCvG7T8QyXVf9vVSBgrlvpQ0ej+EizPfUFz970O1gSmCvSKL3SBjJTKaVKP6p8HKmVxW M8+L68lMuKaBfa53v986ToHCq1pB38vH5dd0HaoSVXo704YIf14NdbgziKToHg+2FFgF ZmMw== X-Gm-Message-State: AJIora/oo5MiSdM0x2lypHabohzyRQrBo6Z0pefxd+xu4wjQIn4epMZd g0UNSA1POlnifjuNc/oVJgZnjEnuvkQ= X-Google-Smtp-Source: AGRyM1tSKVxNRDJ9q6M0VzpVDPbmCDqpz2y1oN/kNs9PNtpUusVrGcyzJ+gGdqOlF6FBBtJD8SMjIw== X-Received: by 2002:a1c:cc1a:0:b0:3a0:39b1:3408 with SMTP id h26-20020a1ccc1a000000b003a039b13408mr5006270wmb.157.1658342314127; Wed, 20 Jul 2022 11:38:34 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b17-20020a5d45d1000000b0021e4536a948sm3748236wrs.79.2022.07.20.11.38.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Jul 2022 11:38:33 -0700 (PDT) Message-Id: <6628001241d378c181564e6a27f7b6bc2fa5aa2e.1658342304.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Jul 2022 18:38:24 +0000 Subject: [PATCH v5 6/6] bitmap-lookup-table: add performance tests for lookup table Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Taylor Blau , Kaartic Sivaram , Derrick Stolee , Philip Oakley , Martin =?utf-8?b?w4VncmVu?= , Abhradeep Chakraborty , Abhradeep Chakraborty Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Abhradeep Chakraborty From: Abhradeep Chakraborty Add performance tests to verify the performance of lookup table with `pack.writeReverseIndex` enabled. This is to check the performance when the above configuration is set. Lookup table makes Git run faster in most of the cases. Below is the result of `t/perf/p5310-pack-bitmaps.sh`.`perf/p5326-multi-pack-bitmaps.sh` gives similar result. The repository used in the test is linux kernel. Test this tree --------------------------------------------------------------------------- 5310.4: repack to disk (lookup=false) 296.55(256.53+14.52) 5310.5: simulated clone 15.64(8.88+1.39) 5310.6: simulated fetch 1.65(2.75+0.20) 5310.7: pack to file (bitmap) 48.71(30.20+7.58) 5310.8: rev-list (commits) 0.61(0.41+0.08) 5310.9: rev-list (objects) 4.38(4.26+0.09) 5310.10: rev-list with tag negated via --not 0.07(0.02+0.04) --all (objects) 5310.11: rev-list with negative tag (objects) 0.05(0.01+0.03) 5310.12: rev-list count with blob:none 0.08(0.03+0.04) 5310.13: rev-list count with blob:limit=1k 7.29(6.92+0.30) 5310.14: rev-list count with tree:0 0.08(0.03+0.04) 5310.15: simulated partial clone 9.45(8.12+0.41) 5310.17: clone (partial bitmap) 21.00(15.04+2.39) 5310.18: pack to file (partial bitmap) 47.98(38.13+5.23) 5310.19: rev-list with tree filter (partial bitmap) 0.70(0.07+0.20) 5310.22: repack to disk (lookup=true) 255.92(188.13+20.47) 5310.23: simulated clone 13.78(8.84+1.09) 5310.24: simulated fetch 0.52(0.63+0.14) 5310.25: pack to file (bitmap) 44.34(28.94+6.84) 5310.26: rev-list (commits) 0.48(0.31+0.06) 5310.27: rev-list (objects) 4.02(3.93+0.07) 5310.28: rev-list with tag negated via --not 0.04(0.00+0.03) --all (objects) 5310.29: rev-list with negative tag (objects) 0.04(0.00+0.03) 5310.30: rev-list count with blob:none 0.04(0.01+0.03) 5310.31: rev-list count with blob:limit=1k 6.48(6.23+0.22) 5310.32: rev-list count with tree:0 0.04(0.01+0.03) 5310.33: simulated partial clone 8.30(7.21+0.36) 5310.35: clone (partial bitmap) 20.34(15.00+2.41) 5310.36: pack to file (partial bitmap) 46.45(38.05+5.20) 5310.37: rev-list with tree filter (partial bitmap) 0.61(0.06+0.20) Test 4-15 are tested without using lookup table. Same tests are repeated in 16-30 (using lookup table). Mentored-by: Taylor Blau Co-Mentored-by: Kaartic Sivaraam Signed-off-by: Abhradeep Chakraborty --- t/perf/p5310-pack-bitmaps.sh | 65 +++++++++++--------- t/perf/p5326-multi-pack-bitmaps.sh | 95 +++++++++++++++++------------- 2 files changed, 91 insertions(+), 69 deletions(-) diff --git a/t/perf/p5310-pack-bitmaps.sh b/t/perf/p5310-pack-bitmaps.sh index 6e8abcd5b21..adc753b6177 100755 --- a/t/perf/p5310-pack-bitmaps.sh +++ b/t/perf/p5310-pack-bitmaps.sh @@ -17,39 +17,50 @@ test_expect_success 'setup bitmap config' ' git config pack.writeReverseIndex true ' -# we need to create the tag up front such that it is covered by the repack and -# thus by generated bitmaps. -test_expect_success 'create tags' ' - git tag --message="tag pointing to HEAD" perf-tag HEAD -' +test_bitmap () { + local enabled="$1" -test_perf 'repack to disk' ' - git repack -ad -' + # we need to create the tag up front such that it is covered by the repack and + # thus by generated bitmaps. + test_expect_success 'create tags' ' + git tag --message="tag pointing to HEAD" perf-tag HEAD + ' -test_full_bitmap + test_expect_success "use lookup table: $enabled" ' + git config pack.writeBitmapLookupTable '"$enabled"' + ' -test_expect_success 'create partial bitmap state' ' - # pick a commit to represent the repo tip in the past - cutoff=$(git rev-list HEAD~100 -1) && - orig_tip=$(git rev-parse HEAD) && + test_perf "repack to disk (lookup=$enabled)" ' + git repack -ad + ' - # now kill off all of the refs and pretend we had - # just the one tip - rm -rf .git/logs .git/refs/* .git/packed-refs && - git update-ref HEAD $cutoff && + test_full_bitmap - # and then repack, which will leave us with a nice - # big bitmap pack of the "old" history, and all of - # the new history will be loose, as if it had been pushed - # up incrementally and exploded via unpack-objects - git repack -Ad && + test_expect_success "create partial bitmap state (lookup=$enabled)" ' + # pick a commit to represent the repo tip in the past + cutoff=$(git rev-list HEAD~100 -1) && + orig_tip=$(git rev-parse HEAD) && - # and now restore our original tip, as if the pushes - # had happened - git update-ref HEAD $orig_tip -' + # now kill off all of the refs and pretend we had + # just the one tip + rm -rf .git/logs .git/refs/* .git/packed-refs && + git update-ref HEAD $cutoff && + + # and then repack, which will leave us with a nice + # big bitmap pack of the "old" history, and all of + # the new history will be loose, as if it had been pushed + # up incrementally and exploded via unpack-objects + git repack -Ad && + + # and now restore our original tip, as if the pushes + # had happened + git update-ref HEAD $orig_tip + ' + + test_partial_bitmap +} -test_partial_bitmap +test_bitmap false +test_bitmap true test_done diff --git a/t/perf/p5326-multi-pack-bitmaps.sh b/t/perf/p5326-multi-pack-bitmaps.sh index f2fa228f16a..1f4c7103529 100755 --- a/t/perf/p5326-multi-pack-bitmaps.sh +++ b/t/perf/p5326-multi-pack-bitmaps.sh @@ -6,47 +6,58 @@ test_description='Tests performance using midx bitmaps' test_perf_large_repo -# we need to create the tag up front such that it is covered by the repack and -# thus by generated bitmaps. -test_expect_success 'create tags' ' - git tag --message="tag pointing to HEAD" perf-tag HEAD -' - -test_expect_success 'start with bitmapped pack' ' - git repack -adb -' - -test_perf 'setup multi-pack index' ' - git multi-pack-index write --bitmap -' - -test_expect_success 'drop pack bitmap' ' - rm -f .git/objects/pack/pack-*.bitmap -' - -test_full_bitmap - -test_expect_success 'create partial bitmap state' ' - # pick a commit to represent the repo tip in the past - cutoff=$(git rev-list HEAD~100 -1) && - orig_tip=$(git rev-parse HEAD) && - - # now pretend we have just one tip - rm -rf .git/logs .git/refs/* .git/packed-refs && - git update-ref HEAD $cutoff && - - # and then repack, which will leave us with a nice - # big bitmap pack of the "old" history, and all of - # the new history will be loose, as if it had been pushed - # up incrementally and exploded via unpack-objects - git repack -Ad && - git multi-pack-index write --bitmap && - - # and now restore our original tip, as if the pushes - # had happened - git update-ref HEAD $orig_tip -' - -test_partial_bitmap +test_bitmap () { + local enabled="$1" + + # we need to create the tag up front such that it is covered by the repack and + # thus by generated bitmaps. + test_expect_success 'create tags' ' + git tag --message="tag pointing to HEAD" perf-tag HEAD + ' + + test_expect_success "use lookup table: $enabled" ' + git config pack.writeBitmapLookupTable '"$enabled"' + ' + + test_expect_success "start with bitmapped pack (lookup=$enabled)" ' + git repack -adb + ' + + test_perf "setup multi-pack index (lookup=$enabled)" ' + git multi-pack-index write --bitmap + ' + + test_expect_success "drop pack bitmap (lookup=$enabled)" ' + rm -f .git/objects/pack/pack-*.bitmap + ' + + test_full_bitmap + + test_expect_success "create partial bitmap state (lookup=$enabled)" ' + # pick a commit to represent the repo tip in the past + cutoff=$(git rev-list HEAD~100 -1) && + orig_tip=$(git rev-parse HEAD) && + + # now pretend we have just one tip + rm -rf .git/logs .git/refs/* .git/packed-refs && + git update-ref HEAD $cutoff && + + # and then repack, which will leave us with a nice + # big bitmap pack of the "old" history, and all of + # the new history will be loose, as if it had been pushed + # up incrementally and exploded via unpack-objects + git repack -Ad && + git multi-pack-index write --bitmap && + + # and now restore our original tip, as if the pushes + # had happened + git update-ref HEAD $orig_tip + ' + + test_partial_bitmap +} + +test_bitmap false +test_bitmap true test_done