From patchwork Mon Mar 15 13:14:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12139213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1BACC433DB for ; Mon, 15 Mar 2021 13:15:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8577164EED for ; Mon, 15 Mar 2021 13:15:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229985AbhCONOx (ORCPT ); Mon, 15 Mar 2021 09:14:53 -0400 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:57941 "EHLO out4-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229570AbhCONOe (ORCPT ); Mon, 15 Mar 2021 09:14:34 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id DAC655C00C1; Mon, 15 Mar 2021 09:14:33 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Mon, 15 Mar 2021 09:14:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=date :from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm3; bh=IiMi9b9Iz1dvQlGWpMKg78Owxs+ cHGYVgF5MAYX7wdo=; b=S30xFYGWDrYAEYXHiB7Ievc1NZcYxg442FI8JHOr6bU XFeI0bBP8QZEQz6AVA+KlEhv30VO7iiePnPGnpvc7zGTEsinsC7jDsc5qloiGACQ OhVMNTAIOU0C78AHru/F6CyRsRVjl/Pv1m7pDDVDmFcDuCsw2P70xdDUeu9KrOm5 qWBNGulUp5JJOeccnth+m4Fq7jYcq6WnX09SoYYJY8cFIGtARyweWVGZUpRw/ZtX S8loVL0BUMaIerg68LvHCPWp5If4wXr+RoFDfSYtVOsReVT+hlbWoNB3VhtnEnoo vGxmuXCLPqIKni3TRh/N27aI9ppv9+elNClJe8jIpYg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=IiMi9b 9Iz1dvQlGWpMKg78Owxs+cHGYVgF5MAYX7wdo=; b=lMOUMZRUoGCbzn4uxnMCLG PhFAdsac58/aJoLO/K+FtMgeNIPImXIwQxfgPUHa/IB+jNpvhFFtPBr8s3Wx5X30 m/iZvaH4LodqK2nZ18Jj34d/5ljwiAh9ue2+q4qrmlYZ5fgUvg0I6kvbw0wCpf// 0bF/3NnPw3JADARLrF6t5WeCC7r9+obm1XMhGq4ZbTEwaGrdk7YFEF07BH+HXcPf lUlMih9gKU68vEY6IgH9PiwKQGYVWt84QedufrO1POVQlcY9jaUiGR/gwLIuRtUu tRaLbkM9IDPJJbY/P0cnrwj/cJuKFrDJ+YzYyJlH8OaeEoQL+xR6nOGtMpTM5AAg == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledruddvledggeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgefhtdefueffheekgfffudelffejtdfhvdejkedthfehvdelgfetgfdvtedthfen ucfkphepkeelrddugedrleeirddvgedunecuvehluhhsthgvrhfuihiivgeptdenucfrrg hrrghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Received: from vm-mail.pks.im (dynamic-089-014-096-241.89.14.pool.telefonica.de [89.14.96.241]) by mail.messagingengine.com (Postfix) with ESMTPA id 6247B1080054; Mon, 15 Mar 2021 09:14:32 -0400 (EDT) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 868ffb28 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 15 Mar 2021 13:14:27 +0000 (UTC) Date: Mon, 15 Mar 2021 14:14:26 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Christian Couder , Taylor Blau , Jeff King Subject: [PATCH v2 0/8] rev-parse: implement object type filter Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Hi, this is the second version of my patch series which implements a new `object:type` filter for git-rev-parse(1) and git-upload-pack(1) and extends support for bitmap indices to work with combined filters. Changes compared to v1: - I've added a patch up front which changes the uploadpack documentation to explicitly document that setting `uploadpackfilter.allow=true` will enable all future filters. I'm not yet saying that this is the correct thing to do, but rather added this patch such that we have a proper place to discuss this topic. In the context of object-type filters, I do think though that it's not an issue to default-enable type filters: they're not expensive to compute anyway. - `uploadpackfilter..allow` documentation was updated to mention the new filter. - A bug was fixed which caused us to reset `--filter-allowed` in case a normal filter was converted to a combined filter. I've added tests to more thoroughly verify that filters work as expected and also filter provided objects. Please see the attached range-diff for more details. Patrick Patrick Steinhardt (8): uploadpack.txt: document implication of `uploadpackfilter.allow` revision: mark commit parents as NOT_USER_GIVEN list-objects: move tag processing into its own function list-objects: support filtering by tag and commit list-objects: implement object type filter pack-bitmap: implement object type filter pack-bitmap: implement combined filter rev-list: allow filtering of provided items Documentation/config/uploadpack.txt | 9 ++- Documentation/rev-list-options.txt | 3 + builtin/rev-list.c | 14 ++++ list-objects-filter-options.c | 18 +++++ list-objects-filter-options.h | 8 ++ list-objects-filter.c | 116 ++++++++++++++++++++++++++++ list-objects-filter.h | 2 + list-objects.c | 32 +++++++- pack-bitmap.c | 71 +++++++++++++++-- revision.c | 4 +- revision.h | 3 - t/t6112-rev-list-filters-objects.sh | 76 ++++++++++++++++++ t/t6113-rev-list-bitmap-filters.sh | 68 +++++++++++++++- 13 files changed, 403 insertions(+), 21 deletions(-) Range-diff against v1: -: ---------- > 1: 270ff80dac uploadpack.txt: document implication of `uploadpackfilter.allow` 1: f2ce5dac89 = 2: ddbec75986 revision: mark commit parents as NOT_USER_GIVEN 2: 9feadba124 = 3: d8da0b24f4 list-objects: move tag processing into its own function 3: 4aa13ee83f = 4: 5545c189c5 list-objects: support filtering by tag and commit 4: 01b9fdbb9c ! 5: acf01472af list-objects: implement object type filter @@ Commit message Signed-off-by: Patrick Steinhardt + ## Documentation/config/uploadpack.txt ## +@@ Documentation/config/uploadpack.txt: uploadpackfilter.allow:: + uploadpackfilter..allow:: + Explicitly allow or ban the object filter corresponding to + ``, where `` may be one of: `blob:none`, +- `blob:limit`, `tree`, `sparse:oid`, or `combine`. If using +- combined filters, both `combine` and all of the nested filter +- kinds must be allowed. Defaults to `uploadpackfilter.allow`. ++ `blob:limit`, `object:type`, `tree`, `sparse:oid`, or `combine`. ++ If using combined filters, both `combine` and all of the nested ++ filter kinds must be allowed. Defaults to `uploadpackfilter.allow`. + + uploadpackfilter.tree.maxDepth:: + Only allow `--filter=tree:` when `` is no more than the value of + ## Documentation/rev-list-options.txt ## @@ Documentation/rev-list-options.txt: or units. n may be zero. The suffixes k, m, and g can be used to name units in KiB, MiB, or GiB. For example, 'blob:limit=1k' is the same 5: c97fd28d8f = 6: 8073ab665b pack-bitmap: implement object type filter 6: fe2b7a1e55 = 7: fac3477d97 pack-bitmap: implement combined filter 7: b43bf401df ! 8: 0e26fee8b3 rev-list: allow filtering of provided items @@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *pr oidset_init(&omitted_objects, DEFAULT_OIDSET_SIZE); if (arg_missing_action == MA_PRINT) + ## list-objects-filter-options.c ## +@@ list-objects-filter-options.c: static void transform_to_combine_type( + memset(filter_options, 0, sizeof(*filter_options)); + filter_options->sub = sub_array; + filter_options->sub_alloc = initial_sub_alloc; ++ filter_options->filter_wants = sub_array[0].filter_wants; + } + filter_options->sub_nr = 1; + filter_options->choice = LOFC_COMBINE; +@@ list-objects-filter-options.c: void parse_list_objects_filter( + parse_error = gently_parse_list_objects_filter( + &filter_options->sub[filter_options->sub_nr - 1], arg, + &errbuf); ++ if (!parse_error) ++ filter_options->sub[filter_options->sub_nr - 1].filter_wants = ++ filter_options->filter_wants; + } + if (parse_error) + die("%s", errbuf.buf); + ## list-objects-filter-options.h ## @@ list-objects-filter-options.h: struct list_objects_filter_options { */ @@ t/t6113-rev-list-bitmap-filters.sh: test_expect_success 'object:type filter' ' ' +test_expect_success 'object:type filter with --filter-provided' ' -+ git rev-list --objects --filter=object:type=tag --filter-provided tag >expect && ++ git rev-list --objects --filter-provided --filter=object:type=tag tag >expect && + git rev-list --use-bitmap-index \ -+ --objects --filter=object:type=tag --filter-provided tag >actual && ++ --objects --filter-provided --filter=object:type=tag tag >actual && + test_cmp expect actual && + -+ git rev-list --objects --filter=object:type=commit --filter-provided tag >expect && ++ git rev-list --objects --filter-provided --filter=object:type=commit tag >expect && + git rev-list --use-bitmap-index \ -+ --objects --filter=object:type=commit --filter-provided tag >actual && ++ --objects --filter-provided --filter=object:type=commit tag >actual && + test_bitmap_traversal expect actual && + -+ git rev-list --objects --filter=object:type=tree --filter-provided tag >expect && ++ git rev-list --objects --filter-provided --filter=object:type=tree tag >expect && + git rev-list --use-bitmap-index \ -+ --objects --filter=object:type=tree --filter-provided tag >actual && ++ --objects --filter-provided --filter=object:type=tree tag >actual && + test_bitmap_traversal expect actual && + -+ git rev-list --objects --filter=object:type=blob --filter-provided tag >expect && ++ git rev-list --objects --filter-provided --filter=object:type=blob tag >expect && + git rev-list --use-bitmap-index \ -+ --objects --filter=object:type=blob --filter-provided tag >actual && ++ --objects --filter-provided --filter=object:type=blob tag >actual && + test_bitmap_traversal expect actual +' + test_expect_success 'combine filter' ' git rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect && git rev-list --use-bitmap-index \ +@@ t/t6113-rev-list-bitmap-filters.sh: test_expect_success 'combine filter' ' + test_bitmap_traversal expect actual + ' + ++test_expect_success 'combine filter with --filter-provided' ' ++ git rev-list --objects --filter-provided --filter=blob:limit=1000 --filter=object:type=blob tag >expect && ++ git rev-list --use-bitmap-index \ ++ --objects --filter-provided --filter=blob:limit=1000 --filter=object:type=blob tag >actual && ++ test_bitmap_traversal expect actual && ++ ++ git cat-file --batch-check="%(objecttype) %(objectsize)" objects && ++ while read objecttype objectsize ++ do ++ test "$objecttype" = blob || return 1 ++ test "$objectsize" -le 1000 || return 1 ++ done