From patchwork Tue Oct 19 11:38:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12569623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2180EC433F5 for ; Tue, 19 Oct 2021 11:39:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 03B426137C for ; Tue, 19 Oct 2021 11:39:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235431AbhJSLlX (ORCPT ); Tue, 19 Oct 2021 07:41:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230361AbhJSLlW (ORCPT ); Tue, 19 Oct 2021 07:41:22 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54919C061745 for ; Tue, 19 Oct 2021 04:39:10 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id om14so14467519pjb.5 for ; Tue, 19 Oct 2021 04:39:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HLo4u9s1X5zLPxhCO8ORjhQf3Y7caS52Un+IAadkMZY=; b=V7EFPCPgOREC9CgkFZkNxRcCB3+WwaeNEziGteMwqKZdfSXzo4QMf4Z1Dj+oKAW+Nq hMiq8AAOVknVClmrsIWmaQurPKtWgcShIwgFdAdrADKw9o9DCpdWBZGZxMdICo355s+B 7tx8csfALvz+f7eQcpkcaoT2f46iEncMxiVfeJRIlB9HEIp3eNYriw6Ktm/4B4tiASj5 XM1TqYOfNlcJx/R2tmpPuOBi0Daq2feAOhY56eV9M6kSoB2kmDiNC2rLWQtszGV9ws8A tl2IlmgH92alDQLX7DKKRsj+XyTSaaLPtXXTLc6OPBmA7klywcQ3tsPrh8sSBawqEXXP 9eqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HLo4u9s1X5zLPxhCO8ORjhQf3Y7caS52Un+IAadkMZY=; b=1IglOtxgH2ySO0xzQPmtaqri8Q0eQfgrqskavdq81DL/z/Zb30bFvfunGjK8l6Afix hz54AArjEXyEr1kB4Kny8wuW3dM+3tOsav7hlxquyqH7WErjKfnNo3FPb77eaRtgUJSg l1X0gM2YRx4J3YnO5tGNbdA/s7DzHTSQ1GUjdd2rqG1pDHi7TJe9yFJdm8XdMlFs8axC lFaBiWc27MfQkK/WcxXDg74jiTjlHQN+aPU5dOWiCVkFaIJCsse9Aq7Di4TW0KXORI71 TtrXF25wF+nG+ikon26noEME52QwYDeRI6TKDaj4uxLhfsQzJA1ti48W9pr4YdD4WGWL 14iA== X-Gm-Message-State: AOAM532RtYcxSusJkkvKST4QhledHCmhzaHBl9BQtBKpRFVQXlRLJ5jl JoTK+vvuQvUvzxz/ASHm1xnKiWMVeCs= X-Google-Smtp-Source: ABdhPJxMqxbdMGveqNNFG4mrD9UBYpOP+z9Umz0rpMCHPNeWrK59EN4z/JGUTd3B3yxnK6azFNaaCg== X-Received: by 2002:a17:903:24c:b0:13f:2377:ef3a with SMTP id j12-20020a170903024c00b0013f2377ef3amr33204590plh.59.1634643549592; Tue, 19 Oct 2021 04:39:09 -0700 (PDT) Received: from localhost.localdomain ([205.204.117.96]) by smtp.gmail.com with ESMTPSA id l207sm16430861pfd.199.2021.10.19.04.39.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Oct 2021 04:39:09 -0700 (PDT) From: Teng Long X-Google-Original-From: Teng Long To: git@vger.kernel.org Cc: gitster@pobox.com, avarab@gmail.com, jonathantanmy@google.com, bagasdotme@gmail.com, adlternative@gmail.com, stolee@gmail.com, Teng Long Subject: [PATCH v6 01/12] objects.c: introduce `exclude_level` enum Date: Tue, 19 Oct 2021 19:38:25 +0800 Message-Id: <3a885678c9cee6ed4a70a7c88c15a7fa1b332150.1634634814.git.tenglong@alibaba-inc.com> X-Mailer: git-send-email 2.31.1.453.g945ddc3a74.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Teng Long Currently packfile-uri supports the exclusion of blob objects, but in some scenarios, users may wish to exclude more types of objects, such as commits, trees and tags, the difference with blob is they are more complicated. In addition to the meaning of a single object itself, it may also represent a collection of the objects that it includes (trees and blob in a root-tree) or reaches (ancestors of a commit). The exclusion range is designed by an enum named `exclude_level` in "pack-objects.c" that enumerate three values, "ET_SELF", "ET_INCLUDE" and "ET_REACHABLE". Here are some explanations for their differences: - Scene 1: "ET_SELF" for excluding object itself. Reason to support the "ET_SELF" is because the definition of exclusion-level should be better to be consistent, no matter what the object type is. Excluding a single object itself has meaning for blobs, because blob is the smallest granularity among object types, and indeed, sometimes there are some frequently used big-size blobs in repository. If you want to exclude more ownership or reachable objects, it can be considered to use "ET_INCLUDE" or "ET_REACHABLE". - Scene 2: "ET_INCLUDE" for excluding object itself and objects it contains. When a commit is specified to be excluded as packfile-uri, more offen, it's hoped to exclude the trees and blobs contained in its top-level tree, as well as the commit itself, but not the ancestors of the commit. This applies to scenarios where we want to exclude a specified non-blob object that includes some big-size objects. Commit, tag and tree are suitable for this scenario. When a tag is specified, it will exclude the dereference commit, and all trees and blobs contained in its top-tree, as well as the tag itself if it's not a lightweight one. - Scene 3: "ET_REACHABLE" for excluding object itself, all the objects it contains, and its ancestors. For further exclusion range, the ancestors are needed to excluded together, for example, the clone scenes. This commit only defines the `exclude_level` enum type. The implementations of non-blob object types will be added in subsequent commits. Signed-off-by: Teng Long --- builtin/pack-objects.c | 44 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 6d13cd3e1a..73b92a0c90 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -127,9 +127,52 @@ enum missing_action { }; static enum missing_action arg_missing_action; static show_object_fn fn_show_object; +/* + * exclude_level defines how to make the exclusion when the object matches + * a packfile-uri (uploadpack.excludeobject or uploadpack.blobpackfileuri) + * configuration. + * + * - ET_SELF; + * This type means only the object itself will be excluded, and all other + * objects it includes or reachable will not. For example, if object type is: + * - BLOB: The blob object will be excluded + * - TREE: The tree object will be excluded, the sub-trees and blobs it + * includes will not be excluded. + * - COMMIT: The commit object will be excluded, all the trees and blobs + * that be included in its top-level tree will not be excluded. + * - TAG: TAG object will be excluded, the referrenced commit object will + * not be excluded. + * - ET_INCLUDE; + * This type means that not only the object itself will be excluded, but + * also the objects it includes. For example, if object type is: + * - BLOB: Same with 'ET_SELF' + * - TREE: The tree object, and also the sub-trees and blobs that + * the object includes will be excluded. + * - COMMIT: The commit object, and also all the trees and blobs + * contained in its top-level tree will be excluded. + * - TAG: The TAG object will be excluded, and also the referrenced + * commit will be excluded (the referrenced commit exclusion will + * treat as a 'ET_INCLUDE' way). + * - ET_REACHABLE; + * This type means that not only the object and all the objects it includes + * will be excluded, but also the reachable objects. For exmple, if object + * type is: + * - BLOB: Same with 'ET_INCLUDE' + * - TREE: Same with 'ET_INCLUDE' + * - COMMIT: The Objects in the case of 'ET_INCLUDE' will be excluded, + * and also the ancestors of the commit will be excluded. + * - TAG: The Objects in the case of 'ET_INCLUDE' will be excluded, and + * also the ancestors of the referrenced commit will be excluded. + */ +enum exclude_level { + ET_SELF, + ET_INCLUDE, + ET_REACHABLE, +}; struct configured_exclusion { struct oidmap_entry e; + int level; char *pack_hash_hex; char *uri; }; @@ -3003,6 +3046,7 @@ static int git_pack_config(const char *k, const char *v, void *cb) if (oidmap_get(&configured_exclusions, &ex->e.oid)) die(_("object already configured in another " "uploadpack.blobpackfileuri (got '%s')"), v); + ex->level = ET_SELF; ex->pack_hash_hex = xcalloc(1, pack_end - oid_end); memcpy(ex->pack_hash_hex, oid_end + 1, pack_end - oid_end - 1); ex->uri = xstrdup(pack_end + 1); From patchwork Tue Oct 19 11:38:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12569625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7359C433F5 for ; Tue, 19 Oct 2021 11:39:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CDF136137D for ; Tue, 19 Oct 2021 11:39:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235421AbhJSLl1 (ORCPT ); Tue, 19 Oct 2021 07:41:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230361AbhJSLl0 (ORCPT ); Tue, 19 Oct 2021 07:41:26 -0400 Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47EB0C061745 for ; Tue, 19 Oct 2021 04:39:14 -0700 (PDT) Received: by mail-pg1-x533.google.com with SMTP id t7so4636023pgl.9 for ; Tue, 19 Oct 2021 04:39:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=I2tnoIzhKzpCWRe4l4tY6XfOp+1WivcSANGbei9HNQQ=; b=j9ZyUQUrwdldffY7S7sAPx/BGNM7WJXPeZ3pZEPKn4EzxG9OJB7WPqPV1pXyCrubHJ x7GEr3Occ91iMWAKStAAXVrlLcPqyJD06/EGJHpPpY3Lmc9EpSrv4xxyy9BzQX5/t9X+ M9QAN3TPYE4hvBbAIq/x+ErjbpWafMt/gf/mNYve21331A6DlPbjEBiMKGz2z8Yjs0xN nEkVJ3+oJ7YzrdU33ohrOVvy2LNWkhr7Fo/RiWQuDWCEw1WamUnF+OhR9SjVxrBNj+QT swWYAKE6Q45AStDjgMhtsdkUN/LMmy3oZG6BCf4QfjKUdL+cE1aFrpO8ofm98aqIAvsl TI0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=I2tnoIzhKzpCWRe4l4tY6XfOp+1WivcSANGbei9HNQQ=; b=siZaNg2D7GzTkmwXMpOerjvDT2jqzz4gDYs3tjVifu6yusKGPVf4OtmMW2RV+Gf+D0 aIxlACHBJ0H0jmJemqLwnVpx0YHt2A+yJPhtBFG5Q10xnsjZjozdqMK3k0v2wgeJnSj9 Jabuk5uTHX5T8/8nBrcZRbRLimjExlXS9uoqwDj2CCFZ6KUWQ8Mm0sxHZJDrWtkFCf9f wOhp4sePzbMex8Tm/xi6/UgIFfD07Xb/KKk6P9G4lAAwlJVoDpxEFE5i9Uhe9XQvpmhB +kkT/URsRPd+J2IRBoeOjzkBxtx7nqwxcwiF7mkfPJzxWgh/UbT+y5yjO4lAtUS/Vj54 NUWQ== X-Gm-Message-State: AOAM531FAeAgrWeW1Dswh+JwiK0Iy2P3TI1d+I+B3PJzSGHLvFPPPQlU YeTRz5G3ycB/6cKCuqB11EJdZP0OZFk= X-Google-Smtp-Source: ABdhPJw8r1NZp2enfxtlo9sudaEOk40KOM6bNyiZ1la7/dq6mdIU3JFlwLvOtnr3loWD5HBZqzmrIg== X-Received: by 2002:a62:7656:0:b0:44c:591b:5a42 with SMTP id r83-20020a627656000000b0044c591b5a42mr35063474pfc.57.1634643553473; Tue, 19 Oct 2021 04:39:13 -0700 (PDT) Received: from localhost.localdomain ([205.204.117.96]) by smtp.gmail.com with ESMTPSA id l207sm16430861pfd.199.2021.10.19.04.39.11 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Oct 2021 04:39:12 -0700 (PDT) From: Teng Long X-Google-Original-From: Teng Long To: git@vger.kernel.org Cc: gitster@pobox.com, avarab@gmail.com, jonathantanmy@google.com, bagasdotme@gmail.com, adlternative@gmail.com, stolee@gmail.com, Teng Long Subject: [PATCH v6 02/12] Introduce function `match_packfile_uri_exclusions` Date: Tue, 19 Oct 2021 19:38:26 +0800 Message-Id: <36426b4d9fc868f144277472605a4384e39e159c.1634634814.git.tenglong@alibaba-inc.com> X-Mailer: git-send-email 2.31.1.453.g945ddc3a74.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Teng Long The matching codes now placed in function `want_object_in_pack`, move it to a new function `match_packfile_uri_exclusions` to prevent subsequent modifications caused by its continuing expansion. Signed-off-by: Teng Long --- builtin/pack-objects.c | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 73b92a0c90..17053dc85a 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1231,6 +1231,24 @@ static int have_duplicate_entry(const struct object_id *oid, return 1; } +static int match_packfile_uri_exclusions(struct configured_exclusion *ex) +{ + int i; + const char *p; + + if (ex) { + for (i = 0; i < uri_protocols.nr; i++) { + if (skip_prefix(ex->uri, + uri_protocols.items[i].string, + &p) && + *p == ':') + return 1; + + } + } + return 0; +} + static int want_found_object(const struct object_id *oid, int exclude, struct packed_git *p) { @@ -1378,19 +1396,10 @@ static int want_object_in_pack(const struct object_id *oid, if (uri_protocols.nr) { struct configured_exclusion *ex = oidmap_get(&configured_exclusions, oid); - int i; - const char *p; - if (ex) { - for (i = 0; i < uri_protocols.nr; i++) { - if (skip_prefix(ex->uri, - uri_protocols.items[i].string, - &p) && - *p == ':') { - oidset_insert(&excluded_by_config, oid); - return 0; - } - } + if (ex && match_packfile_uri_exclusions(ex)) { + oidset_insert(&excluded_by_config, oid); + return 0; } } From patchwork Tue Oct 19 11:38:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12569627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04C26C433F5 for ; Tue, 19 Oct 2021 11:39:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D7FF461372 for ; Tue, 19 Oct 2021 11:39:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230361AbhJSLlc (ORCPT ); Tue, 19 Oct 2021 07:41:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235350AbhJSLla (ORCPT ); Tue, 19 Oct 2021 07:41:30 -0400 Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5F88C06161C for ; Tue, 19 Oct 2021 04:39:17 -0700 (PDT) Received: by mail-pj1-x102a.google.com with SMTP id ls14-20020a17090b350e00b001a00e2251c8so2422501pjb.4 for ; Tue, 19 Oct 2021 04:39:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uYHWS382IB+PsHQ2WHWpSXNLEyy5R5xwegvG/GZvoN0=; b=OpYRLLhlQmjmF7/fAcdlhd3dN86zoC3sdsp2mb0a7bBsJhtgOr1+p4KYXR7R3+MXb/ YeOegty8yKBzYUVGW88lqH/clL1GlLCNF//M8DSkHtrIredyqbpW9f86YIbxF3YAdwK4 pgtTzCFuAJJ7b//+OiEz7lyJmSFkAWp23gppGkbl/VwA3JdSuSwK0fgx+DkGmEFJPRqG 43/tHt2veO6spIyYn9tt9sBfZX9qEjkAOBLWJUgr1l2AGz0g5IByNmAHVFbqcEhgDoQs YoXvD9qtPAnr62xsbxoGcSOgPXpfHzFg/dINNob9By/u9A67wWe0yZpyo+/R2R0o5ISu 7JTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uYHWS382IB+PsHQ2WHWpSXNLEyy5R5xwegvG/GZvoN0=; b=c9hwecLUaB1D+zq8Ks1rvsTugjnLIX9r4kukbPkq5SLxjD6ydXt8k0R+xHF2H2EF4O anOmA63U48Ras8tKfDYlHIm2O7mLv7DwGOsji2no/GSG4m9tjRsPfdp/ca9r3w3vpcqY t3ypfLOpFliknODM9XjSZJBmouP2Bz8xo5Lz8FlgHdstZYsypeKmch8gXwiSeB1Da4pa Z+wBgakpOrbV69boFWZYivYZOhjBCAY773wOoBKCbrxxlTA8S+EgVuQ5mMUmuSpZADsv w811nsM1GalOKWT3qdsPgLrAUsB5Nfppk7/cA11FzTbXaNZeCMfJbPOQsIhPBwyv9c/G HRqw== X-Gm-Message-State: AOAM533rTHjYJhdedYsrhfvX3IunttVa2Izn2l/umXNfsO10Q872d3dS 3kbDJhroehS7vPz9evEmKDJCKxrrmZc= X-Google-Smtp-Source: ABdhPJyyTaRptwGzMX682dMfhqKT1eqljlugEE1s1/+WbqpJGcfOysnNJwYZfm+rqxNVubUS6+2FYQ== X-Received: by 2002:a17:902:6902:b0:13f:c1cd:88f1 with SMTP id j2-20020a170902690200b0013fc1cd88f1mr10012905plk.36.1634643557028; Tue, 19 Oct 2021 04:39:17 -0700 (PDT) Received: from localhost.localdomain ([205.204.117.96]) by smtp.gmail.com with ESMTPSA id l207sm16430861pfd.199.2021.10.19.04.39.14 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Oct 2021 04:39:16 -0700 (PDT) From: Teng Long X-Google-Original-From: Teng Long To: git@vger.kernel.org Cc: gitster@pobox.com, avarab@gmail.com, jonathantanmy@google.com, bagasdotme@gmail.com, adlternative@gmail.com, stolee@gmail.com, Teng Long Subject: [PATCH v6 03/12] Replace `show_data` with structure `show_info` Date: Tue, 19 Oct 2021 19:38:27 +0800 Message-Id: X-Mailer: git-send-email 2.31.1.453.g945ddc3a74.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Teng Long During the pack-objects process, "show_object" function will be called to find and show the object the function contains three parameters: 1. struct object *obj 2. const char *name 3. void *show_data This commit replace "show_data" with with a new structure `show_info`, it has two members: 1. void *show_data 2. void *show_cache The `show_data` in `show_info `is the same purpose with the original one, is the callback when showing the objects. The `show_cache` is the cache that ownership the relationship data. It stores around the extra data for showing the objects, for example, carry the ownership relationship between blob or tree objects and the referred commit to avoid redundant and expensive calculations. Signed-off-by: Teng Long --- builtin/describe.c | 9 +++++---- builtin/pack-objects.c | 23 +++++++++++++---------- builtin/rev-list.c | 11 ++++++----- bundle.c | 5 +++-- list-objects.c | 29 +++++++++++++++++++++++------ list-objects.h | 11 ++++++++--- pack-bitmap.c | 16 +++++++++------- reachable.c | 11 +++++++---- shallow.c | 4 ++-- 9 files changed, 76 insertions(+), 43 deletions(-) diff --git a/builtin/describe.c b/builtin/describe.c index 40482d8e9f..a6a9e64fb0 100644 --- a/builtin/describe.c +++ b/builtin/describe.c @@ -479,15 +479,16 @@ struct process_commit_data { struct rev_info *revs; }; -static void process_commit(struct commit *commit, void *data) +static void process_commit(struct commit *commit, struct show_info *info) { - struct process_commit_data *pcd = data; + struct process_commit_data *pcd = info->show_data; pcd->current_commit = commit->object.oid; } -static void process_object(struct object *obj, const char *path, void *data) +static void process_object(struct object *obj, const char *path, + struct show_info *info) { - struct process_commit_data *pcd = data; + struct process_commit_data *pcd = info->show_data; if (oideq(&pcd->looking_for, &obj->oid) && !pcd->dst->len) { reset_revision_walk(); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 17053dc85a..a41a0a3ea7 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3106,13 +3106,13 @@ static int add_object_entry_from_pack(const struct object_id *oid, return 0; } -static void show_commit_pack_hint(struct commit *commit, void *_data) +static void show_commit_pack_hint(struct commit *commit, struct show_info *info) { /* nothing to do; commits don't have a namehash */ } static void show_object_pack_hint(struct object *object, const char *name, - void *_data) + struct show_info *info) { struct object_entry *oe = packlist_find(&to_pack, &object->oid); if (!oe) @@ -3284,7 +3284,7 @@ static void read_object_list_from_stdin(void) /* Remember to update object flag allocation in object.h */ #define OBJECT_ADDED (1u<<20) -static void show_commit(struct commit *commit, void *data) +static void show_commit(struct commit *commit, struct show_info *info) { add_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0); commit->object.flags |= OBJECT_ADDED; @@ -3296,7 +3296,8 @@ static void show_commit(struct commit *commit, void *data) propagate_island_marks(commit); } -static void show_object(struct object *obj, const char *name, void *data) +static void show_object(struct object *obj, const char *name, + struct show_info *info) { add_preferred_base_object(name); add_object_entry(&obj->oid, obj->type, name, 0); @@ -3318,7 +3319,8 @@ static void show_object(struct object *obj, const char *name, void *data) } } -static void show_object__ma_allow_any(struct object *obj, const char *name, void *data) +static void show_object__ma_allow_any(struct object *obj, const char *name, + struct show_info *info) { assert(arg_missing_action == MA_ALLOW_ANY); @@ -3329,10 +3331,11 @@ static void show_object__ma_allow_any(struct object *obj, const char *name, void if (!has_object(the_repository, &obj->oid, 0)) return; - show_object(obj, name, data); + show_object(obj, name, info); } -static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *data) +static void show_object__ma_allow_promisor(struct object *obj, const char *name, + struct show_info *info) { assert(arg_missing_action == MA_ALLOW_PROMISOR); @@ -3343,7 +3346,7 @@ static void show_object__ma_allow_promisor(struct object *obj, const char *name, if (!has_object(the_repository, &obj->oid, 0) && is_promisor_object(&obj->oid)) return; - show_object(obj, name, data); + show_object(obj, name, info); } static int option_parse_missing_action(const struct option *opt, @@ -3591,12 +3594,12 @@ static int get_object_list_from_bitmap(struct rev_info *revs) static void record_recent_object(struct object *obj, const char *name, - void *data) + struct show_info *info) { oid_array_append(&recent_objects, &obj->oid); } -static void record_recent_commit(struct commit *commit, void *data) +static void record_recent_commit(struct commit *commit, struct show_info *info) { oid_array_append(&recent_objects, &commit->object.oid); } diff --git a/builtin/rev-list.c b/builtin/rev-list.c index b4d8ea0a35..116d8d39f1 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -94,9 +94,9 @@ static off_t get_object_disk_usage(struct object *obj) } static void finish_commit(struct commit *commit); -static void show_commit(struct commit *commit, void *data) +static void show_commit(struct commit *commit, struct show_info *s_info) { - struct rev_list_info *info = data; + struct rev_list_info *info = s_info->show_data; struct rev_info *revs = info->revs; display_progress(progress, ++progress_counter); @@ -266,12 +266,13 @@ static int finish_object(struct object *obj, const char *name, void *cb_data) return 0; } -static void show_object(struct object *obj, const char *name, void *cb_data) +static void show_object(struct object *obj, const char *name, + struct show_info *s_info) { - struct rev_list_info *info = cb_data; + struct rev_list_info *info = s_info->show_data; struct rev_info *revs = info->revs; - if (finish_object(obj, name, cb_data)) + if (finish_object(obj, name, info)) return; display_progress(progress, ++progress_counter); if (show_disk_usage) diff --git a/bundle.c b/bundle.c index 693d619551..d4c71ee2b4 100644 --- a/bundle.c +++ b/bundle.c @@ -437,9 +437,9 @@ struct bundle_prerequisites_info { int fd; }; -static void write_bundle_prerequisites(struct commit *commit, void *data) +static void write_bundle_prerequisites(struct commit *commit, struct show_info *info) { - struct bundle_prerequisites_info *bpi = data; + struct bundle_prerequisites_info *bpi = info->show_data; struct object *object; struct pretty_print_context ctx = { 0 }; struct strbuf buf = STRBUF_INIT; @@ -530,6 +530,7 @@ int create_bundle(struct repository *r, const char *path, die("revision walk setup failed"); bpi.fd = bundle_fd; bpi.pending = &revs_copy.pending; + traverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi); object_array_remove_duplicates(&revs_copy.pending); diff --git a/list-objects.c b/list-objects.c index e19589baa0..bffce67dd8 100644 --- a/list-objects.c +++ b/list-objects.c @@ -29,6 +29,10 @@ static void process_blob(struct traversal_context *ctx, struct object *obj = &blob->object; size_t pathlen; enum list_objects_filter_result r; + struct show_info show_info; + + show_info.show_data = ctx->show_data; + show_info.show_cache = NULL; if (!ctx->revs->blob_objects) return; @@ -60,7 +64,7 @@ static void process_blob(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, path->buf, ctx->show_data); + ctx->show_object(obj, path->buf, &show_info); strbuf_setlen(path, pathlen); } @@ -154,10 +158,14 @@ static void process_tree(struct traversal_context *ctx, { struct object *obj = &tree->object; struct rev_info *revs = ctx->revs; + struct show_info show_info; int baselen = base->len; enum list_objects_filter_result r; int failed_parse; + show_info.show_data = ctx->show_data; + show_info.show_cache = NULL; + if (!revs->tree_objects) return; if (!obj) @@ -191,7 +199,7 @@ static void process_tree(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, base->buf, ctx->show_data); + ctx->show_object(obj, base->buf, &show_info); if (base->len) strbuf_addch(base, '/'); @@ -207,8 +215,7 @@ static void process_tree(struct traversal_context *ctx, if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - ctx->show_object(obj, base->buf, ctx->show_data); - + ctx->show_object(obj, base->buf, &show_info); strbuf_setlen(base, baselen); free_tree_buffer(tree); } @@ -322,8 +329,12 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree) static void traverse_trees_and_blobs(struct traversal_context *ctx, struct strbuf *base) { + struct show_info show_info; int i; + show_info.show_data = ctx->show_data; + show_info.show_cache = NULL; + assert(base->len == 0); for (i = 0; i < ctx->revs->pending.nr; i++) { @@ -335,7 +346,7 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx, continue; if (obj->type == OBJ_TAG) { obj->flags |= SEEN; - ctx->show_object(obj, name, ctx->show_data); + ctx->show_object(obj, name, &show_info); continue; } if (!path) @@ -358,8 +369,13 @@ static void do_traverse(struct traversal_context *ctx) { struct commit *commit; struct strbuf csp; /* callee's scratch pad */ + struct show_info show_info; strbuf_init(&csp, PATH_MAX); + + show_info.show_data = ctx->show_data; + show_info.show_cache = NULL; + while ((commit = get_revision(ctx->revs)) != NULL) { /* * an uninteresting boundary commit may not have its tree @@ -375,7 +391,8 @@ static void do_traverse(struct traversal_context *ctx) die(_("unable to load root tree for commit %s"), oid_to_hex(&commit->object.oid)); } - ctx->show_commit(commit, ctx->show_data); + + ctx->show_commit(commit, &show_info); if (ctx->revs->tree_blobs_in_commit_order) /* diff --git a/list-objects.h b/list-objects.h index a952680e46..c7b61e9e10 100644 --- a/list-objects.h +++ b/list-objects.h @@ -5,9 +5,14 @@ struct commit; struct object; struct rev_info; -typedef void (*show_commit_fn)(struct commit *, void *); -typedef void (*show_object_fn)(struct object *, const char *, void *); -void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *); +struct show_info { + void *show_data; /* the data necessary for showing the object */ + void *show_cache; /* the cache ownership relationship data for showing the object */ +}; + +typedef void (*show_commit_fn)(struct commit *, struct show_info *); +typedef void (*show_object_fn)(struct object *, const char *, struct show_info *); +void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *show_data); typedef void (*show_edge_fn)(struct commit *); void mark_edges_uninteresting(struct rev_info *revs, diff --git a/pack-bitmap.c b/pack-bitmap.c index 3ed15431cd..0dcfa5b50e 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -459,9 +459,10 @@ struct bitmap_show_data { struct bitmap *base; }; -static void show_object(struct object *object, const char *name, void *data_) +static void show_object(struct object *object, const char *name, + struct show_info *info) { - struct bitmap_show_data *data = data_; + struct bitmap_show_data *data = info->show_data; int bitmap_pos; bitmap_pos = bitmap_position(data->bitmap_git, &object->oid); @@ -473,7 +474,7 @@ static void show_object(struct object *object, const char *name, void *data_) bitmap_set(data->base, bitmap_pos); } -static void show_commit(struct commit *commit, void *data) +static void show_commit(struct commit *commit, struct show_info *info) { } @@ -628,6 +629,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git, show_data.bitmap_git = bitmap_git; show_data.base = base; + traverse_commit_list_filtered(filter, revs, show_commit, show_object, &show_data, NULL); @@ -1268,9 +1270,9 @@ struct bitmap_test_data { }; static void test_show_object(struct object *object, const char *name, - void *data) + struct show_info *info) { - struct bitmap_test_data *tdata = data; + struct bitmap_test_data *tdata = info->show_data; int bitmap_pos; bitmap_pos = bitmap_position(tdata->bitmap_git, &object->oid); @@ -1281,9 +1283,9 @@ static void test_show_object(struct object *object, const char *name, display_progress(tdata->prg, ++tdata->seen); } -static void test_show_commit(struct commit *commit, void *data) +static void test_show_commit(struct commit *commit, struct show_info *info) { - struct bitmap_test_data *tdata = data; + struct bitmap_test_data *tdata = info->show_data; int bitmap_pos; bitmap_pos = bitmap_position(tdata->bitmap_git, diff --git a/reachable.c b/reachable.c index 77a60c70a5..4f78954c62 100644 --- a/reachable.c +++ b/reachable.c @@ -47,14 +47,15 @@ static int add_one_ref(const char *path, const struct object_id *oid, * The traversal will have already marked us as SEEN, so we * only need to handle any progress reporting here. */ -static void mark_object(struct object *obj, const char *name, void *data) +static void mark_object(struct object *obj, const char *name, + struct show_info *info) { - update_progress(data); + update_progress(info->show_data); } -static void mark_commit(struct commit *c, void *data) +static void mark_commit(struct commit *c, struct show_info *info) { - mark_object(&c->object, NULL, data); + mark_object(&c->object, NULL, info); } struct recent_data { @@ -230,6 +231,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog, return; } + /* * Set up the revision walk - this will move all commits * from the pending list to the commit walking list. @@ -244,6 +246,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog, die("unable to mark recent objects"); if (prepare_revision_walk(revs)) die("revision walk setup failed"); + traverse_commit_list(revs, mark_commit, mark_object, &cp); } diff --git a/shallow.c b/shallow.c index 9ed18eb884..ab1e49eba4 100644 --- a/shallow.c +++ b/shallow.c @@ -185,9 +185,9 @@ struct commit_list *get_shallow_commits(struct object_array *heads, int depth, return result; } -static void show_commit(struct commit *commit, void *data) +static void show_commit(struct commit *commit, struct show_info *info) { - commit_list_insert(commit, data); + commit_list_insert(commit, info->show_data); } /* From patchwork Tue Oct 19 11:38:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12569629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC4FEC433EF for ; Tue, 19 Oct 2021 11:39:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B66636137C for ; Tue, 19 Oct 2021 11:39:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235439AbhJSLlg (ORCPT ); Tue, 19 Oct 2021 07:41:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53528 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235204AbhJSLld (ORCPT ); Tue, 19 Oct 2021 07:41:33 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B8BCC06161C for ; Tue, 19 Oct 2021 04:39:21 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id g13-20020a17090a3c8d00b00196286963b9so1799268pjc.3 for ; Tue, 19 Oct 2021 04:39:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fcjdlnazJFixfNBZyGZFW8vyt2Ag975HFZiW7FqM/pg=; b=S1M70a63y4yxBoMvhj5h+tta0Lt2ziBJeE7uppr9FzZux0C03gavWqomdSPoHyA4iF 9gcrmxwDY6bmInWyacDiBm4W6mzy9TJMMdvPNdCE+YE1PBDQoBEV+a9Bh/XR8bC+LwCt Yqe5Z8EcOIpb4o4M9iy1EfDt+g8qUQerel7v4WoohC4BRY9w+fanpIhcErShZVDjiDQ4 y7NtF30fPHqCBWb1spv90DDv34pSXmB8zikkFkVgY/Z4W45+8zOyKEa1Tqs7rjetxBWE hOHL18SgmUJGYHlEX6GY/Emnt3WLJR98VClzonGdrR2hAovbqrAt9nmhd3bNcQgqBiXT UAgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fcjdlnazJFixfNBZyGZFW8vyt2Ag975HFZiW7FqM/pg=; b=nMrRJtldp3qTkLFKtnlY2huFdtpNPYmQBhuWcpuXxzdU8uVFp9+F/hfVN9nId579G3 a6m7Tho7l12gUAepIW/goAmBQWRIeo+lJoKhb75R7mG2fo6ODKhpqY7sZ8h+r4c9srxI LqNf/oZZ7r6rws29jdODMd51N4LA5T1YqREsMJJU9lH7ehpV4gVZcJ3uXdWUujdYUqvS EHOht82ED3dTzrdoZ3cPh9A8XHVzQRYduWurdO2eplOlFITV5fnFiqf8fk4WbmMmel9z qCvQYz4d0PP5PcJzSTUe1LAM62UWXu9Xz2MrxIXvfnRv/N9ZITI2rAdf9b2qR88ZzYcI TcTQ== X-Gm-Message-State: AOAM532DIZzOFK667HKM2awprRyZD0VvAKnoiUYtte5HVFY0B+k9z6Za OBGFyDK6HxTwWC0HDpf0DikiyUp+LlE= X-Google-Smtp-Source: ABdhPJxdLF3heRUB5Yhukn2GZiGts2JeuauHg6DB0OFi+AyxB5IlLJFAM5RGzDtEhydqI8e3slBBnQ== X-Received: by 2002:a17:902:6906:b0:13f:d2d1:6aff with SMTP id j6-20020a170902690600b0013fd2d16affmr3070354plk.39.1634643560372; Tue, 19 Oct 2021 04:39:20 -0700 (PDT) Received: from localhost.localdomain ([205.204.117.96]) by smtp.gmail.com with ESMTPSA id l207sm16430861pfd.199.2021.10.19.04.39.18 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Oct 2021 04:39:20 -0700 (PDT) From: Teng Long X-Google-Original-From: Teng Long To: git@vger.kernel.org Cc: gitster@pobox.com, avarab@gmail.com, jonathantanmy@google.com, bagasdotme@gmail.com, adlternative@gmail.com, stolee@gmail.com, Teng Long Subject: [PATCH v6 04/12] Introduce `uploadpack.excludeobject` configuration Date: Tue, 19 Oct 2021 19:38:28 +0800 Message-Id: X-Mailer: git-send-email 2.31.1.453.g945ddc3a74.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Teng Long Background: The `uploadpack.blobpackfileuri` is made for the "packfile-uri" feature, the feature is similar to a CDN cache, it supports the client to download pack file by a URI directly, without or reducing the server load when make packing and transporting. Reasons: First reason, `uploadpack.blobpackfileuri` supports to exclude a single object, but actually the object type can be not only a blob, so the name is inaccurate currently. Secondly, the name of the old configuration is not abstract enough, this make the furthur extension difficult. If do not change its name, to let different object types use different configuration names, the configuration items will be bloated and difficult to maintain, so the new configuration is more abstract in name. Configuation format diff: old: uploadpack.blobPackfileUri= new: uploadpack.excludeobject= The new configuration `uploadpack.excludeobject` not only supports to exclude a single object itself, but also to exclude related objects with it at once, the scope of exclusion is determined by the object type and the specified `` value in the entry. -Compatibility: Although a new configuration has been introduced, the old one is available to use and compatible with the new configuration. Signed-off-by: Teng Long --- builtin/pack-objects.c | 33 ++++++++++++++++++++++++++++++++- upload-pack.c | 7 +++++++ 2 files changed, 39 insertions(+), 1 deletion(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index a41a0a3ea7..75461483c0 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3061,6 +3061,36 @@ static int git_pack_config(const char *k, const char *v, void *cb) ex->uri = xstrdup(pack_end + 1); oidmap_put(&configured_exclusions, ex); } + if (!strcmp(k, "uploadpack.excludeobject")) { + struct configured_exclusion *ex = xmalloc(sizeof(*ex)); + const char *oid_end, *pack_end, *type_end; + struct object_id pack_hash; + char type[2]; + int level; + + if (parse_oid_hex(v, &ex->e.oid, &oid_end) || + *oid_end != ' ' || + !strlcpy(type, oid_end + 1, sizeof(type)) || + parse_oid_hex(oid_end + 3, &pack_hash, &pack_end) || + *pack_end != ' ') + die(_("value of uploadpack.excludeobject must be " + "of the form ' ' (got '%s')"), v); + if (oidmap_get(&configured_exclusions, &ex->e.oid)) + die(_("object already configured by an earlier " + "uploadpack.excludeobject (got '%s')"), v); + + level = atoi(type); + if (level < ET_SELF || level > ET_REACHABLE) { + die(_("value of must be 0 or 1 or 2 (got '%s')"), v); + } + ex->level = level; + type_end = oid_end + 2; + ex->pack_hash_hex = xcalloc(1, pack_end - type_end); + memcpy(ex->pack_hash_hex, type_end + 1, pack_end - type_end - 1); + ex->uri = xstrdup(pack_end + 1); + oidmap_put(&configured_exclusions, ex); + } + return git_default_config(k, v, cb); } @@ -3887,7 +3917,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix) N_("respect islands during delta compression")), OPT_STRING_LIST(0, "uri-protocol", &uri_protocols, N_("protocol"), - N_("exclude any configured uploadpack.blobpackfileuri with this protocol")), + N_("exclude any configured uploadpack.excludeobject or " + "uploadpack.blobpackfileuri with this protocol")), OPT_END(), }; diff --git a/upload-pack.c b/upload-pack.c index 5c1cd19612..d26fb351a3 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -1751,6 +1751,13 @@ int upload_pack_advertise(struct repository *r, strbuf_addstr(value, " packfile-uris"); free(str); } + + if (!repo_config_get_string(the_repository, + "uploadpack.excludeobject", + &str) && str) { + strbuf_addstr(value, " packfile-uris"); + free(str); + } } return 1; From patchwork Tue Oct 19 11:38:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12569631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70F72C433F5 for ; Tue, 19 Oct 2021 11:39:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5AF616137C for ; Tue, 19 Oct 2021 11:39:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235400AbhJSLlj (ORCPT ); Tue, 19 Oct 2021 07:41:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53560 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235204AbhJSLlg (ORCPT ); Tue, 19 Oct 2021 07:41:36 -0400 Received: from mail-pg1-x52f.google.com (mail-pg1-x52f.google.com [IPv6:2607:f8b0:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5091FC061745 for ; Tue, 19 Oct 2021 04:39:24 -0700 (PDT) Received: by mail-pg1-x52f.google.com with SMTP id t7so4636378pgl.9 for ; Tue, 19 Oct 2021 04:39:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=doN6CEkNOv1ViKl2YU/8DQz+fwXACOsiBPv2p14nJ0o=; b=lWyBWxdoHYenyWDZwI2m9vzCrPifA9l9MHE1qs7sE+8qMUCQvFJfYI1yjgLyhUKips lOK0TG7lfF+g2iPpJxwGW6bPHq2KCdBMQJjRTUvCTBRCEqz4YhNsAI6gjNyjbYFIwMTm pUTWYdqAZON6RiHHl81+MAMPIG53XDFaufTHEL/wNYlocM2a8o34QpJBds0AkPHnvLzJ Qh2DNaMF0SA96IeJC5Oi9Mz8r/1l/WKkzciHkVt+xO6Pn96xy+IE5X7vVxsq3ka74ec0 jCzI4lY7A9J4788Y65MQYCAkTgx1U9tNxUzCXPTiWkJvqtwYW3ONA9nzSf7WqA9+/9R5 q8EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=doN6CEkNOv1ViKl2YU/8DQz+fwXACOsiBPv2p14nJ0o=; b=sb2pjAIF9y7JWXnGJffBeGviJ45It/Gw73LZWT3BPHGjWG3n3RQmchU/1EO66jO119 qglEfXLCS1WsuXgyaEGY1GvqCCNhcXkZXG7+nZAb6B2XTYW6NCFSkfyVFCByYmoOzASo xvgsgD6cmrNzDH8Euqw9gsRZZSQjao8cgSVUMkvwXGY1p2519zpBEpngLsoN0aToetvE sHOgU8TayH3dqk1GbK1barTyS7Ol7JP9oWtMsjLZOQ3IaFW8eELHuzVZ1LwcBXqPB68W aUb6m/QTtihHoP38733lTaoVgxnxyO786yaUPLAwPnKa9wh8puUoZhheKDJvIBuWmfbt Oq8w== X-Gm-Message-State: AOAM532EwBPZbtdstAYur40FzWPdlgFQ4GTLDROp6XMXpxErhcAxVu9V lBAIjtrLuhxs+R2qMd2LzsFCQD4jz1Y= X-Google-Smtp-Source: ABdhPJwraTNgOHa9k9gyN7cBIpLnTl+9c4NnZ0ShjQtzK2XEbtFnxyNBf4U/SnyPlEkZQ96Qdi+8ZA== X-Received: by 2002:a63:9f12:: with SMTP id g18mr18768884pge.322.1634643563536; Tue, 19 Oct 2021 04:39:23 -0700 (PDT) Received: from localhost.localdomain ([205.204.117.96]) by smtp.gmail.com with ESMTPSA id l207sm16430861pfd.199.2021.10.19.04.39.21 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Oct 2021 04:39:23 -0700 (PDT) From: Teng Long X-Google-Original-From: Teng Long To: git@vger.kernel.org Cc: gitster@pobox.com, avarab@gmail.com, jonathantanmy@google.com, bagasdotme@gmail.com, adlternative@gmail.com, stolee@gmail.com, Teng Long Subject: [PATCH v6 05/12] t5702: test cases for `uploadpack.excludeobject` Date: Tue, 19 Oct 2021 19:38:29 +0800 Message-Id: X-Mailer: git-send-email 2.31.1.453.g945ddc3a74.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This commit expends the function `configure_exclusion` to support new excluding type: commit, tree and tag. Signed-off-by: Teng Long --- t/t5702-protocol-v2.sh | 146 +++++++++++++++++++++++++++-------------- 1 file changed, 98 insertions(+), 48 deletions(-) diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index 2e1243ca40..ccd3678311 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -824,17 +824,63 @@ test_expect_success 'when server does not send "ready", expect FLUSH' ' ' configure_exclusion () { - git -C "$1" hash-object "$2" >objh && - git -C "$1" pack-objects "$HTTPD_DOCUMENT_ROOT_PATH/mypack" packh && - git -C "$1" config --add \ - "uploadpack.blobpackfileuri" \ - "$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack" && - cat objh + objt="$1" + P="$2" + oid="$3" + version="$4" + excluding_type="$5" + + oldc="uploadpack.blobpackfileuri" + newc="uploadpack.excludeobject" + configkey="" + + if test "$version" = "old" + then + configkey="$oldc" + else + configkey="$newc" + fi + + if test "$objt" = "blob" + then + excluding_type="0" + git -C "$P" hash-object "$oid" >objh && + git -C "$P" pack-objects "$HTTPD_DOCUMENT_ROOT_PATH/mypack" packh && + if test "$version" = "old" + then + git -C "$P" config --add \ + "$configkey" \ + "$(cat objh) $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack" + else + git -C "$P" config --add \ + "$configkey" \ + "$(cat objh) $excluding_type $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack" + fi && + cat objh + elif test "$objt" = "commit" || test "$objt" = "tree" || test "$objt" = "tag" + then + echo "$oid" >objh && + if test "$excluding_type" = "0" + then + git -C "$P" pack-objects "$HTTPD_DOCUMENT_ROOT_PATH/mypack" packh + else + git -C "$P" pack-objects --revs "$HTTPD_DOCUMENT_ROOT_PATH/mypack" packh + fi && + + git -C "$P" config --add \ + "$configkey" \ + "$(cat objh) $excluding_type $(cat packh) $HTTPD_URL/dumb/mypack-$(cat packh).pack" && + cat objh + else + echo "unsupported object type in configure_exclusion (got $objt)" + fi } -test_expect_success 'part of packfile response provided as URI' ' +part_of_packfile_response_verify() { + + config="$1" && P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child log && + test_when_finished "rm -rf \"$P\" http_child log *found" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && @@ -843,10 +889,10 @@ test_expect_success 'part of packfile response provided as URI' ' git -C "$P" add my-blob && echo other-blob >"$P/other-blob" && git -C "$P" add other-blob && - git -C "$P" commit -m x && + test_commit -C "$P" A && - configure_exclusion "$P" my-blob >h && - configure_exclusion "$P" other-blob >h2 && + configure_exclusion blob "$P" my-blob "$config" >h && + configure_exclusion blob "$P" other-blob "$config" >h2 && GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ git -c protocol.version=2 \ @@ -879,20 +925,22 @@ test_expect_success 'part of packfile response provided as URI' ' ls http_child/.git/objects/pack/*.pack \ http_child/.git/objects/pack/*.idx >filelist && test_line_count = 6 filelist -' +} + +blobpackfileuri_fetch () { + config="$1" -test_expect_success 'packfile URIs with fetch instead of clone' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child log && + test_when_finished "rm -rf \"$P\" http_child log" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && echo my-blob >"$P/my-blob" && git -C "$P" add my-blob && - git -C "$P" commit -m x && + test_commit -C "$P" A && - configure_exclusion "$P" my-blob >h && + configure_exclusion blob "$P" my-blob $config >h && git init http_child && @@ -900,12 +948,28 @@ test_expect_success 'packfile URIs with fetch instead of clone' ' git -C http_child -c protocol.version=2 \ -c fetch.uriprotocols=http,https \ fetch "$HTTPD_URL/smart/http_parent" +} + +test_expect_success 'blob-exclusion (using uploadpack.blobpackfileuri): part of packfile response provided as URI' ' + rm -rf "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" http_child log && + part_of_packfile_response_verify old +' + +test_expect_success 'blob-exclusion (using uploadpack.excludeobject): part of packfile response provided as URI' ' + part_of_packfile_response_verify new +' + +test_expect_success 'blob-exclusion (using uploadpack.blobpackfileuri): packfile URIs with fetch instead of clone' ' + blobpackfileuri_fetch old +' + +test_expect_success 'blob-exclusion (using uploadpack.excludeobject): packfile URIs with fetch instead of clone' ' + blobpackfileuri_fetch new ' test_expect_success 'fetching with valid packfile URI but invalid hash fails' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child log && - + test_when_finished "rm -rf \"$P\" http_child log" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && @@ -913,9 +977,8 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' ' git -C "$P" add my-blob && echo other-blob >"$P/other-blob" && git -C "$P" add other-blob && - git -C "$P" commit -m x && - - configure_exclusion "$P" my-blob >h && + test_commit -C "$P" A && + configure_exclusion blob "$P" my-blob >h && # Configure a URL for other-blob. Just reuse the hash of the object as # the hash of the packfile, since the hash does not matter for this # test as long as it is not the hash of the pack, and it is of the @@ -923,9 +986,8 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' ' git -C "$P" hash-object other-blob >objh && git -C "$P" pack-objects "$HTTPD_DOCUMENT_ROOT_PATH/mypack" packh && git -C "$P" config --add \ - "uploadpack.blobpackfileuri" \ - "$(cat objh) $(cat objh) $HTTPD_URL/dumb/mypack-$(cat packh).pack" && - + "uploadpack.excludeobject" \ + "$(cat objh) 0 $(cat objh) $HTTPD_URL/dumb/mypack-$(cat packh).pack" && test_must_fail env GIT_TEST_SIDEBAND_ALL=1 \ git -c protocol.version=2 \ -c fetch.uriprotocols=http,https \ @@ -935,17 +997,14 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' ' test_expect_success 'packfile-uri with transfer.fsckobjects' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child log && - + test_when_finished "rm -rf \"$P\" http_child log" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && echo my-blob >"$P/my-blob" && git -C "$P" add my-blob && - git -C "$P" commit -m x && - - configure_exclusion "$P" my-blob >h && - + test_commit -C "$P" A && + configure_exclusion blob "$P" my-blob >h && sane_unset GIT_TEST_SIDEBAND_ALL && git -c protocol.version=2 -c transfer.fsckobjects=1 \ -c fetch.uriprotocols=http,https \ @@ -959,8 +1018,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' ' test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child log && - + test_when_finished "rm -rf \"$P\" http_child log" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && @@ -976,10 +1034,8 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' echo my-blob >"$P/my-blob" && git -C "$P" add my-blob && - git -C "$P" commit -m x && - - configure_exclusion "$P" my-blob >h && - + test_commit -C "$P" A && + configure_exclusion blob "$P" my-blob >h && sane_unset GIT_TEST_SIDEBAND_ALL && test_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \ -c fetch.uriprotocols=http,https \ @@ -989,8 +1045,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmodules is separate from tree' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child && - + test_when_finished "rm -rf \"$P\" http_child" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && @@ -999,9 +1054,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo echo "url = git://example.com/git/lib.git" >>"$P/.gitmodules" && git -C "$P" add .gitmodules && git -C "$P" commit -m x && - - configure_exclusion "$P" .gitmodules >h && - + configure_exclusion blob "$P" .gitmodules >h && sane_unset GIT_TEST_SIDEBAND_ALL && git -c protocol.version=2 -c transfer.fsckobjects=1 \ -c fetch.uriprotocols=http,https \ @@ -1015,8 +1068,7 @@ test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmo test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodules separate from tree is invalid' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && - rm -rf "$P" http_child err && - + test_when_finished "rm -rf \"$P\" http_child err" && git init "$P" && git -C "$P" config "uploadpack.allowsidebandall" "true" && @@ -1024,10 +1076,8 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul echo "path = include/foo" >>"$P/.gitmodules" && echo "url = git://example.com/git/lib.git" >>"$P/.gitmodules" && git -C "$P" add .gitmodules && - git -C "$P" commit -m x && - - configure_exclusion "$P" .gitmodules >h && - + test_commit -C "$P" A && + configure_exclusion blob "$P" .gitmodules >h && sane_unset GIT_TEST_SIDEBAND_ALL && test_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \ -c fetch.uriprotocols=http,https \ @@ -1038,4 +1088,4 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul # DO NOT add non-httpd-specific tests here, because the last part of this # test script is only executed when httpd is available and enabled. -test_done +test_done \ No newline at end of file From patchwork Tue Oct 19 11:38:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12569633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DB11C433F5 for ; Tue, 19 Oct 2021 11:40:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 13C306137D for ; Tue, 19 Oct 2021 11:40:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235489AbhJSLmL (ORCPT ); Tue, 19 Oct 2021 07:42:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235498AbhJSLly (ORCPT ); Tue, 19 Oct 2021 07:41:54 -0400 Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D39DC06176D for ; Tue, 19 Oct 2021 04:39:28 -0700 (PDT) Received: by mail-pl1-x636.google.com with SMTP id v20so13463455plo.7 for ; Tue, 19 Oct 2021 04:39:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mTjz36S2/kKRiYRHnpQKCD7edZZpLQNZ2bu0JLnEONg=; b=gRtUYtIxv/pDXYfNmgHq3wSBcdqu3NEBrwVwlvw10uM9JcLSfNtNb5PEQJJDKtxvZf CUskBYvRZonfps4UKZzjWhqslRtFu4J8Y8tn2R4drXbEL2NBMri5lcZnUaXLcJonLqF7 aJzaWbs+wIXo78m6H1iiibeuU6yF4iESmxLhitoYFA2EB+Rsl6DqbNMJuAXovrmp9naf K6kqs27BMbtK0fZiUK35e9UertdAJQJpExSOx1kZbvDgAzrnirk6Qa46uCHPrIRNQXDW s/IlIpKTCcXzkxLcP9wgtRRbuZLj87KhtNoy0sil4k56KW5cr6NSOUDbcqYpdE+cHjlA q6ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mTjz36S2/kKRiYRHnpQKCD7edZZpLQNZ2bu0JLnEONg=; b=2z8C9H0rHIxvVrMMXbK5hchqTos66kz99wv9iSgvlpUPjaOAh6JhuJ3lmB2fJ8Ta4+ iQAtRogwbLqSgDaJaMuCVFH/9JAU1KmjzAyED5sFiuneP0xHpQZcbW0fOyQa6gCkmCGg UJrleiT//fRb4KPLhO6VptRI/ezCb6yqiOtmA8EXg5C6nPlH3V7AoWhu+0oPHdA/Lqei K6iz2dL/kqN5rtQnC6THIZ53ThZiiOA973Ichs+8of8+6tcg/zFbsQ9OcP+iBxHmiB8w tzUC0YcjLIqjvF0SbAJB+CtV5z9edVtqmSWzJXkQg2dDHe93W6Oe+txVQllsXzZIqBGD RQrw== X-Gm-Message-State: AOAM533TLuFM1FoURS72PYt09teOeJNgxSAc+ShpF+sV5RGjIug8Wir/ Dnlnczpyr4kV4Ph5zgit7dglMMWlnyM= X-Google-Smtp-Source: ABdhPJzwrqqGCeM4h+fX647MEZZidfRK1Ncn3BJsBVGeqzAW6T54nCUdBvX5MwI1KdA2pbZJKLrLVA== X-Received: by 2002:a17:903:32c7:b0:13e:ea76:f8cb with SMTP id i7-20020a17090332c700b0013eea76f8cbmr32801966plr.74.1634643567235; Tue, 19 Oct 2021 04:39:27 -0700 (PDT) Received: from localhost.localdomain ([205.204.117.96]) by smtp.gmail.com with ESMTPSA id l207sm16430861pfd.199.2021.10.19.04.39.24 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Oct 2021 04:39:26 -0700 (PDT) From: Teng Long X-Google-Original-From: Teng Long To: git@vger.kernel.org Cc: gitster@pobox.com, avarab@gmail.com, jonathantanmy@google.com, bagasdotme@gmail.com, adlternative@gmail.com, stolee@gmail.com, Teng Long Subject: [PATCH v6 06/12] packfile-uri: support for excluding commits Date: Tue, 19 Oct 2021 19:38:30 +0800 Message-Id: X-Mailer: git-send-email 2.31.1.453.g945ddc3a74.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Teng Long This commit work on the exlusion of commit objects, and the Its excluding `level` can be configured as "ET_SELF", "ET_INCLUDE" or "ET_REACHABLE". Exclusion scope on different level: 1. When a commit is specified to be excluded with level "ET_SELF", only the commit object itself will be excluded. 2. When it's specified to be excluded with level "ET_INCLUDE", exclude all trees and blobs contained in its top-level tree, as well as the commit itself. 3. When it is specified with level "ET_REACHABLE", exclude its ancestors, as well as the objects need to be excluded under the level "ET_INCLUDED". Signed-off-by: Teng Long --- builtin/pack-objects.c | 83 +++++++++++++++++++++++++++++++++--------- list-objects.c | 37 +++++++++++-------- object.c | 19 ++++++++-- object.h | 14 ++++++- revision.c | 34 ++++++++++++----- revision.h | 3 ++ 6 files changed, 144 insertions(+), 46 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 75461483c0..e7b27ef443 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -36,6 +36,7 @@ #include "trace2.h" #include "shallow.h" #include "promisor-remote.h" +#include "commit-reach.h" #define IN_PACK(obj) oe_in_pack(&to_pack, obj) #define SIZE(obj) oe_size(&to_pack, obj) @@ -1354,11 +1355,14 @@ static int want_object_in_pack_one(struct packed_git *p, static int want_object_in_pack(const struct object_id *oid, int exclude, struct packed_git **found_pack, - off_t *found_offset) + off_t *found_offset, + struct referred_objects *referred_objs) { int want; struct list_head *pos; struct multi_pack_index *m; + struct configured_exclusion *ex; + struct configured_exclusion *commit_ex; if (!exclude && local && has_loose_object_nonlocal(oid)) return 0; @@ -1394,9 +1398,16 @@ static int want_object_in_pack(const struct object_id *oid, } if (uri_protocols.nr) { - struct configured_exclusion *ex = - oidmap_get(&configured_exclusions, oid); + if (referred_objs) { + struct commit *commit = referred_objs->commit; + if (commit) { + commit_ex = oidmap_get(&configured_exclusions, &commit->object.oid); + if (match_packfile_uri_exclusions(commit_ex) && commit_ex->level > ET_SELF) + return 0; + } + } + ex = oidmap_get(&configured_exclusions, oid); if (ex && match_packfile_uri_exclusions(ex)) { oidset_insert(&excluded_by_config, oid); return 0; @@ -1436,7 +1447,8 @@ static const char no_closure_warning[] = N_( ); static int add_object_entry(const struct object_id *oid, enum object_type type, - const char *name, int exclude) + const char *name, int exclude, + struct referred_objects *referred_objs) { struct packed_git *found_pack = NULL; off_t found_offset = 0; @@ -1446,7 +1458,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type, if (have_duplicate_entry(oid, exclude)) return 0; - if (!want_object_in_pack(oid, exclude, &found_pack, &found_offset)) { + if (!want_object_in_pack(oid, exclude, &found_pack, &found_offset, referred_objs)) { /* The pack is missing an object, so it will not have closure */ if (write_bitmap_index) { if (write_bitmap_index != WRITE_BITMAP_QUIET) @@ -1472,7 +1484,7 @@ static int add_object_entry_from_bitmap(const struct object_id *oid, if (have_duplicate_entry(oid, 0)) return 0; - if (!want_object_in_pack(oid, 0, &pack, &offset)) + if (!want_object_in_pack(oid, 0, &pack, &offset, NULL)) return 0; create_object_entry(oid, type, name_hash, 0, 0, pack, offset); @@ -1612,7 +1624,7 @@ static void add_pbase_object(struct tree_desc *tree, if (name[cmplen] != '/') { add_object_entry(&entry.oid, object_type(entry.mode), - fullname, 1); + fullname, 1, NULL); return; } if (S_ISDIR(entry.mode)) { @@ -1680,7 +1692,7 @@ static void add_preferred_base_object(const char *name) cmplen = name_cmp_len(name); for (it = pbase_tree; it; it = it->next) { if (cmplen == 0) { - add_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1); + add_object_entry(&it->pcache.oid, OBJ_TREE, NULL, 1, NULL); } else { struct tree_desc tree; @@ -2882,7 +2894,7 @@ static void add_tag_chain(const struct object_id *oid) die(_("unable to pack objects reachable from tag %s"), oid_to_hex(oid)); - add_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0); + add_object_entry(&tag->object.oid, OBJ_TAG, NULL, 0, NULL); if (tag->tagged->type != OBJ_TAG) return; @@ -3050,8 +3062,6 @@ static int git_pack_config(const char *k, const char *v, void *cb) *oid_end != ' ' || parse_oid_hex(oid_end + 1, &pack_hash, &pack_end) || *pack_end != ' ') - die(_("value of uploadpack.blobpackfileuri must be " - "of the form ' ' (got '%s')"), v); if (oidmap_get(&configured_exclusions, &ex->e.oid)) die(_("object already configured in another " "uploadpack.blobpackfileuri (got '%s')"), v); @@ -3114,7 +3124,7 @@ static int add_object_entry_from_pack(const struct object_id *oid, return 0; ofs = nth_packed_object_offset(p, pos); - if (!want_object_in_pack(oid, 0, &p, &ofs)) + if (!want_object_in_pack(oid, 0, &p, &ofs, NULL)) return 0; oi.typep = &type; @@ -3307,7 +3317,7 @@ static void read_object_list_from_stdin(void) die(_("expected object ID, got garbage:\n %s"), line); add_preferred_base_object(p + 1); - add_object_entry(&oid, OBJ_NONE, p + 1, 0); + add_object_entry(&oid, OBJ_NONE, p + 1, 0, NULL); } } @@ -3316,7 +3326,7 @@ static void read_object_list_from_stdin(void) static void show_commit(struct commit *commit, struct show_info *info) { - add_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0); + add_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL); commit->object.flags |= OBJECT_ADDED; if (write_bitmap_index) @@ -3329,8 +3339,9 @@ static void show_commit(struct commit *commit, struct show_info *info) static void show_object(struct object *obj, const char *name, struct show_info *info) { + struct referred_objects *referred_objs = info->show_cache; add_preferred_base_object(name); - add_object_entry(&obj->oid, obj->type, name, 0); + add_object_entry(&obj->oid, obj->type, name, 0, referred_objs); obj->flags |= OBJECT_ADDED; if (use_delta_islands) { @@ -3483,7 +3494,7 @@ static void add_objects_in_unpacked_packs(void) QSORT(in_pack.array, in_pack.nr, ofscmp); for (i = 0; i < in_pack.nr; i++) { struct object *o = in_pack.array[i].object; - add_object_entry(&o->oid, o->type, "", 0); + add_object_entry(&o->oid, o->type, "", 0, NULL); } } free(in_pack.array); @@ -3499,7 +3510,7 @@ static int add_loose_object(const struct object_id *oid, const char *path, return 0; } - add_object_entry(oid, type, "", 0); + add_object_entry(oid, type, "", 0, NULL); return 0; } @@ -3665,6 +3676,42 @@ static void mark_bitmap_preferred_tips(void) } } +static void reuse_exclusion_packfile(struct rev_info *revs) +{ + struct commit *commit; + struct oidmap_iter iter; + struct configured_exclusion *ex; + struct object_id ex_oid; + struct commit *ex_commit; + struct commit_list *list = revs->commits; + struct commit_list *newlist = NULL; + struct commit_list **p = &newlist; + + if (revs->limited) + return; + while (list) { + commit = pop_commit(&list); + if (commit_list_contains(commit, newlist)) { + continue; + } + p = commit_list_append(commit, p); + oidmap_iter_init(&configured_exclusions, &iter); + while ((ex = oidmap_iter_next(&iter)) && ex->level == ET_REACHABLE) { + ex_oid = ex->e.oid; + ex_commit = lookup_commit_reference(the_repository, &ex_oid); + if (!ex_commit) + die("Not a valid commit name %s", oid_to_hex(&ex_oid)); + if (!in_merge_bases(ex_commit, commit)) + continue; + oidset_insert(&excluded_by_config, &ex_oid); + ex_commit->object.flags |= UNINTERESTING; + p = commit_list_append(ex_commit, p); + break; + } + } + revs->commits = newlist; +} + static void get_object_list(int ac, const char **av) { struct rev_info revs; @@ -3726,6 +3773,8 @@ static void get_object_list(int ac, const char **av) die(_("revision walk setup failed")); mark_edges_uninteresting(&revs, show_edge, sparse); + reuse_exclusion_packfile(&revs); + if (!fn_show_object) fn_show_object = show_object; traverse_commit_list_filtered(&filter_options, &revs, diff --git a/list-objects.c b/list-objects.c index bffce67dd8..b32213ecf1 100644 --- a/list-objects.c +++ b/list-objects.c @@ -24,7 +24,8 @@ struct traversal_context { static void process_blob(struct traversal_context *ctx, struct blob *blob, struct strbuf *path, - const char *name) + const char *name, + struct referred_objects *referred_obj) { struct object *obj = &blob->object; size_t pathlen; @@ -32,7 +33,7 @@ static void process_blob(struct traversal_context *ctx, struct show_info show_info; show_info.show_data = ctx->show_data; - show_info.show_cache = NULL; + show_info.show_cache = referred_obj; if (!ctx->revs->blob_objects) return; @@ -101,11 +102,13 @@ static void process_gitlink(struct traversal_context *ctx, static void process_tree(struct traversal_context *ctx, struct tree *tree, struct strbuf *base, - const char *name); + const char *name, + struct referred_objects *referred_objs); static void process_tree_contents(struct traversal_context *ctx, struct tree *tree, - struct strbuf *base) + struct strbuf *base, + struct referred_objects *referred_objs) { struct tree_desc desc; struct name_entry entry; @@ -133,7 +136,7 @@ static void process_tree_contents(struct traversal_context *ctx, entry.path, oid_to_hex(&tree->object.oid)); } t->object.flags |= NOT_USER_GIVEN; - process_tree(ctx, t, base, entry.path); + process_tree(ctx, t, base, entry.path, referred_objs); } else if (S_ISGITLINK(entry.mode)) process_gitlink(ctx, entry.oid.hash, @@ -146,7 +149,7 @@ static void process_tree_contents(struct traversal_context *ctx, entry.path, oid_to_hex(&tree->object.oid)); } b->object.flags |= NOT_USER_GIVEN; - process_blob(ctx, b, base, entry.path); + process_blob(ctx, b, base, entry.path, referred_objs); } } } @@ -154,7 +157,8 @@ static void process_tree_contents(struct traversal_context *ctx, static void process_tree(struct traversal_context *ctx, struct tree *tree, struct strbuf *base, - const char *name) + const char *name, + struct referred_objects *referred_objs) { struct object *obj = &tree->object; struct rev_info *revs = ctx->revs; @@ -163,9 +167,6 @@ static void process_tree(struct traversal_context *ctx, enum list_objects_filter_result r; int failed_parse; - show_info.show_data = ctx->show_data; - show_info.show_cache = NULL; - if (!revs->tree_objects) return; if (!obj) @@ -196,6 +197,9 @@ static void process_tree(struct traversal_context *ctx, LOFS_BEGIN_TREE, obj, base->buf, &base->buf[baselen], ctx->filter); + show_info.show_cache = referred_objs; + show_info.show_data = ctx->show_data; + if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) @@ -206,7 +210,7 @@ static void process_tree(struct traversal_context *ctx, if (r & LOFR_SKIP_TREE) trace_printf("Skipping contents of tree %s...\n", base->buf); else if (!failed_parse) - process_tree_contents(ctx, tree, base); + process_tree_contents(ctx, tree, base, referred_objs); r = list_objects_filter__filter_object(ctx->revs->repo, LOFS_END_TREE, obj, @@ -321,9 +325,9 @@ void mark_edges_uninteresting(struct rev_info *revs, } } -static void add_pending_tree(struct rev_info *revs, struct tree *tree) +static void add_pending_tree(struct rev_info *revs, struct tree *tree, struct commit *referred_commit) { - add_pending_object(revs, &tree->object, ""); + add_pending_object_with_referred_commit(revs, &tree->object, "", referred_commit); } static void traverse_trees_and_blobs(struct traversal_context *ctx, @@ -340,6 +344,7 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx, for (i = 0; i < ctx->revs->pending.nr; i++) { struct object_array_entry *pending = ctx->revs->pending.objects + i; struct object *obj = pending->item; + struct referred_objects *referred_objs = pending->referred_objects; const char *name = pending->name; const char *path = pending->path; if (obj->flags & (UNINTERESTING | SEEN)) @@ -352,11 +357,11 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx, if (!path) path = ""; if (obj->type == OBJ_TREE) { - process_tree(ctx, (struct tree *)obj, base, path); + process_tree(ctx, (struct tree *)obj, base, path, referred_objs); continue; } if (obj->type == OBJ_BLOB) { - process_blob(ctx, (struct blob *)obj, base, path); + process_blob(ctx, (struct blob *)obj, base, path, referred_objs); continue; } die("unknown pending object %s (%s)", @@ -386,7 +391,7 @@ static void do_traverse(struct traversal_context *ctx) else if (get_commit_tree(commit)) { struct tree *tree = get_commit_tree(commit); tree->object.flags |= NOT_USER_GIVEN; - add_pending_tree(ctx->revs, tree); + add_pending_tree(ctx->revs, tree, commit); } else if (commit->object.parsed) { die(_("unable to load root tree for commit %s"), oid_to_hex(&commit->object.oid)); diff --git a/object.c b/object.c index 14188453c5..f86b52c4d6 100644 --- a/object.c +++ b/object.c @@ -322,14 +322,18 @@ void object_list_free(struct object_list **list) */ static char object_array_slopbuf[1]; -void add_object_array_with_path(struct object *obj, const char *name, - struct object_array *array, - unsigned mode, const char *path) +void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name, + struct object_array *array, + unsigned mode, const char *path, + struct commit *referred_commit) { unsigned nr = array->nr; unsigned alloc = array->alloc; struct object_array_entry *objects = array->objects; struct object_array_entry *entry; + struct referred_objects *referred_objs; + referred_objs = xmalloc(sizeof(struct referred_objects)); + referred_objs->commit = referred_commit; if (nr >= alloc) { alloc = (alloc + 32) * 2; @@ -339,6 +343,7 @@ void add_object_array_with_path(struct object *obj, const char *name, } entry = &objects[nr]; entry->item = obj; + entry->referred_objects = referred_objs; if (!name) entry->name = NULL; else if (!*name) @@ -354,6 +359,13 @@ void add_object_array_with_path(struct object *obj, const char *name, array->nr = ++nr; } +void add_object_array_with_path(struct object *obj, const char *name, + struct object_array *array, + unsigned mode, const char *path) +{ + add_object_array_with_path_and_referred_commit(obj, name, array, mode, path, NULL); +} + void add_object_array(struct object *obj, const char *name, struct object_array *array) { add_object_array_with_path(obj, name, array, S_IFINVALID, NULL); @@ -368,6 +380,7 @@ static void object_array_release_entry(struct object_array_entry *ent) if (ent->name != object_array_slopbuf) free(ent->name); free(ent->path); + free(ent->referred_objects); } struct object *object_array_pop(struct object_array *array) diff --git a/object.h b/object.h index 87a6da47c8..4db0ecc3f2 100644 --- a/object.h +++ b/object.h @@ -52,9 +52,20 @@ struct object_array { char *name; char *path; unsigned mode; + /* + * referred_objects or NULL. If non-NULL, it will + * temporary storage the referred objects when + * traversing the specified object. Space for time, + * reduce related computing costs (such as packfile-uri + * exclusion), clean up when the traversal is over. + */ + struct referred_objects *referred_objects; } *objects; }; +struct referred_objects{ + struct commit *commit; +}; #define OBJECT_ARRAY_INIT { 0, 0, NULL } /* @@ -157,7 +168,8 @@ void object_list_free(struct object_list **list); /* Object array handling .. */ void add_object_array(struct object *obj, const char *name, struct object_array *array); void add_object_array_with_path(struct object *obj, const char *name, struct object_array *array, unsigned mode, const char *path); - +void add_object_array_with_path_and_referred_commit(struct object *obj, const char *name, struct object_array *array, + unsigned mode, const char *path, struct commit *referred_commit); /* * Returns NULL if the array is empty. Otherwise, returns the last object * after removing its entry from the array. Other resources associated diff --git a/revision.c b/revision.c index 4853c85d0b..89a8b311ea 100644 --- a/revision.c +++ b/revision.c @@ -304,10 +304,11 @@ void mark_parents_uninteresting(struct commit *commit) commit_stack_clear(&pending); } -static void add_pending_object_with_path(struct rev_info *revs, - struct object *obj, - const char *name, unsigned mode, - const char *path) +static void add_pending_object_with_path_and_referred_commit(struct rev_info *revs, + struct object *obj, + const char *name, unsigned mode, + const char *path, + struct commit *referred_commit) { struct interpret_branch_name_options options = { 0 }; if (!obj) @@ -326,20 +327,35 @@ static void add_pending_object_with_path(struct rev_info *revs, strbuf_release(&buf); return; /* do not add the commit itself */ } - add_object_array_with_path(obj, name, &revs->pending, mode, path); + add_object_array_with_path_and_referred_commit(obj, name, &revs->pending, mode, path, referred_commit); } +static void add_pending_object_with_path(struct rev_info *revs, + struct object *obj, + const char *name, unsigned mode, + const char *path) +{ + add_pending_object_with_path_and_referred_commit(revs, obj, name, mode, path, NULL); +} static void add_pending_object_with_mode(struct rev_info *revs, struct object *obj, - const char *name, unsigned mode) + const char *name, unsigned mode, + struct commit *referred_commit) +{ + add_pending_object_with_path_and_referred_commit(revs, obj, name, mode, NULL, referred_commit); +} + +void add_pending_object_with_referred_commit(struct rev_info *revs, + struct object *obj, const char *name, + struct commit *referred_commit) { - add_pending_object_with_path(revs, obj, name, mode, NULL); + add_pending_object_with_mode(revs, obj, name, S_IFINVALID, referred_commit); } void add_pending_object(struct rev_info *revs, struct object *obj, const char *name) { - add_pending_object_with_mode(revs, obj, name, S_IFINVALID); + add_pending_object_with_mode(revs, obj, name, S_IFINVALID, NULL); } void add_head_to_pending(struct rev_info *revs) @@ -2817,7 +2833,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s if (get_oid_with_context(revs->repo, revs->def, 0, &oid, &oc)) diagnose_missing_default(revs->def); object = get_reference(revs, revs->def, &oid, 0); - add_pending_object_with_mode(revs, object, revs->def, oc.mode); + add_pending_object_with_mode(revs, object, revs->def, oc.mode, NULL); } /* Did the user ask for any diff output? Run the diff! */ diff --git a/revision.h b/revision.h index a24f72dcd1..b2e0c0b9b7 100644 --- a/revision.h +++ b/revision.h @@ -423,6 +423,9 @@ void show_object_with_name(FILE *, struct object *, const char *); */ void add_pending_object(struct rev_info *revs, struct object *obj, const char *name); +void add_pending_object_with_referred_commit(struct rev_info *revs, + struct object *obj, const char *name, + struct commit *referred_commit); void add_pending_oid(struct rev_info *revs, const char *name, const struct object_id *oid, From patchwork Tue Oct 19 11:38:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12569635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 530B6C433EF for ; Tue, 19 Oct 2021 11:40:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3AE406137D for ; Tue, 19 Oct 2021 11:40:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235492AbhJSLmM (ORCPT ); Tue, 19 Oct 2021 07:42:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53678 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235298AbhJSLmA (ORCPT ); Tue, 19 Oct 2021 07:42:00 -0400 Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CAE18C061772 for ; Tue, 19 Oct 2021 04:39:30 -0700 (PDT) Received: by mail-pj1-x102a.google.com with SMTP id e5-20020a17090a804500b001a116ad95caso2430486pjw.2 for ; Tue, 19 Oct 2021 04:39:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ikrrlh13K1mgZ435whYcNFo5qkKACD6jnoPIS8qPH7Q=; b=Gv3FtEehxNLT0Oc6NHzySvQPmknadV4jtY570Z8BfnaPgkxXeEyInAL7XJzkTIsumh ugzbdc74Wytvdl1HAoVLUgeOFy561BkRn5lohxrnNyThmguzBaJD1UrmSHWZExXJN4Bk EBpaot+60J2BYEzWEhHwZLYIBTaOB0MkjOl/g8d7H27a71FrhCNVslxpqxqWOFPuIDKf 132n680GpLn11kNsEycEQLbycxZFy/Urfcs89VaaTu+1HfFUdDB8WAVwV0D3S7ZYepCH WW8yl5/uuQ/t1aMnmRIuwP5QUK3erlqNIyi39uA4406L/7ZsSP6jac37uE61XzaCCBfT ZPSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ikrrlh13K1mgZ435whYcNFo5qkKACD6jnoPIS8qPH7Q=; b=Yt6CqWmvwNsFiYHO16d2PWyf6/x0vKn4BaYJLboBmjYvhuiEv54SymDrPXYBOUlUC9 KFRg2OBteyjMpGaUhYy8XgmFzD8UrQs1veOYqr+1Yi1Calphb0Ci9xxGtjwjYZglNyNx 8TIbZcLS/r6FChVWcH/LtO7VYweDsi6NmGA9WrDHSjhD3yu1uCyX95AE/ovvkvmztxQ/ YpPg95xwFNIiG7aoAAR5X5DvJP9yQUCMDiI+eQ1dFjrUYkBzvSYQ48+iigViPsbjLsI2 y6VcA6Yueu+zv2EZZhT1/0euqY3Lg3uI/3nW/Y7ByRpaT5vEhCzT6qmt0aIPRUhgTFD3 GHTg== X-Gm-Message-State: AOAM530oWLsNRjgIcJysH63tOJour0z63qADxrn4l+OXt636OvfDGc4z +Zd33su9yLSyHQrl6YJEw+U/7Q3TwUg= X-Google-Smtp-Source: ABdhPJxZvMA/Z5v+0E9sMtYU3dS9XbEwxmXZBxNXzRoB/orjuh47tsHAQu6/eBncQg9KbtlbYXN/8A== X-Received: by 2002:a17:902:e884:b0:13f:19bf:fc16 with SMTP id w4-20020a170902e88400b0013f19bffc16mr32775250plg.67.1634643569994; Tue, 19 Oct 2021 04:39:29 -0700 (PDT) Received: from localhost.localdomain ([205.204.117.96]) by smtp.gmail.com with ESMTPSA id l207sm16430861pfd.199.2021.10.19.04.39.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Oct 2021 04:39:29 -0700 (PDT) From: Teng Long X-Google-Original-From: Teng Long To: git@vger.kernel.org Cc: gitster@pobox.com, avarab@gmail.com, jonathantanmy@google.com, bagasdotme@gmail.com, adlternative@gmail.com, stolee@gmail.com, Teng Long Subject: [PATCH v6 07/12] t5702: test cases for excluding commits Date: Tue, 19 Oct 2021 19:38:31 +0800 Message-Id: X-Mailer: git-send-email 2.31.1.453.g945ddc3a74.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- t/t5702-protocol-v2.sh | 300 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 300 insertions(+) diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index ccd3678311..6e323253f7 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -967,6 +967,306 @@ test_expect_success 'blob-exclusion (using uploadpack.excludeobject): packfile U blobpackfileuri_fetch new ' +test_expect_success 'commit-exclusion(excluding_type=ET_SELF): part of packfile response provided as URI' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + test_when_finished "rm -rf \"$P\" http_child log *found" && + excluding_type="0" && + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + mkdir "$P"/my-tree && + echo my-blob >"$P"/my-tree/my-blob && + git -C "$P" add my-tree && + mkdir "$P"/my-tree/sub-tree && + echo sub-blob >"$P"/my-tree/sub-tree/sub-blob && + git -C "$P" add my-tree && + test_commit -C "$P" A && + commith=$(git -C "$P" rev-parse A) && + roottreeh=$(git -C "$P" rev-parse A:) && + mytreeh=$(git -C "$P" ls-tree HEAD my-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + subtreeh=$(git -C "$P" ls-tree HEAD my-tree/sub-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + ah=$(git -C "$P" hash-object A.t) && + myblobh=$(git -C "$P" hash-object my-tree/my-blob) && + subblobh=$(git -C "$P" hash-object my-tree/sub-tree/sub-blob) && + configure_exclusion commit "$P" "$commith" new "$excluding_type" >h && + GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ + git -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child +' + +test_expect_success 'commit-exclusion(excluding_type=ET_INCLUDE): part of packfile response provided as URI' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + test_when_finished "rm -rf \"$P\" http_child log *found" && + excluding_type="1" && + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + mkdir "$P"/my-tree && + echo my-blob >"$P"/my-tree/my-blob && + git -C "$P" add my-tree && + mkdir "$P"/my-tree/sub-tree && + echo sub-blob >"$P"/my-tree/sub-tree/sub-blob && + git -C "$P" add my-tree && + test_commit -C "$P" A && + commith=$(git -C "$P" rev-parse A) && + roottreeh=$(git -C "$P" rev-parse A:) && + mytreeh=$(git -C "$P" ls-tree HEAD my-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + subtreeh=$(git -C "$P" ls-tree HEAD my-tree/sub-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + ah=$(git -C "$P" hash-object A.t) && + myblobh=$(git -C "$P" hash-object my-tree/my-blob) && + subblobh=$(git -C "$P" hash-object my-tree/sub-tree/sub-blob) && + configure_exclusion commit "$P" "$commith" new "$excluding_type" >h && + GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ + git -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child && + for idx in http_child/.git/objects/pack/*.idx + do + git verify-pack --object-format=$(test_oid algo) --verbose $idx >out && + { + grep "^[0-9a-f]\{16,\} " out || : + } >out.objectlist && + if test_line_count = 7 out.objectlist + then + if grep $commith out + then + >commithfound + fi && + if grep $roottreeh out + then + >roottreehfound + fi && + if grep $ah out + then + >ahfound + fi && + if grep $mytreeh out + then + >mytreehfound + fi && + if grep $myblobh out + then + >myblobhfound + fi && + if grep $subtreeh out + then + >subtreehfound + fi && + if grep $subblobh out + then + >subblobhfound + fi + fi + done && + test -f mytreehfound && + test -f myblobhfound && + test -f subtreehfound && + test -f subblobhfound && + test -f commithfound && + test -f roottreehfound && + test -f ahfound && + # Ensure that there are exactly 2 packfiles with associated .idx + ls http_child/.git/objects/pack/*.pack \ + http_child/.git/objects/pack/*.idx >filelist && + test_line_count = 4 filelist +' + +test_expect_success 'commit-exclusion(excluding_type=ET_REACHABLE): hitten a full packfile response provided as URI' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + test_when_finished "rm -rf \"$P\" http_child log *found" && + excluding_type="2" && + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + mkdir "$P"/my-tree && + echo my-blob >"$P"/my-tree/my-blob && + git -C "$P" add my-tree && + test_commit -C "$P" A && + commith=$(git -C "$P" rev-parse A) && + roottreeh=$(git -C "$P" rev-parse A:) && + mytreeh=$(git -C "$P" ls-tree HEAD my-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + ah=$(git -C "$P" hash-object A.t) && + myblobh=$(git -C "$P" hash-object my-tree/my-blob) && + configure_exclusion commit "$P" "$commith" new "$excluding_type" >h && + GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ + git -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child && + for idx in http_child/.git/objects/pack/*.idx + do + git verify-pack --object-format=$(test_oid algo) --verbose $idx >out && + { + grep "^[0-9a-f]\{16,\} " out || : + } >out.objectlist && + if test_line_count = 5 out.objectlist + then + if grep $commith out + then + >commithfound + fi && + if grep $roottreeh out + then + >roottreehfound + fi && + if grep $mytreeh out + then + >mytreehfound + fi && + if grep $ah out + then + >ahfound + fi && + if grep $myblobh out + then + >myblobhfound + fi + elif test_line_count = 0 out.objectlist + then + >emptypackfound + fi + done && + test -f emptypackfound && + test -f mytreehfound && + test -f myblobhfound && + test -f commithfound && + test -f roottreehfound && + test -f ahfound && + # Ensure that there are exactly 2 packfiles with associated .idx + ls http_child/.git/objects/pack/*.pack \ + http_child/.git/objects/pack/*.idx >filelist && + test_line_count = 4 filelist +' + +test_expect_success 'commit-exclusion(excluding_type=ET_REACHABLE): part of packfile response provided as URI' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + test_when_finished "rm -rf \"$P\" http_child log *found" && + excluding_type="2" && + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + mkdir "$P"/my-tree && + echo my-blob >"$P"/my-tree/my-blob && + git -C "$P" add my-tree && + test_commit -C "$P" A && + mkdir "$P"/other-tree && + echo other-blob >"$P"/other-tree/other-blob && + git -C "$P" add other-tree && + test_commit -C "$P" B && + ah=$(git -C "$P" hash-object A.t) && + bh=$(git -C "$P" hash-object B.t) && + myblobh=$(git -C "$P" hash-object my-tree/my-blob) && + otherblobh=$(git -C "$P" hash-object other-tree/other-blob) && + rm -rf "$P"/my-tree "$P"/other-tree "$P"/A.t "$P"/B.t && + mkdir "$P"/another-tree && + echo another-blob >"$P"/another-tree/another-blob && + git -C "$P" add . && + test_commit -C "$P" C && + commitAh=$(git -C "$P" rev-parse A) && + commitBh=$(git -C "$P" rev-parse B) && + commitCh=$(git -C "$P" rev-parse C) && + roottreeAh=$(git -C "$P" rev-parse A:) && + roottreeBh=$(git -C "$P" rev-parse B:) && + roottreeCh=$(git -C "$P" rev-parse C:) && + mytreeh=$(git -C "$P" ls-tree A my-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + othertreeh=$(git -C "$P" ls-tree B other-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + anothertreeh=$(git -C "$P" ls-tree C another-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + ch=$(git -C "$P" hash-object C.t) && + anotherblobh=$(git -C "$P" hash-object another-tree/another-blob) && + configure_exclusion commit "$P" "$commitBh" new "$excluding_type" >h && + GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ + git -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child && + for idx in http_child/.git/objects/pack/*.idx + do + git verify-pack --object-format=$(test_oid algo) --verbose $idx >out && + { + grep "^[0-9a-f]\{16,\} " out || : + } >out.objectlist && + if test_line_count = 5 out.objectlist + then + if grep $commitCh out + then + >commitChfound + fi && + if grep $roottreeCh out + then + >roottreeChfound + fi && + if grep $anothertreeh out + then + >anothertreehfound + fi && + if grep $anotherblobh out + then + >anotherblobhfound + fi && + if grep $ch out + then + >chfound + fi + elif test_line_count = 10 out.objectlist + then + if grep $commitAh out + then + >commitAhfound + fi && + if grep $commitBh out + then + >commitBhfound + fi && + if grep $roottreeAh out + then + >roottreeAhfound + fi && + if grep $roottreeBh out + then + >roottreeBhfound + fi && + if grep $mytreeh out + then + >mytreehfound + fi && + if grep $othertreeh out + then + >othertreehfound + fi && + if grep $myblobh out + then + >myblobhfound + fi && + if grep $otherblobh out + then + >otherblobhfound + fi && + if grep $ah out + then + >ahfound + fi && + if grep $bh out + then + >bhfound + fi + fi + done && + test -f commitChfound && + test -f roottreeChfound && + test -f anothertreehfound && + test -f anotherblobhfound && + test -f chfound && + test -f commitAhfound && + test -f commitBhfound && + test -f roottreeAhfound && + test -f roottreeBhfound && + test -f mytreehfound && + test -f othertreehfound && + test -f myblobhfound && + test -f otherblobhfound && + test -f ahfound && + test -f bhfound && + # Ensure that there are exactly 2 packfiles with associated .idx + ls http_child/.git/objects/pack/*.pack \ + http_child/.git/objects/pack/*.idx >filelist && + test_line_count = 4 filelist +' + test_expect_success 'fetching with valid packfile URI but invalid hash fails' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && test_when_finished "rm -rf \"$P\" http_child log" && From patchwork Tue Oct 19 11:38:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12569637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C586AC433EF for ; Tue, 19 Oct 2021 11:40:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A69946137D for ; Tue, 19 Oct 2021 11:40:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230281AbhJSLmS (ORCPT ); Tue, 19 Oct 2021 07:42:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230303AbhJSLmK (ORCPT ); Tue, 19 Oct 2021 07:42:10 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08FA9C061777 for ; Tue, 19 Oct 2021 04:39:33 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id e5-20020a17090a804500b001a116ad95caso2430581pjw.2 for ; Tue, 19 Oct 2021 04:39:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NaAP2QAIoZTZsr+6wUC+yxCPQXY2JRO7stjPsRNyWwo=; b=N6O5Mot6A6d6lizV8Y9PkMeP72GZ3NSk5VUMDVuFMvbTt1ousJfKNrU3lt+HOIXz/G h0wKRzIPbxkg88XwZJ4gJBTA7z4354Kf0QZXCyZQ9zKdt9+0hbIpiD/skQSyoRZGMJcD apuJeNxAiXoYIlSYdsQ17Zvn7OvnwBKI+rDeMz1yrVwtiNpCe+Jlow3DwK/rcqQZq3qE QmAv5GHZqQvT7hTifGM0mZx0wUOXOvOfi2GzitdxmgpE1BpnbbGO3OKNV8LlN042qUuK edaRV9Gr5CeG6ek7BPz38YBvzIoeLIu4aV5xOw2Sgc3QGTe8wOyN868HBVqy3s35GqG2 KZuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NaAP2QAIoZTZsr+6wUC+yxCPQXY2JRO7stjPsRNyWwo=; b=oFQv61DDcFno4XFIz9XbhD3u8xEq2dPsjBDpjdrPp4Smrwyj9wBOa+84ELjmws9K96 NA1LWKhLU7c0k/lj5vRYFbb+G4o5QbxjYoAWcQhBjjmVXVu1QpGBTQWKzE6GTpTIvzEs 5iNjWRVjjlmWihPpXamse1BGJz4istnrHk4584VvbcPeqXFB0qYsi2/MRto34pEhVkOb xdu6b6Mjygu9TRStWD4j48edcpBO+wFzIGUK7YptGq0lf14QsdnfC9aSss7amTt+KYKf DtRPrmNW1rC39sx/ksrq+Q8/FGlMN9olgb0qms5ZoIvLH5n7t+qKcOh7iUFSlt00B7nZ eIhA== X-Gm-Message-State: AOAM5324SLzeKUuz9glIrpc/Kf7jWCLVTd1HfdDsMRWdMN5BsMusuh9W n4XtTIHzMuO4Qu73o4wjzDJv4A2D7NI= X-Google-Smtp-Source: ABdhPJw1dapzjOO3/7235ThjEi62KIsf980e62tgqzN+1C1Uysxp0M/FdPlX22stXrbW33MuS1Gr7g== X-Received: by 2002:a17:902:c206:b0:13f:68f:675c with SMTP id 6-20020a170902c20600b0013f068f675cmr33452109pll.8.1634643572337; Tue, 19 Oct 2021 04:39:32 -0700 (PDT) Received: from localhost.localdomain ([205.204.117.96]) by smtp.gmail.com with ESMTPSA id l207sm16430861pfd.199.2021.10.19.04.39.30 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Oct 2021 04:39:32 -0700 (PDT) From: Teng Long X-Google-Original-From: Teng Long To: git@vger.kernel.org Cc: gitster@pobox.com, avarab@gmail.com, jonathantanmy@google.com, bagasdotme@gmail.com, adlternative@gmail.com, stolee@gmail.com, Teng Long Subject: [PATCH v6 08/12] packfile-uri: support for excluding trees Date: Tue, 19 Oct 2021 19:38:32 +0800 Message-Id: X-Mailer: git-send-email 2.31.1.453.g945ddc3a74.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Teng Long This commit introduce the exclusion of tree objects. The exclusion range is designed by an enum named `exclude_level` in "pack-objects.c" that enumerate three values, "ET_SELF", "ET_INCLUDE" and "ET_REACHABLE". Exclusion scope on different level: 1. When a tree is specified to be excluded with level "ET_SELF", only the tree object itself will be excluded. 2. When a tree specified to be excluded with level "ET_INCLUDE", exclude the tree itself, as well as all the trees and blobs it contains. 3. When it is specified with level "ET_REACHABLE", the excluding scope is as same as level "ET_INCLUDE", because tree do not have any ancestors. Signed-off-by: Teng Long --- builtin/pack-objects.c | 18 ++++++++++++++++++ list-objects.c | 8 ++++++-- object.c | 1 + object.h | 1 + 4 files changed, 26 insertions(+), 2 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index e7b27ef443..6713e734fb 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1250,6 +1250,21 @@ static int match_packfile_uri_exclusions(struct configured_exclusion *ex) return 0; } +static int want_exclude_object(struct object_list *objects) +{ + struct object_list *p; + struct configured_exclusion *ex; + + if (!objects) + return 0; + for (p = objects; p; p = p->next) { + ex = oidmap_get(&configured_exclusions, &p->item->oid); + if (match_packfile_uri_exclusions(ex) && ex->level > ET_SELF) + return 1; + } + return 0; +} + static int want_found_object(const struct object_id *oid, int exclude, struct packed_git *p) { @@ -1400,11 +1415,14 @@ static int want_object_in_pack(const struct object_id *oid, if (uri_protocols.nr) { if (referred_objs) { struct commit *commit = referred_objs->commit; + struct object_list *trees = referred_objs->trees; if (commit) { commit_ex = oidmap_get(&configured_exclusions, &commit->object.oid); if (match_packfile_uri_exclusions(commit_ex) && commit_ex->level > ET_SELF) return 0; } + if (want_exclude_object(trees)) + return 0; } ex = oidmap_get(&configured_exclusions, oid); diff --git a/list-objects.c b/list-objects.c index b32213ecf1..40292e2cc8 100644 --- a/list-objects.c +++ b/list-objects.c @@ -114,8 +114,11 @@ static void process_tree_contents(struct traversal_context *ctx, struct name_entry entry; enum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ? all_entries_interesting : entry_not_interesting; + struct referred_objects *referred_buf; init_tree_desc(&desc, tree->buffer, tree->size); + referred_buf = xmemdupz(referred_objs, sizeof(struct referred_objects)); + object_list_insert(&tree->object, &referred_buf->trees); while (tree_entry(&desc, &entry)) { if (match != all_entries_interesting) { @@ -136,7 +139,7 @@ static void process_tree_contents(struct traversal_context *ctx, entry.path, oid_to_hex(&tree->object.oid)); } t->object.flags |= NOT_USER_GIVEN; - process_tree(ctx, t, base, entry.path, referred_objs); + process_tree(ctx, t, base, entry.path, referred_buf); } else if (S_ISGITLINK(entry.mode)) process_gitlink(ctx, entry.oid.hash, @@ -149,9 +152,10 @@ static void process_tree_contents(struct traversal_context *ctx, entry.path, oid_to_hex(&tree->object.oid)); } b->object.flags |= NOT_USER_GIVEN; - process_blob(ctx, b, base, entry.path, referred_objs); + process_blob(ctx, b, base, entry.path, referred_buf); } } + free(referred_buf); } static void process_tree(struct traversal_context *ctx, diff --git a/object.c b/object.c index f86b52c4d6..895068cbc2 100644 --- a/object.c +++ b/object.c @@ -334,6 +334,7 @@ void add_object_array_with_path_and_referred_commit(struct object *obj, const ch struct referred_objects *referred_objs; referred_objs = xmalloc(sizeof(struct referred_objects)); referred_objs->commit = referred_commit; + referred_objs->trees = NULL; if (nr >= alloc) { alloc = (alloc + 32) * 2; diff --git a/object.h b/object.h index 4db0ecc3f2..618d674249 100644 --- a/object.h +++ b/object.h @@ -65,6 +65,7 @@ struct object_array { struct referred_objects{ struct commit *commit; + struct object_list *trees; }; #define OBJECT_ARRAY_INIT { 0, 0, NULL } From patchwork Tue Oct 19 11:38:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12569639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3FD1C433FE for ; Tue, 19 Oct 2021 11:40:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9CF96613A1 for ; Tue, 19 Oct 2021 11:40:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235396AbhJSLmT (ORCPT ); Tue, 19 Oct 2021 07:42:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53646 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235291AbhJSLmK (ORCPT ); Tue, 19 Oct 2021 07:42:10 -0400 Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 715F5C061779 for ; Tue, 19 Oct 2021 04:39:35 -0700 (PDT) Received: by mail-pl1-x634.google.com with SMTP id y1so13458917plk.10 for ; Tue, 19 Oct 2021 04:39:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fiArQ1yQVGbHOqJST1guSHQiHUQpAWMT+0Npi8y5QOM=; b=QR26m74Euch7+msf8Pwv0veHd+3K0CkYnGKKq1pupj0i3l5gkX7WJqM2nSjbstqPKP uwtbiYwWmVU6Xfds5GWverhb2bNobE90gz/m/XeS19yd0DLPvXA+jd5W/c7akvKpppc0 FFrcWp9pYqb2ae2ht3bIwSUBsOiimlXGxeDwNB5uUqKE8uG5ZlJm5gw9VQ+Gc4pR2wrQ aQOx5onRu903bIWiP8TL4WsAp+GmebwbaMnf6owX1nzpAvsyz2QUCYmpnFU6kgcY9+qW tNNzE95m+PSor7uLStrzdZOsSvba3x3pNDZ5XHWomlGQB+lRaf8uGa/mjJl1tzlPcnPJ m+6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fiArQ1yQVGbHOqJST1guSHQiHUQpAWMT+0Npi8y5QOM=; b=cjhf1mFOFjA+aCjyw3zdX4qZKlyeJ7XOI0OVbeXmiNBkFNQP9BfBKOsKexQ72QmzOg mvYNQJEaQtWYJStde/Lx0CqVeXkZ37fNFbKe+qFeaEl11mlIaKOaKjSer7wTULtvRL4k BTnqW2hGfSctqU+we14aGd9MmLIJi9pJ5uuSu3OkiYnNydCEUr8LcAp49ajop6GWi/Ii W+Gk6Pc/p+YWI9imrEwHSUHqJDkeUkBPahJwmc2G/aBsnufpWtFeWLxOaGgQCyxuRp1k XBCAlcp1ZL+Ap2wGb9kzbxgES9grDzHMyOAKEzV330tSwMkecoMlzLpYdLicHRr3IUYr 2jlQ== X-Gm-Message-State: AOAM531yEUn8LF4lCl49q8aXcx91ZQFYEwjNSIYPSBxbrDPOa8vfq1AN jurlqPy5gcpRrns9rcxVNKPrHwjHFME= X-Google-Smtp-Source: ABdhPJziyDJw6LkpNYzDMIxzkZkrjyMnBrHRLO6ncJ6cBOONOrl6wh4xiM6DvuNCGkdYt1SZCZ+OVQ== X-Received: by 2002:a17:903:246:b0:13f:2ff9:8b93 with SMTP id j6-20020a170903024600b0013f2ff98b93mr33118807plh.54.1634643574709; Tue, 19 Oct 2021 04:39:34 -0700 (PDT) Received: from localhost.localdomain ([205.204.117.96]) by smtp.gmail.com with ESMTPSA id l207sm16430861pfd.199.2021.10.19.04.39.32 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Oct 2021 04:39:34 -0700 (PDT) From: Teng Long X-Google-Original-From: Teng Long To: git@vger.kernel.org Cc: gitster@pobox.com, avarab@gmail.com, jonathantanmy@google.com, bagasdotme@gmail.com, adlternative@gmail.com, stolee@gmail.com, Teng Long Subject: [PATCH v6 09/12] t5702: test cases for excluding trees Date: Tue, 19 Oct 2021 19:38:33 +0800 Message-Id: <5fc79a9a3213a17f1648dda9ee17d5fd82b0f979.1634634814.git.tenglong@alibaba-inc.com> X-Mailer: git-send-email 2.31.1.453.g945ddc3a74.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- t/t5702-protocol-v2.sh | 213 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 213 insertions(+) diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index 6e323253f7..762a9c2505 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -1267,6 +1267,219 @@ test_expect_success 'commit-exclusion(excluding_type=ET_REACHABLE): part of pack test_line_count = 4 filelist ' +test_expect_success 'tree-exclusion(excluding_type=ET_SELF): part of packfile response provided as URI' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + test_when_finished "rm -rf \"$P\" http_child log *found" && + excluding_type="0" && + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + # Dir struct + # . + # |-- A.t + # |-- my-tree + # | `-- my-blob + # `-- other-tree + # |-- other-blob + # `-- sub-tree + # `-- sub-blob + mkdir "$P"/my-tree && + echo my-blob >"$P"/my-tree/my-blob && + git -C "$P" add my-tree && + mkdir "$P"/other-tree && + echo other-blob >"$P"/other-tree/other-blob && + mkdir "$P"/other-tree/sub-tree && + echo sub-blob >"$P"/other-tree/sub-tree/sub-blob && + git -C "$P" add other-tree && + test_commit -C "$P" A && + commith=$(git -C "$P" rev-parse A) && + roottreeh=$(git -C "$P" rev-parse A:) && + ah=$(git -C "$P" hash-object A.t) && + mytreeh=$(git -C "$P" ls-tree HEAD my-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + othertreeh=$(git -C "$P" ls-tree HEAD other-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + subtreeh=$(git -C "$P" ls-tree HEAD other-tree/sub-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + myblobh=$(git -C "$P" hash-object my-tree/my-blob) && + otherblobh=$(git -C "$P" hash-object other-tree/other-blob) && + subblobh=$(git -C "$P" hash-object other-tree/sub-tree/sub-blob) && + + configure_exclusion tree "$P" "$mytreeh" new 0 >h && + configure_exclusion tree "$P" "$othertreeh" new 0 >h2 && + + GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ + git -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child && +# Ensure that my-tree and other-tree and theirs complementary set are in separate packfiles. + for idx in http_child/.git/objects/pack/*.idx + do + git verify-pack --object-format=$(test_oid algo) --verbose $idx >out && + { + grep "^[0-9a-f]\{16,\} " out || : + } >out.objectlist && + if test_line_count = 1 out.objectlist + then + if grep $mytreeh out + then + >mytreehfound + fi && + if grep $othertreeh out + then + >othertreehfound + fi + elif test_line_count = 7 out.objectlist + then + if grep $commith out + then + >commithfound + fi && + if grep $roottreeh out + then + >roottreehfound + fi && + if grep $subtreeh out + then + >subtreehfound + fi && + if grep $ah out + then + >ahfound + fi && + if grep $myblobh out + then + >myblobhfound + fi && + if grep $otherblobh out + then + >otherblobhfound + fi && + if grep $subblobh out + then + >subblobhfound + fi + fi + done && + test -f mytreehfound && + test -f myblobhfound && + test -f othertreehfound && + test -f otherblobhfound && + test -f subtreehfound && + test -f subblobhfound && + test -f commithfound && + test -f roottreehfound && + test -f ahfound && + # Ensure that there are exactly 3 packfiles with associated .idx + ls http_child/.git/objects/pack/*.pack \ + http_child/.git/objects/pack/*.idx >filelist && + test_line_count = 6 filelist +' + +test_expect_success 'tree-exclusion(excluding_type=ET_INCLUDE), part of packfile response provided as URI' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + test_when_finished "rm -rf \"$P\" http_child log *found" && + excluding_type="1" && + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + # Dir struct + # . + # |-- A.t + # |-- my-tree + # | `-- my-blob + # `-- other-tree + # |-- other-blob + # `-- sub-tree + # `-- sub-blob + mkdir "$P"/my-tree && + echo my-blob >"$P"/my-tree/my-blob && + git -C "$P" add my-tree && + mkdir "$P"/other-tree && + echo other-blob >"$P"/other-tree/other-blob && + mkdir "$P"/other-tree/sub-tree && + echo sub-blob >"$P"/other-tree/sub-tree/sub-blob && + git -C "$P" add other-tree && + test_commit -C "$P" A && + + commith=$(git -C "$P" rev-parse A) && + roottreeh=$(git -C "$P" rev-parse A:) && + ah=$(git -C "$P" hash-object A.t) && + mytreeh=$(git -C "$P" ls-tree HEAD my-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + othertreeh=$(git -C "$P" ls-tree HEAD other-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + subtreeh=$(git -C "$P" ls-tree HEAD other-tree/sub-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + myblobh=$(git -C "$P" hash-object my-tree/my-blob) && + otherblobh=$(git -C "$P" hash-object other-tree/other-blob) && + subblobh=$(git -C "$P" hash-object other-tree/sub-tree/sub-blob) && + + configure_exclusion tree "$P" "$mytreeh" new $excluding_type >h && + configure_exclusion tree "$P" "$othertreeh" new $excluding_type >h2 && + + GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ + git -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child && + # Ensure that my-tree and other-tree and theirs complementary set are in separate packfiles. + for idx in http_child/.git/objects/pack/*.idx + do + git verify-pack --object-format=$(test_oid algo) --verbose $idx >out && + { + grep "^[0-9a-f]\{16,\} " out || : + } >out.objectlist && + if test_line_count = 3 out.objectlist + then + if grep $commith out + then + >commithfound + fi && + if grep $roottreeh out + then + >roottreehfound + fi && + if grep $ah out + then + >ahfound + fi + elif test_line_count = 2 out.objectlist + then + if grep $mytreeh out + then + >mytreehfound + fi && + if grep $myblobh out + then + >myblobhfound + fi + elif test_line_count = 4 out.objectlist + then + if grep $othertreeh out + then + >othertreehfound + fi && + if grep $otherblobh out + then + >otherblobhfound + fi && + if grep $subtreeh out + then + >subtreehfound + fi && + if grep $subblobh out + then + >subblobhfound + fi + fi + done && + test -f mytreehfound && + test -f myblobhfound && + test -f othertreehfound && + test -f otherblobhfound && + test -f subtreehfound && + test -f subblobhfound && + test -f commithfound && + test -f roottreehfound && + test -f ahfound && + # Ensure that there are exactly 3 packfiles with associated .idx + ls http_child/.git/objects/pack/*.pack \ + http_child/.git/objects/pack/*.idx >filelist && + test_line_count = 6 filelist +' + test_expect_success 'fetching with valid packfile URI but invalid hash fails' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && test_when_finished "rm -rf \"$P\" http_child log" && From patchwork Tue Oct 19 11:38:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12569641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41179C433EF for ; Tue, 19 Oct 2021 11:40:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 29A836137C for ; Tue, 19 Oct 2021 11:40:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235481AbhJSLmU (ORCPT ); Tue, 19 Oct 2021 07:42:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235480AbhJSLmK (ORCPT ); Tue, 19 Oct 2021 07:42:10 -0400 Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 04B10C06177E for ; Tue, 19 Oct 2021 04:39:38 -0700 (PDT) Received: by mail-pg1-x52b.google.com with SMTP id s136so15854590pgs.4 for ; Tue, 19 Oct 2021 04:39:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DPYV815Opfv1tsavPSYetwkcL45amc6HO/kOMN247hE=; b=K6njnMRi0u5f7hE0riaoYe5kTJxRL+Xj2j1FwzRiO6s7NCsrnW+ZLylsFXOedP2v4+ 7HUfJNw7XgrKPixvhAfAHCISV/eSpabKPYHmoGcmLyGTkTYgJoTtSe6hfrBUo35/W8lA ojS7CEhmKkSpKCCqZVB/JPxREYjcQ1LPXV0muwbfpunHBEx9NOQsih5/dWGrzKbeZzRU ZCgO75a6XB+RwrmtXCRAV10KDgAEjce8EdmWiryw9ykHOKY6CNXaITz6IZAfsobalB5i YTfrmxETZYbBJt0G+t4OB1rH+zYZwK+5mZwcOesUHHWuKyv4pcvWU20bliaa/i20kgLZ YRjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DPYV815Opfv1tsavPSYetwkcL45amc6HO/kOMN247hE=; b=8O9fp2VlY/oSoIycYEcizYBZrUsKq0iL0x5V/NvieDSAav0+AVoXetolZXy5W5z592 dALAEWOv0W991jc8MwD31nK9Vg+a0LIUaDgPisa38hrMqoYy1FSGqmi8NB+VRQsORYlH yGzUXe1IH9IEpzkwl2My3OKuiWvJtFAJsFcIiPXeKOMRT2gKo1MNwqj+7fgG+Sv9cnsv 6hJ/h2B/LynfQPQ/u+GARClmepxEXhzeP5cVvOBPt2613KQq69/mn8iNW3BySMBwBZpB TkZLH9qE8GdmUXg/xmgqNJXpohR1qtZ0m29c7aqW5WeGxVVSr02xN4LgEfEcM+MpE4ya x5iQ== X-Gm-Message-State: AOAM532ola2AIofqq6RcZsGmVqILCDFUhRV4qI1GqdqWn/XKLL4SDA9P +7p+HiTtZBRROObXbIDAah2DjjLcLkk= X-Google-Smtp-Source: ABdhPJy70Ta5QdQuMIAOkjrbz+o+fSpnG+kDNoah3ebdjIeQHj2+qNsPcHrGmbccEx/14vpnH4/4rg== X-Received: by 2002:a62:7904:0:b0:44c:ca08:1956 with SMTP id u4-20020a627904000000b0044cca081956mr34712339pfc.47.1634643577306; Tue, 19 Oct 2021 04:39:37 -0700 (PDT) Received: from localhost.localdomain ([205.204.117.96]) by smtp.gmail.com with ESMTPSA id l207sm16430861pfd.199.2021.10.19.04.39.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Oct 2021 04:39:37 -0700 (PDT) From: Teng Long X-Google-Original-From: Teng Long To: git@vger.kernel.org Cc: gitster@pobox.com, avarab@gmail.com, jonathantanmy@google.com, bagasdotme@gmail.com, adlternative@gmail.com, stolee@gmail.com, Teng Long Subject: [PATCH v6 10/12] packfile-uri: support for excluding tags Date: Tue, 19 Oct 2021 19:38:34 +0800 Message-Id: <16c41c40a26e72d64959a332fbf7dc1c9794e70c.1634634814.git.tenglong@alibaba-inc.com> X-Mailer: git-send-email 2.31.1.453.g945ddc3a74.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Teng Long This commit brings the tags exclusion feature of packfile-uri. The excluding level of tag is supported with "ET_SELF", "ET_INCLUDE" and "ET_REACHABLE". Exclusion scope on different level: 1. When a tag is specified to be excluded with level "ET_SELF", only the tag object itself will be excluded. 2. When it's specified to be excluded with level "ET_INCLUDE", exclude the referenced commit, and all trees and blobs contained in its top-level. If it's a annotated tag, the tag object will be excluded too. 3. When it is specified with level "ET_REACHABLE", exclude the ancestors of the referenced commit, as well as the objects need to be excluded under the level "ET_INCLUDED". Signed-off-by: Teng Long --- builtin/pack-objects.c | 6 +++++- list-objects.c | 20 ++++++++++++++++---- object.c | 1 + object.h | 1 + revision.c | 13 ++++++++++++- revision.h | 9 +++++++++ 6 files changed, 44 insertions(+), 6 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 6713e734fb..073c3815a1 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1416,6 +1416,9 @@ static int want_object_in_pack(const struct object_id *oid, if (referred_objs) { struct commit *commit = referred_objs->commit; struct object_list *trees = referred_objs->trees; + struct object_list *tags = referred_objs->tags; + if (want_exclude_object(tags)) + return 0; if (commit) { commit_ex = oidmap_get(&configured_exclusions, &commit->object.oid); if (match_packfile_uri_exclusions(commit_ex) && commit_ex->level > ET_SELF) @@ -3344,7 +3347,8 @@ static void read_object_list_from_stdin(void) static void show_commit(struct commit *commit, struct show_info *info) { - add_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, NULL); + struct referred_objects *referred_objs = info->show_cache; + add_object_entry(&commit->object.oid, OBJ_COMMIT, NULL, 0, referred_objs); commit->object.flags |= OBJECT_ADDED; if (write_bitmap_index) diff --git a/list-objects.c b/list-objects.c index 40292e2cc8..2e241e8707 100644 --- a/list-objects.c +++ b/list-objects.c @@ -348,9 +348,10 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx, for (i = 0; i < ctx->revs->pending.nr; i++) { struct object_array_entry *pending = ctx->revs->pending.objects + i; struct object *obj = pending->item; - struct referred_objects *referred_objs = pending->referred_objects; const char *name = pending->name; const char *path = pending->path; + struct referred_objects *referred_objs = pending->referred_objects; + struct commit_wraps_entry *cw_entry = NULL; if (obj->flags & (UNINTERESTING | SEEN)) continue; if (obj->type == OBJ_TAG) { @@ -358,6 +359,10 @@ static void traverse_trees_and_blobs(struct traversal_context *ctx, ctx->show_object(obj, name, &show_info); continue; } + if (referred_objs->commit) + cw_entry = oidmap_get(ctx->revs->commit_wraps, &referred_objs->commit->object.oid); + if (cw_entry) + referred_objs->tags = cw_entry->wraps; if (!path) path = ""; if (obj->type == OBJ_TREE) { @@ -378,12 +383,14 @@ static void do_traverse(struct traversal_context *ctx) { struct commit *commit; struct strbuf csp; /* callee's scratch pad */ - struct show_info show_info; + struct commit_wraps_entry *entry; + struct oidmap *commit_wraps = ctx->revs->commit_wraps; + struct referred_objects referred_objs = { NULL, NULL, NULL }; + struct show_info show_info = { ctx->show_data , NULL }; strbuf_init(&csp, PATH_MAX); - show_info.show_data = ctx->show_data; - show_info.show_cache = NULL; + show_info.show_cache = &referred_objs; while ((commit = get_revision(ctx->revs)) != NULL) { /* @@ -401,6 +408,11 @@ static void do_traverse(struct traversal_context *ctx) oid_to_hex(&commit->object.oid)); } + if (commit_wraps) { + entry = oidmap_get(commit_wraps, &commit->object.oid); + referred_objs.tags = entry ? entry->wraps : NULL; + } + ctx->show_commit(commit, &show_info); if (ctx->revs->tree_blobs_in_commit_order) diff --git a/object.c b/object.c index 895068cbc2..167cc87ec9 100644 --- a/object.c +++ b/object.c @@ -335,6 +335,7 @@ void add_object_array_with_path_and_referred_commit(struct object *obj, const ch referred_objs = xmalloc(sizeof(struct referred_objects)); referred_objs->commit = referred_commit; referred_objs->trees = NULL; + referred_objs->tags = NULL; if (nr >= alloc) { alloc = (alloc + 32) * 2; diff --git a/object.h b/object.h index 618d674249..42a2178531 100644 --- a/object.h +++ b/object.h @@ -66,6 +66,7 @@ struct object_array { struct referred_objects{ struct commit *commit; struct object_list *trees; + struct object_list *tags; }; #define OBJECT_ARRAY_INIT { 0, 0, NULL } diff --git a/revision.c b/revision.c index 89a8b311ea..d16f9bbbb9 100644 --- a/revision.c +++ b/revision.c @@ -416,14 +416,17 @@ static struct commit *handle_commit(struct rev_info *revs, const char *path = entry->path; unsigned int mode = entry->mode; unsigned long flags = object->flags; + struct object_list *wraps = NULL; /* * Tag object? Look what it points to.. */ while (object->type == OBJ_TAG) { struct tag *tag = (struct tag *) object; - if (revs->tag_objects && !(flags & UNINTERESTING)) + if (revs->tag_objects && !(flags & UNINTERESTING)) { add_pending_object(revs, object, tag->tag); + object_list_insert(object, &wraps); + } object = parse_object(revs->repo, get_tagged_oid(tag)); if (!object) { if (revs->ignore_missing_links || (flags & UNINTERESTING)) @@ -449,6 +452,14 @@ static struct commit *handle_commit(struct rev_info *revs, */ if (object->type == OBJ_COMMIT) { struct commit *commit = (struct commit *)object; + struct oidmap *commit_wraps = malloc(sizeof(struct oidmap)); + struct commit_wraps_entry *cw_entry = xmalloc(sizeof(struct commit_wraps_entry)); + + oidmap_init(commit_wraps, 0); + cw_entry->e.oid = object->oid; + cw_entry->wraps = wraps; + oidmap_put(commit_wraps, cw_entry); + revs->commit_wraps = commit_wraps; if (repo_parse_commit(revs->repo, commit) < 0) die("unable to parse commit %s", name); diff --git a/revision.h b/revision.h index b2e0c0b9b7..a633ea2174 100644 --- a/revision.h +++ b/revision.h @@ -8,6 +8,7 @@ #include "pretty.h" #include "diff.h" #include "commit-slab-decl.h" +#include "oidmap.h" /** * The revision walking API offers functions to build a list of revisions @@ -64,6 +65,11 @@ struct bloom_key; struct bloom_filter_settings; define_shared_commit_slab(revision_sources, char *); +struct commit_wraps_entry { + struct oidmap_entry e; + struct object_list *wraps; +}; + struct rev_cmdline_info { unsigned int nr; unsigned int alloc; @@ -321,6 +327,9 @@ struct rev_info { /* misc. flags related to '--no-kept-objects' */ unsigned keep_pack_cache_flags; + + /* The commit_wraps caches the referred wrapped objects(such as tags) of a commit */ + struct oidmap *commit_wraps; }; int ref_excluded(struct string_list *, const char *path); From patchwork Tue Oct 19 11:38:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12569643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6DA5C433F5 for ; Tue, 19 Oct 2021 11:40:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A0C8A6137D for ; Tue, 19 Oct 2021 11:40:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230097AbhJSLmX (ORCPT ); Tue, 19 Oct 2021 07:42:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235482AbhJSLmK (ORCPT ); Tue, 19 Oct 2021 07:42:10 -0400 Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB05CC061746 for ; Tue, 19 Oct 2021 04:39:41 -0700 (PDT) Received: by mail-pg1-x52c.google.com with SMTP id t7so4636981pgl.9 for ; Tue, 19 Oct 2021 04:39:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xeU5XF/yxD/YIrF+nKxiwA2RS/Ezk89f5MbGGR2qIz4=; b=F6X59cWH8rCm/Q93rBHYsWoxfWWPWlfoRXKAk3vC12UyJXLc9nkjgQjw/yF30jilCt zllsRAqkD4jU5QDKCUMxWiiHag7gDIqB6r33mxrSSHeHu5fO888Sc4hlgvcYHt6+0NED +JFW9sy8sWBFRrC7b8Nl+g9T1DflmZ/+qXmH3zJiAzxGe3gxZ7RGLa6doQ/4pi8dL0vu jWtQtRcY6i3jdKyhuohD5iEV69LMbmYGoO+CKF/t9jKkemjyNc0RK30gGlb0JxTyKxmm pSlkGtV2gnYdV4BT+DqX+ZuM3/x4bmPj9+B0hj90HBEoWK+70S+h8YMPA/8/+UPu7ePP x5jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xeU5XF/yxD/YIrF+nKxiwA2RS/Ezk89f5MbGGR2qIz4=; b=0KEIstox6b1ccRpbGqkYiYkSc+udqfRo+wMMyi1Umj/7hFTaINsFmu8GcjCLFT8YOu 6Jm9FZyc9qc6fe08T7RZG20eKqRGNQJP0lINred+uVkqzdDDB0I0D5ZYx4zB2LQy/w0Y /CFg+keokwVkKGHNoWmczmI5+qpbOePjiqwDoTOLkQxgjAx0VGFyeYRKySenFgELcfjo rgaHpjoA/2CzeA4auU8yo5ygDDMCUksLQ0gTimiuLpnnkAa/m25HKNQs0rqvot0PElU+ JQsYZpaDJjNSZTkCVtGQPWzw/44pTt4Mo8KIS1cBKXyQ2b3JdhQPXPvJH5hOhapFONm6 mB+Q== X-Gm-Message-State: AOAM530utaHLoRwaQeSDIZCS7IvZqa3GH7NMljJ9ZbG0aVKw4OWl7xQJ 8HabOByYI0pYl1d2DTIXewgYEY0pwsU= X-Google-Smtp-Source: ABdhPJyN806i7P4JFvyuD1MRtn1NsvE5ogNcSkkdNrdqCV/BCZzxqYAnat5w6+aoQv/6PP8IccG9og== X-Received: by 2002:a05:6a00:b94:b0:44c:db51:30cc with SMTP id g20-20020a056a000b9400b0044cdb5130ccmr34850105pfj.47.1634643579758; Tue, 19 Oct 2021 04:39:39 -0700 (PDT) Received: from localhost.localdomain ([205.204.117.96]) by smtp.gmail.com with ESMTPSA id l207sm16430861pfd.199.2021.10.19.04.39.37 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Oct 2021 04:39:39 -0700 (PDT) From: Teng Long X-Google-Original-From: Teng Long To: git@vger.kernel.org Cc: gitster@pobox.com, avarab@gmail.com, jonathantanmy@google.com, bagasdotme@gmail.com, adlternative@gmail.com, stolee@gmail.com, Teng Long Subject: [PATCH v6 11/12] t5702: test cases for excluding tags Date: Tue, 19 Oct 2021 19:38:35 +0800 Message-Id: <171ece533b4aecf270d93f523086a257dd3edb32.1634634814.git.tenglong@alibaba-inc.com> X-Mailer: git-send-email 2.31.1.453.g945ddc3a74.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- t/t5702-protocol-v2.sh | 197 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 197 insertions(+) diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index 762a9c2505..f91ae62fd8 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -1480,6 +1480,203 @@ test_expect_success 'tree-exclusion(excluding_type=ET_INCLUDE), part of packfile test_line_count = 6 filelist ' +test_expect_success 'tag-exclusion(excluding_type=ET_SELF): part of packfile response provided as URI' ' + test_when_finished "rm -rf \"$P\" http_child log" && + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + echo my-blob >"$P/my-blob" && + git -C "$P" add my-blob && + test_commit -C "$P" A && + git -C "$P" tag -a -m "annotated_tag" tagA && + + tagh=$(git -C "$P" rev-parse tagA) && + commith=$(git -C "$P" rev-parse A) && + roottreeh=$(git -C "$P" rev-parse A:) && + ah=$(git -C "$P" hash-object A.t) && + myblobh=$(git -C "$P" hash-object my-blob) && + + configure_exclusion tag "$P" "$tagh" new 0 >h2 && + git init http_child && + GIT_TRACE=1 GIT_TRACE_PACKET=`pwd`/log GIT_TEST_SIDEBAND_ALL=1 \ + git -C http_child \ + -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + fetch --tags "$HTTPD_URL/smart/http_parent" && + + # Ensure that my-tree and other-tree and theirs complementary set are in separate packfiles. + for idx in http_child/.git/objects/pack/*.idx + do + git verify-pack --object-format=$(test_oid algo) --verbose $idx >out && + { + grep "^[0-9a-f]\{16,\} " out || : + } >out.objectlist && + if test_line_count = 1 out.objectlist + then + if grep $tagh out + then + >taghfound + fi + + elif test_line_count = 4 out.objectlist + then + if grep $commith out + then + >commithfound + fi && + if grep $roottreeh out + then + >roottreehfound + fi && + if grep $ah out + then + >ahfound + fi && + if grep $myblobh out + then + >myblobhfound + fi + fi + done && + test -f myblobhfound && + test -f commithfound && + test -f roottreehfound && + test -f ahfound && + test -f taghfound && + # Ensure that there are exactly 3 packfiles with associated .idx + ls http_child/.git/objects/pack/*.pack \ + http_child/.git/objects/pack/*.idx >filelist && + test_line_count = 4 filelist +' + +test_expect_success 'tag-exclusion(excluding_type=ET_INCLUDE): part of packfile response provided as URI' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + test_when_finished "rm -rf \"$P\" http_child log" && + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + echo my-blob >"$P/my-blob" && + git -C "$P" add my-blob && + test_commit -C "$P" A && + git -C "$P" tag -a -m "annotated_tag" tagA && + + tagh=$(git -C "$P" rev-parse tagA) && + commith=$(git -C "$P" rev-parse A) && + roottreeh=$(git -C "$P" rev-parse A:) && + ah=$(git -C "$P" hash-object A.t) && + myblobh=$(git -C "$P" hash-object my-blob) && + + configure_exclusion tag "$P" "$tagh" new 1 >h2 && + + git init http_child && + GIT_TRACE=1 GIT_TRACE_PACKET=`pwd`/log GIT_TEST_SIDEBAND_ALL=1 \ + git -C http_child \ + -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + fetch --tags "$HTTPD_URL/smart/http_parent" && + + # Ensure that my-tree and other-tree and theirs complementary set are in separate packfiles. + for idx in http_child/.git/objects/pack/*.idx + do + git verify-pack --object-format=$(test_oid algo) --verbose $idx >out && + { + grep "^[0-9a-f]\{16,\} " out || : + } >out.objectlist && + if test_line_count = 5 out.objectlist + then + if grep $tagh out + then + >taghfound + fi && + if grep $commith out + then + >commithfound + fi && + if grep $roottreeh out + then + >roottreehfound + fi && + if grep $ah out + then + >ahfound + fi && + if grep $myblobh out + then + >myblobhfound + fi + fi + done && + test -f myblobhfound && + test -f commithfound && + test -f roottreehfound && + test -f ahfound && + test -f taghfound && + # Ensure that there are exactly 3 packfiles with associated .idx + ls http_child/.git/objects/pack/*.pack \ + http_child/.git/objects/pack/*.idx >filelist && + test_line_count = 4 filelist +' + +test_expect_success 'tag-exclusion(excluding_type=ET_REACHABLE): part of packfile response provided as URI' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + test_when_finished "rm -rf \"$P\" http_child log" && + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + + + mkdir "$P"/my-tree && + echo my-blob >"$P"/my-tree/my-blob && + git -C "$P" add my-tree && + test_commit -C "$P" A && + + mkdir "$P"/other-tree && + echo other-blob >"$P"/other-tree/other-blob && + git -C "$P" add other-tree && + test_commit -C "$P" B && + + git -C "$P" tag -a -m "tag X" tagX && + + tagh=$(git -C "$P" rev-parse tagX) && + commitAh=$(git -C "$P" rev-parse A) && + commitBh=$(git -C "$P" rev-parse B) && + roottreeAh=$(git -C "$P" rev-parse A:) && + roottreeBh=$(git -C "$P" rev-parse B:) && + mytreeh=$(git -C "$P" ls-tree HEAD my-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + othertreeh=$(git -C "$P" ls-tree HEAD other-tree | sed -ne "s/.*\($OID_REGEX\).*/\1/p") && + ah=$(git -C "$P" hash-object A.t) && + bh=$(git -C "$P" hash-object B.t) && + myblobh=$(git -C "$P" hash-object my-tree/my-blob) && + otherblobh=$(git -C "$P" hash-object other-tree/other-blob) && + + configure_exclusion tag "$P" "$tagh" new 2 >h && + + GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ + git -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child && + + # Ensure that my-tree and other-tree and theirs complementary set are in separate packfiles. + for idx in http_child/.git/objects/pack/*.idx + do + git verify-pack --object-format=$(test_oid algo) --verbose $idx >out && + { + grep "^[0-9a-f]\{16,\} " out || : + } >out.objectlist && + if test_line_count = 11 out.objectlist + then + >fullpackfound + elif test_line_count = 0 out.objectlist + then + >emptypackfound + fi + done && + test -f fullpackfound && + test -f emptypackfound && + # Ensure that there are exactly 3 packfiles with associated .idx + ls http_child/.git/objects/pack/*.pack \ + http_child/.git/objects/pack/*.idx >filelist && + test_line_count = 4 filelist +' + test_expect_success 'fetching with valid packfile URI but invalid hash fails' ' P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && test_when_finished "rm -rf \"$P\" http_child log" && From patchwork Tue Oct 19 11:38:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Teng Long X-Patchwork-Id: 12569645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD63AC433F5 for ; Tue, 19 Oct 2021 11:40:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AA4D86137C for ; Tue, 19 Oct 2021 11:40:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235518AbhJSLmZ (ORCPT ); Tue, 19 Oct 2021 07:42:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53728 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235483AbhJSLmL (ORCPT ); Tue, 19 Oct 2021 07:42:11 -0400 Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CF0F1C061745 for ; Tue, 19 Oct 2021 04:39:42 -0700 (PDT) Received: by mail-pj1-x102d.google.com with SMTP id ez7-20020a17090ae14700b001a132a1679bso1839808pjb.0 for ; Tue, 19 Oct 2021 04:39:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8EofLOsIBlOIuEE/Mkpq365W7kqYJ2FXkqaC3BwMhBs=; b=Unixh4M9FSaTudRonCtakcvaNepAby2Wf+i1DFR0Dmb4w5HFdWZSsTCmfqTsaccRri QXQKgnAXXimVlargxQXM5zmMqJ1l6ODNCDC5ZOSFHt5867ECg5bni+X1tPTd1XDXCovy ZAbY9Hm+oOFlt0eQ0ytDurlffzrBgpiMKyWMZ3DDaPNXQqsgbgQSrAU9yUhYSe7xg3XT zWM7sBemlsYQ14R5xGTsrCsz+Oc1iEgwf0XK+pllt4XVVZaWKehyp6tXoLCXhsAqc5g7 U/Et4GYgBzsBwa0RuEDEHU5D3x7ozHLW3TtzTYCgSIRd0jE8RAyZhAzxmpI+24YDKC24 Qfiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8EofLOsIBlOIuEE/Mkpq365W7kqYJ2FXkqaC3BwMhBs=; b=7N4VQmmcjW2EiqdTnBUsC5zGnpG9RKdH4Dy3xRGYN+DRgHUF85whWe+dZV3mCzihNJ Fgz75YtJAH5FiYfw/2amMyCnsHd0XkPks2XIJ+nMWkOVc9jQj8mOZSNU7BTWcginSpgY Rvt+GtJr+yCH3H3mfLnAd4ip9ClGrHMHkwA3I1NmtYWSLVJBD5vy5xWks9iR5ws+odTh eTyFJoWW0NKUBhZM2g/3kFklBtMqswJUSzCh1X10qZEhJDWhQobWlq9GdIz8mZZd95N8 ljQXYF2iTvqXjuZ/galWqALs6GezNcqblSaz0ArMhZIt8GT7GW7hGea8O1GyswkqRAD1 cLQg== X-Gm-Message-State: AOAM531mpgBq96dsoWtwwAv98dBXuJxJhcsaN2WbZUevNAIIpsle3w+f 4yTlBuArq1l6YvNNuSRvWghj0V7Bg0I= X-Google-Smtp-Source: ABdhPJz9j8a3GpIYzdvZA7TjjCvjALAgKkON3E2a7h708lX9+zDqL2UxqnR9DsZ4Nb0eQzJrGOJB9A== X-Received: by 2002:a17:90b:3ec3:: with SMTP id rm3mr6019038pjb.186.1634643582085; Tue, 19 Oct 2021 04:39:42 -0700 (PDT) Received: from localhost.localdomain ([205.204.117.96]) by smtp.gmail.com with ESMTPSA id l207sm16430861pfd.199.2021.10.19.04.39.40 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Oct 2021 04:39:41 -0700 (PDT) From: Teng Long X-Google-Original-From: Teng Long To: git@vger.kernel.org Cc: gitster@pobox.com, avarab@gmail.com, jonathantanmy@google.com, bagasdotme@gmail.com, adlternative@gmail.com, stolee@gmail.com, Teng Long Subject: [PATCH v6 12/12] packfile-uri.txt: support multiple object types Date: Tue, 19 Oct 2021 19:38:36 +0800 Message-Id: <79fc2c23cf053cb525d818b6b47e24c49b57f672.1634634814.git.tenglong@alibaba-inc.com> X-Mailer: git-send-email 2.31.1.453.g945ddc3a74.dirty In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Signed-off-by: Teng Long --- Documentation/technical/packfile-uri.txt | 74 ++++++++++++++++++++---- 1 file changed, 63 insertions(+), 11 deletions(-) diff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt index f7eabc6c76..0e6d7c8dda 100644 --- a/Documentation/technical/packfile-uri.txt +++ b/Documentation/technical/packfile-uri.txt @@ -34,14 +34,69 @@ having it advertise `packfile-uris`, tolerating the client sending include some sort of non-trivial implementation in the Minimum Viable Product, at least so that we can test the client. -This is the implementation: a feature, marked experimental, that allows the -server to be configured by one or more `uploadpack.blobPackfileUri= -` entries. Whenever the list of objects to be sent is assembled, all such -blobs are excluded, replaced with URIs. As noted in "Future work" below, the -server can evolve in the future to support excluding other objects (or other -implementations of servers could be made that support excluding other objects) -without needing a protocol change, so clients should not expect that packfiles -downloaded in this way only contain single blobs. +This is the implementation: a feature, marked experimental, that allows +the server to be configured by one or more entries with the format: + + uploadpack.excludeobject= + +Value `` is the key of entry, and the object type can be +blob, tree, commit, or tag. Value of entry has three parts, +`` is used to identify the packfile which contains the given +`` object, and `` is the URI to download the packfile by +client. For example, When a blob is configured with `uploadpack.excludeobject` +that means whenever the blob to be send is assembled, the object will +be excluded. + +In addition to excluding a single object like blob, sometimes it's +hoped to exclude not only the object itself, but also all the related +objects with it, like all the objects a tree contains or the ancestors +that a commit can reach. In these cases, the `` is designed to +distinguish the scope of exclusion, it supports three levels: + +- Level 0: Excluding a single object itself, without any objects that + have a relationship with it. + +- Level 1: Excluding object itself, and objects it contains. + +- Level 2: Excluding object itself, the objects it contains, and the + ancestors it can reach. + +If `` is configured as 0, only the object itself will be +excluded, no matter what the object type is. It is a common scenario +for large size blobs, but it does much not sense for other object types +(e.g. download a singe commit without downloading the blobs and tree +in it). + +If `` is configured as 1, not only the single object but also all +the objects in it will be excluded. This applies to scenarios where +it's wanted to exclude a specified non-blob object that includes some +lage size objects. + +- If is a blob, the result is the same as level 0, because blob +contains nothing just itself. + +- If is a tree, the tree itself, and all blobs and trees + in it will be excluded. + +- If is a commit, the commit itself, the referenced + root-tree, and all blobs and trees in the root-tree will be excluded. + +- If is a tag, the tag itself, the dereferenced commit + and all trees and blobs contained in its root-tree will be excluded. + +If `` is configured as 2, not only the objects in the scope of +level 1 , but also the reachable ancestors will be excluded if +`` is commit or tag. + +Configuration compatibility +--------------------------- + +The old configuration of packfile-uri: + + uploadpack.blobPackfileUri= + +For the old configuration is compatible with the new one, but it only +supports the exclusion of blob objects. Client design ------------- @@ -65,9 +120,6 @@ The protocol design allows some evolution of the server and client without any need for protocol changes, so only a small-scoped design is included here to form the MVP. For example, the following can be done: - * On the server, more sophisticated means of excluding objects (e.g. by - specifying a commit to represent that commit and all objects that it - references). * On the client, resumption of clone. If a clone is interrupted, information could be recorded in the repository's config and a "clone-resume" command can resume the clone in progress. (Resumption of subsequent fetches is more