From patchwork Thu Nov 29 00:27:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10703671 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A035E14D6 for ; Thu, 29 Nov 2018 00:28:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 91A6F2DBBA for ; Thu, 29 Nov 2018 00:28:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 855DB2DE8F; Thu, 29 Nov 2018 00:28:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E4E552DBBA for ; Thu, 29 Nov 2018 00:28:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727171AbeK2Lbb (ORCPT ); Thu, 29 Nov 2018 06:31:31 -0500 Received: from mail-qk1-f201.google.com ([209.85.222.201]:35217 "EHLO mail-qk1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726668AbeK2Lbb (ORCPT ); Thu, 29 Nov 2018 06:31:31 -0500 Received: by mail-qk1-f201.google.com with SMTP id k203so198609qke.2 for ; Wed, 28 Nov 2018 16:28:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Hh5iajLx3HhWu+SUvl7wG1EKAL6uFfs9Ft4PKk/aPKE=; b=eyYSZsoJDltFh8x8E0xOQ6joReX732ks9a5m0Ch1OgfSVGGSJJ02XfquMTuS5FASKH QTOOG4qEyH2qSk/XVerpIT7o/2A/QWI1rqw9y0w1jYNJKpSmi/51K1G6dGan7tONFJ// VQubFeo00Ah+Sdl5mLTUMB6UaZr51ICRSK3sOUEDDVZxchQsuSTH9l849cdkpYMdO9sk hOPCuVDvHNEKtwjEn0D/QK19ZqDCw1EtmpfgPyglk4tQX0L7ffiQz5QAybeos7Apm/5A Ezahzy6mjFn+vHQR0iIe3gJD+5fJ4XxldTp5suXjV9oVgA7im2OtU1buhgmoCzT04//0 +jVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Hh5iajLx3HhWu+SUvl7wG1EKAL6uFfs9Ft4PKk/aPKE=; b=Z5psMrjKJCmXENftwR8FBr7MlxrQl3CuQROE2bbdqUhRyqqZkg6Sc7UvkuQf9ZIix4 dxGB5RnfVqAyNQ45og08zCp7f0gBQU4dQX9gwVOnOcX9vmDU9OQsQ6vhLpEO9scfgLrV 4WhnrS10bdQlkT5WL19C4XCZqcOgMQkEwTbUJw/pBVI0eDQnCYJxfZYD9sj14z6ZTkX5 WRZp3zY6iH4RKtUH5akOwdM6qXLflY7vkqQQyz3gRhw1z+mZPCSd3VaAJF81qxe+B2TN XlofBMZstUQdIXvQdsuztmh+h1RrjQdH9+ZCsgIlgm68lBcZhKvOVSdSaoAUFJqs/Qrs e0QA== X-Gm-Message-State: AA+aEWZzmrGH35tZDBXbO+vAbzwp8LxCQEcqr/ZU2rwWi4cbgVKLgoMm L41vLdwSkYgDFFEC8y7UNsmUInGKM9VASjrs9ZPTOY4y5yUjX0ISxSl22unOTup4mFkwjElX/s9 0RdrDOmbMwoUl12+X1umIkBBWpTh1qPiY7ul0A8Y7/pllDRzg+ipN1APILeGQ X-Google-Smtp-Source: AFSGD/WE/VVWYsK7AAJp9BhfZTkp2fVbQgYtijXTVAeXWuv2YEzhvEiEn/dwgEyps8MsQjBuxr8iaY+NNy2k X-Received: by 2002:a37:12e1:: with SMTP id 94mr5035772qks.25.1543451283395; Wed, 28 Nov 2018 16:28:03 -0800 (PST) Date: Wed, 28 Nov 2018 16:27:48 -0800 In-Reply-To: <20181129002756.167615-1-sbeller@google.com> Message-Id: <20181129002756.167615-2-sbeller@google.com> Mime-Version: 1.0 References: <20181129002756.167615-1-sbeller@google.com> X-Mailer: git-send-email 2.20.0.rc1.387.gf8505762e3-goog Subject: [PATCH 1/9] sha1-array: provide oid_array_filter From: Stefan Beller To: git@vger.kernel.org Cc: jonathantanmy@google.com, Stefan Beller , Junio C Hamano Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Helped-by: Junio C Hamano Signed-off-by: Stefan Beller Signed-off-by: Junio C Hamano --- Documentation/technical/api-oid-array.txt | 5 +++++ sha1-array.c | 17 +++++++++++++++++ sha1-array.h | 3 +++ 3 files changed, 25 insertions(+) diff --git a/Documentation/technical/api-oid-array.txt b/Documentation/technical/api-oid-array.txt index 9febfb1d52..c97428c2c3 100644 --- a/Documentation/technical/api-oid-array.txt +++ b/Documentation/technical/api-oid-array.txt @@ -48,6 +48,11 @@ Functions is not sorted, this function has the side effect of sorting it. +`oid_array_filter`:: + Apply the callback function `want` to each entry in the array, + retaining only the entries for which the function returns true. + Preserve the order of the entries that are retained. + Examples -------- diff --git a/sha1-array.c b/sha1-array.c index b94e0ec0f5..d922e94e3f 100644 --- a/sha1-array.c +++ b/sha1-array.c @@ -77,3 +77,20 @@ int oid_array_for_each_unique(struct oid_array *array, } return 0; } + +void oid_array_filter(struct oid_array *array, + for_each_oid_fn want, + void *cb_data) +{ + unsigned nr = array->nr, src, dst; + struct object_id *oids = array->oid; + + for (src = dst = 0; src < nr; src++) { + if (want(&oids[src], cb_data)) { + if (src != dst) + oidcpy(&oids[dst], &oids[src]); + dst++; + } + } + array->nr = dst; +} diff --git a/sha1-array.h b/sha1-array.h index 232bf95017..55d016c4bf 100644 --- a/sha1-array.h +++ b/sha1-array.h @@ -22,5 +22,8 @@ int oid_array_for_each(struct oid_array *array, int oid_array_for_each_unique(struct oid_array *array, for_each_oid_fn fn, void *data); +void oid_array_filter(struct oid_array *array, + for_each_oid_fn want, + void *cbdata); #endif /* SHA1_ARRAY_H */ From patchwork Thu Nov 29 00:27:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10703673 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A969D15A8 for ; Thu, 29 Nov 2018 00:28:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9D7772DD6E for ; Thu, 29 Nov 2018 00:28:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 924432DDB2; Thu, 29 Nov 2018 00:28:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 332472DD8B for ; Thu, 29 Nov 2018 00:28:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727196AbeK2Lbd (ORCPT ); Thu, 29 Nov 2018 06:31:33 -0500 Received: from mail-it1-f201.google.com ([209.85.166.201]:53705 "EHLO mail-it1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726668AbeK2Lbd (ORCPT ); Thu, 29 Nov 2018 06:31:33 -0500 Received: by mail-it1-f201.google.com with SMTP id i12so689369ita.3 for ; Wed, 28 Nov 2018 16:28:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=YdgE2YzBF1vjcZzDDRV+FbG8YGc2AgCmNzvdAnXMspQ=; b=RCleeREoMTA0WctvGwzErUe1oc7FgtMUhN63zIdCAUyzm3oV7FymkoVHhdtt+fKpMG gPf0hLXxscz9eE2oyXnv77gK5D+oSx+a1ZNZZJjOfS3mpgnnBeH0tevuMY6xKTBTfOvi unSvjjm1ldYsWoyG49eEivDXcqDm84jc/1WsgRmvc4g/Lc3Qikr7z8ZYZ84D7/v+O7as 1JKhDfdCPkI3jt0akwrK1nsc4g/cUw4xgzlT1Htv3FTF9HRdDCo33UTQQb0kUVYnPhOg xuZODYfhx03MdmDI5yFwUHL5MiBgZ9W4CA8CUlihLBtC7T11jupe6keoOEtkThcDdLmW trNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=YdgE2YzBF1vjcZzDDRV+FbG8YGc2AgCmNzvdAnXMspQ=; b=NRPWuqwKrWeLd9/JAGJ/lo4ZKXkarnKOIuNWl22gANHuJGnaaxrYv0TD1crp3ilBTu +EhMgDtM1SR5PUgId1SFWkwkQG5cEZYpYqTKParw4q95WD+nG3q4jjyGc1tjMw6oCeLA qEnNo5O5+cuUQMneAa1KuSo4n1yGGT2AhqJb8TiRi7NiAJYcN8r4L98/bhB6XR6zcIU4 D1whvcrc+dRiysLBzqmmepVq39p/h5lOzJRG5f4TgtMQ2IPxQNSQ2R5ZP6Z9ufesAl1A kHhpUbyz98uO1YkUqmvh+Wt/0CcI4vQKogF1Q2YdOHywlSMQ3ylQ0yVV8881Lw1b0nb/ JCDg== X-Gm-Message-State: AA+aEWarYSKfA+RCkTx0S50qvrdQCtXYNkbwFkm2ZLrcekjUB5h68JY3 rbCYHDuxIdxn6e5olfOGDkP23xImjNLgp1zzcHsOCU/htLxYi1rDUinOg9UGQBqskZVq47eJ8g1 KbnLYR6usiy2HYTswXp/0+aEqW2xvJXziCARXFv66K9KG9vUBr3wjoeSZR9xm X-Google-Smtp-Source: AFSGD/WsbcY00LHU7SlECIrlMX42vgBtgKSSAb9Az3Jwd0b/EMDYBBRPdXygObToLa9e8Vpcm/tU4bThXZKk X-Received: by 2002:a24:138f:: with SMTP id 137mr4061043itz.18.1543451285743; Wed, 28 Nov 2018 16:28:05 -0800 (PST) Date: Wed, 28 Nov 2018 16:27:49 -0800 In-Reply-To: <20181129002756.167615-1-sbeller@google.com> Message-Id: <20181129002756.167615-3-sbeller@google.com> Mime-Version: 1.0 References: <20181129002756.167615-1-sbeller@google.com> X-Mailer: git-send-email 2.20.0.rc1.387.gf8505762e3-goog Subject: [PATCH 2/9] submodule.c: fix indentation From: Stefan Beller To: git@vger.kernel.org Cc: jonathantanmy@google.com, Stefan Beller , Junio C Hamano Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The submodule subsystem is really bad at staying within 80 characters. Fix it while we are here. Signed-off-by: Stefan Beller Signed-off-by: Junio C Hamano --- submodule.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/submodule.c b/submodule.c index 6415cc5580..bc48ea3b68 100644 --- a/submodule.c +++ b/submodule.c @@ -1271,7 +1271,8 @@ static int get_next_submodule(struct child_process *cp, if (!submodule) { const char *name = default_name_or_path(ce->name); if (name) { - default_submodule.path = default_submodule.name = name; + default_submodule.path = name; + default_submodule.name = name; submodule = &default_submodule; } } @@ -1281,8 +1282,10 @@ static int get_next_submodule(struct child_process *cp, default: case RECURSE_SUBMODULES_DEFAULT: case RECURSE_SUBMODULES_ON_DEMAND: - if (!submodule || !unsorted_string_list_lookup(&changed_submodule_names, - submodule->name)) + if (!submodule || + !unsorted_string_list_lookup( + &changed_submodule_names, + submodule->name)) continue; default_argv = "on-demand"; break; From patchwork Thu Nov 29 00:27:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10703675 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A181715A8 for ; Thu, 29 Nov 2018 00:28:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 95E2E2DD03 for ; Thu, 29 Nov 2018 00:28:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 89FAE2DD8B; Thu, 29 Nov 2018 00:28:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 338D62DD93 for ; Thu, 29 Nov 2018 00:28:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727292AbeK2Lbf (ORCPT ); Thu, 29 Nov 2018 06:31:35 -0500 Received: from mail-pl1-f202.google.com ([209.85.214.202]:49044 "EHLO mail-pl1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726668AbeK2Lbe (ORCPT ); Thu, 29 Nov 2018 06:31:34 -0500 Received: by mail-pl1-f202.google.com with SMTP id m13so113800pls.15 for ; Wed, 28 Nov 2018 16:28:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=yiW8lXiaLJEOVYH0TLWhnHJ69Ec+jl2zbZwFM4K71ew=; b=Jejvayay0q0sv+bDQl6zh6j/ncT1AwPgJQx2pvoD6tFOmMm3ttOrtnvSU+5Pd0PxQv 2RNQw4nACu5QRxGTVgprUrIjfosMHtZLN1VIv0j6JXx8JwEWCjBcpZ6l4jr0kpGqAtnl K1vU7ntExsJGP6F9hQ8ezlmbWiFeUC00hFSgVyQPAhh6xoctbfiI6VgLdXjcoY70hZ4Q rpasV5J3qQ97QjEuU8sEu/zx4q9PwPXJCEJdyFtiypfsNjSmisYoODehM8XmCh8sYB+f bSxYuLlsKUcVVdVXl+SCpvDclTOvBmGfe4KwbngZXsygbNGS8qzPLGVxM1m/twFvfqLW hG3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=yiW8lXiaLJEOVYH0TLWhnHJ69Ec+jl2zbZwFM4K71ew=; b=RnMEDdwWxIqma8xxJEam1bq0FuJ0PUuf0EA9exZGoD+7kU8jSsStpORuPAV68wSzL3 s+DzmB7buv0sHotOSaTaJbXP2m8nFvXvLp3LBTN09W9VqMSZCl4Aw8RpRkFCMnl7QLQ7 m15W4c1TkwmzGYNyHxjC4zZGnNSZmh2isrJU9K8Td+PvDIY54QQeXf2nSjKCA+ku57BX cjtgQDPsCLjHr4ndA7suWMrpi5keoiyeQv2ZwWdKWP3x91CwE4EroC8YGwvQzj17yoe0 cIH7EFdlUftjHEWzUOJMZ+wP2RXK92ZvcgngOVJ3t+mplcAsbgbnytulnLEEvp8THJZu 7ZDg== X-Gm-Message-State: AA+aEWa6ygERGceJjfuYrWU4vmvk5gUCEmZXGAP3RVN790skCcrvttmQ NLCsqndsQgDEliyAm+5uDMQesj77as8UJPy6VSM2xgZrUvtaV5jDbrwUt95okamZv+MXCyaB8un cDuXq2ySjpuqKWQrSwTnk89Y5g+0EkBoQXzTAH+ysjHab2g3Z3lSVWAef3C3y X-Google-Smtp-Source: AFSGD/WRZLwZTsv3A0iZhCPVgBSENQ1RHR8OmTrgM/hhu1goDvFdFixLR5+Eoe0VaQ41GxEovFwKFzQq42nF X-Received: by 2002:a17:902:3281:: with SMTP id z1mr2948784plb.126.1543451287601; Wed, 28 Nov 2018 16:28:07 -0800 (PST) Date: Wed, 28 Nov 2018 16:27:50 -0800 In-Reply-To: <20181129002756.167615-1-sbeller@google.com> Message-Id: <20181129002756.167615-4-sbeller@google.com> Mime-Version: 1.0 References: <20181129002756.167615-1-sbeller@google.com> X-Mailer: git-send-email 2.20.0.rc1.387.gf8505762e3-goog Subject: [PATCH 3/9] submodule.c: sort changed_submodule_names before searching it From: Stefan Beller To: git@vger.kernel.org Cc: jonathantanmy@google.com, Stefan Beller , Junio C Hamano Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We can string_list_insert() to maintain sorted-ness of the list as we find new items, or we can string_list_append() to build an unsorted list and sort it at the end just once. As we do not rely on the sortedness while building the list, we pick the "append and sort at the end" as it has better worst case execution times. Signed-off-by: Stefan Beller Signed-off-by: Junio C Hamano --- submodule.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/submodule.c b/submodule.c index bc48ea3b68..3c388f85cc 100644 --- a/submodule.c +++ b/submodule.c @@ -1283,7 +1283,7 @@ static int get_next_submodule(struct child_process *cp, case RECURSE_SUBMODULES_DEFAULT: case RECURSE_SUBMODULES_ON_DEMAND: if (!submodule || - !unsorted_string_list_lookup( + !string_list_lookup( &changed_submodule_names, submodule->name)) continue; @@ -1377,6 +1377,7 @@ int fetch_populated_submodules(struct repository *r, /* default value, "--submodule-prefix" and its value are added later */ calculate_changed_submodule_paths(r); + string_list_sort(&changed_submodule_names); run_processes_parallel(max_parallel_jobs, get_next_submodule, fetch_start_failure, From patchwork Thu Nov 29 00:27:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10703677 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4BA7814D6 for ; Thu, 29 Nov 2018 00:28:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3E1D62DAD7 for ; Thu, 29 Nov 2018 00:28:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3297F2DD8B; Thu, 29 Nov 2018 00:28:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B7A5F2DAD7 for ; Thu, 29 Nov 2018 00:28:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727295AbeK2Lbi (ORCPT ); Thu, 29 Nov 2018 06:31:38 -0500 Received: from mail-qt1-f202.google.com ([209.85.160.202]:37784 "EHLO mail-qt1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726668AbeK2Lbi (ORCPT ); Thu, 29 Nov 2018 06:31:38 -0500 Received: by mail-qt1-f202.google.com with SMTP id h17so190542qto.4 for ; Wed, 28 Nov 2018 16:28:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=gqLB1myZxel5xx7uyxBgqY5OupazpyBAhfKOVin3hXs=; b=YGnT3RFL2qwk3o7GDfTHq7o7QW7jWSs81O3MQ3WYfWC/b5bTD+vhv547zAyu4B7Uyj KbzLJGsfHcMQmbHo/Y+/QJABzV/+0Xeg78CVihuqkP7h9HwWnT08lrrsufwbldks+uA4 /JmCzy+fasfapjYKXIeMr3PeTgz7ErnlwoDc1aVLrTFbOluixOFcugoRysL3q7x6VBB3 wG0p9WhDNGIGIO9GhGNm9C8vakBetqXDHOx1e7HwRbqN80o75BZTb1lQ7t7YFW5fQ4aa YsLHbGWIlPVkwVf3nUmyoFxQcCfkfq5Ft3m8q5R7CCFkLs8oXo1gFATDmIy3AchQGI0r gJ/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=gqLB1myZxel5xx7uyxBgqY5OupazpyBAhfKOVin3hXs=; b=n4xWgW3NNn6DV8E5QiJpUSXxfe9yQw8rv+P1fx2KQh9hMuD4lqUvCx6hCunUwb7aaZ 0SDHGhhEKWbWmotCqjxyVJny55gYStWYl/z8tinxCEEnYpkpBhIzBldzJrsd9pjDsyCn 1UaEFwrFrBW7vjMG6eLXN1dkRH4yLGZl+l4PmNLb/tDuPdOLN0ls0RHflEII4RlXN/1F 5uM+SFNZ12zKiZDAZpYr+m+s+LTrKJaBqexxCMPHIcUQA6qB4nJH/HXfgu2Itpg+h+H8 9lI0lpSOIyA6s2uPTQ7rkl2Pen3IZDpcufE5puSJV4SSif2UPISb8mN/SVO7ScYA7qAP Lflw== X-Gm-Message-State: AA+aEWaMMYMlBXTEIT135MplS8Av+P/y8vcXCdn5ZnFMQlphUWoxMMvN WUocU34wORPy0Aqjxrkrrb0vlGYsmgRcfP4mlKe2r3H2J2Fq73pZaf4Wg5NbnhSvz/J9M3cgx5q kUReDl2ah6lUha5NY8ODQMAfZo71DdYdDy6HONYnnFE9aINqc2G0Aw4H7aXAg X-Google-Smtp-Source: AFSGD/XIMDu3am1lmwVMpvmsGT4GN8gVGo8DYY6tzQf5xQIQeoQOP43eMbN8MVg3GHpssvjwJ29K6rmv8B1D X-Received: by 2002:ac8:4046:: with SMTP id j6mr5262147qtl.29.1543451290013; Wed, 28 Nov 2018 16:28:10 -0800 (PST) Date: Wed, 28 Nov 2018 16:27:51 -0800 In-Reply-To: <20181129002756.167615-1-sbeller@google.com> Message-Id: <20181129002756.167615-5-sbeller@google.com> Mime-Version: 1.0 References: <20181129002756.167615-1-sbeller@google.com> X-Mailer: git-send-email 2.20.0.rc1.387.gf8505762e3-goog Subject: [PATCH 4/9] submodule.c: tighten scope of changed_submodule_names struct From: Stefan Beller To: git@vger.kernel.org Cc: jonathantanmy@google.com, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The `changed_submodule_names` are only used for fetching, so let's make it part of the struct that is passed around for fetching submodules. Signed-off-by: Stefan Beller --- submodule.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/submodule.c b/submodule.c index 3c388f85cc..f93f0aff82 100644 --- a/submodule.c +++ b/submodule.c @@ -25,7 +25,6 @@ #include "commit-reach.h" static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF; -static struct string_list changed_submodule_names = STRING_LIST_INIT_DUP; static int initialized_fetch_ref_tips; static struct oid_array ref_tips_before_fetch; static struct oid_array ref_tips_after_fetch; @@ -1136,7 +1135,8 @@ void check_for_new_submodule_commits(struct object_id *oid) oid_array_append(&ref_tips_after_fetch, oid); } -static void calculate_changed_submodule_paths(struct repository *r) +static void calculate_changed_submodule_paths(struct repository *r, + struct string_list *changed_submodule_names) { struct argv_array argv = ARGV_ARRAY_INIT; struct string_list changed_submodules = STRING_LIST_INIT_DUP; @@ -1174,7 +1174,8 @@ static void calculate_changed_submodule_paths(struct repository *r) continue; if (!submodule_has_commits(r, path, commits)) - string_list_append(&changed_submodule_names, name->string); + string_list_append(changed_submodule_names, + name->string); } free_submodules_oids(&changed_submodules); @@ -1221,8 +1222,10 @@ struct submodule_parallel_fetch { int default_option; int quiet; int result; + + struct string_list changed_submodule_names; }; -#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0} +#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0, STRING_LIST_INIT_DUP } static int get_fetch_recurse_config(const struct submodule *submodule, struct submodule_parallel_fetch *spf) @@ -1284,7 +1287,7 @@ static int get_next_submodule(struct child_process *cp, case RECURSE_SUBMODULES_ON_DEMAND: if (!submodule || !string_list_lookup( - &changed_submodule_names, + &spf->changed_submodule_names, submodule->name)) continue; default_argv = "on-demand"; @@ -1376,8 +1379,8 @@ int fetch_populated_submodules(struct repository *r, argv_array_push(&spf.args, "--recurse-submodules-default"); /* default value, "--submodule-prefix" and its value are added later */ - calculate_changed_submodule_paths(r); - string_list_sort(&changed_submodule_names); + calculate_changed_submodule_paths(r, &spf.changed_submodule_names); + string_list_sort(&spf.changed_submodule_names); run_processes_parallel(max_parallel_jobs, get_next_submodule, fetch_start_failure, @@ -1386,7 +1389,7 @@ int fetch_populated_submodules(struct repository *r, argv_array_clear(&spf.args); out: - string_list_clear(&changed_submodule_names, 1); + string_list_clear(&spf.changed_submodule_names, 1); return spf.result; } From patchwork Thu Nov 29 00:27:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10703679 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5BBD715A8 for ; Thu, 29 Nov 2018 00:28:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4FFF52DBBA for ; Thu, 29 Nov 2018 00:28:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 449812DD98; Thu, 29 Nov 2018 00:28:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D9EFB2DBBA for ; Thu, 29 Nov 2018 00:28:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727299AbeK2Lbk (ORCPT ); Thu, 29 Nov 2018 06:31:40 -0500 Received: from mail-yw1-f74.google.com ([209.85.161.74]:35355 "EHLO mail-yw1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726668AbeK2Lbk (ORCPT ); Thu, 29 Nov 2018 06:31:40 -0500 Received: by mail-yw1-f74.google.com with SMTP id d73so180495ywd.2 for ; Wed, 28 Nov 2018 16:28:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=faUM1Nfot+zZxoWKIsgc+p5XMPnc5lROOZW+XfMYYhU=; b=VrIX23SvyxvNweoVtK9H9oEOYGHhoWRYAj1FXNjaSGu7vOQKLVGz3v2qwQOB5tzHy6 x4n70mIHNpd4tvhuWyGe2ckP+m5t8iwGdiGfC06B/CZcXqGDYzGiHTN9Lf1XP+x5pP59 QqKRxQ1MYgSJx2c4GVgjIAz3OLnu3ol43Xn99QB7Arp432jQOk1qQhJS7hFTuIRhPH91 7Z8L3OZJNKvagRgLSf0ShPr3DOzqaEhBaZ0fPrZ4gXGgS907uMI33GZmJD5G+3GgnEy0 OtH3sUZzqw51Zc1K3F9AeSl2ma5XpfWn7gF6EV0SeqzYe4kAYxjxvEOS0eWrufZJt7pP DGgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=faUM1Nfot+zZxoWKIsgc+p5XMPnc5lROOZW+XfMYYhU=; b=SjbznrFjrMTb+AnN+P+9frsuvEmwQTNTROGGS4r3PkMak37hCyfHGqngW6Obl3dRkY 0Nk9JSkIXG0HxBPLgra5yUP/390SJeeTW8Rhj3QP82NuoS9W/JQLZmP7GCJEMIN5SA4t G1LbRE2kjbZHksYChjGnHxdj+VGUCJb0mYLesdmbsgKdJzlbRBjSlBheYpEwexbVXi6l TYQ3iDB1ICk6kBPd6dGEkg2UutKFb/9e0kJOHR5HFzkr/pt0EQmRC8dGEGtIspawX5WX uvGkkPdClZBtlsXlFU+X9SNazCHXNXmOmvgAnYJm9JnVEhhrR1BNHUNfoC4OMiVMd0ck 8fDw== X-Gm-Message-State: AA+aEWZr3wt78qLNKovljdzEYUUH9kCRa/N8iF/SUsnyvX7hRKdKQlwK mUmtUSiM9Z0hKSZ6LB+B9SSH1fwCYWH1pNP5hhe02F1OKNwCd+7T4V26NkFWtVeQrkCvLjJ/Vx+ 8RjOZtf02u3q0rBJeajl6EXp9hff8UL0r2eYXpikLlMIAUU/mRkFI984B/QH8 X-Google-Smtp-Source: AFSGD/Uo1xdwbqsETq9ctHYAWqEo/xFdcQhTOiKHA0Udd/QgelLHFDPIInSlgdLlGKtCsopCvHRLEKTLqQc2 X-Received: by 2002:a25:4292:: with SMTP id p140-v6mr3510514yba.98.1543451292574; Wed, 28 Nov 2018 16:28:12 -0800 (PST) Date: Wed, 28 Nov 2018 16:27:52 -0800 In-Reply-To: <20181129002756.167615-1-sbeller@google.com> Message-Id: <20181129002756.167615-6-sbeller@google.com> Mime-Version: 1.0 References: <20181129002756.167615-1-sbeller@google.com> X-Mailer: git-send-email 2.20.0.rc1.387.gf8505762e3-goog Subject: [PATCH 5/9] submodule: store OIDs in changed_submodule_names From: Stefan Beller To: git@vger.kernel.org Cc: jonathantanmy@google.com, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP 'calculate_changed_submodule_paths' uses a local list to compute the changed submodules, and then produces the result by copying appropriate items into the result list. Instead use the result list directly and prune items afterwards using string_list_remove_empty_items. By doing so we'll have access to the util pointer for longer that contains the commits that we need to fetch, which will be useful in a later patch. Signed-off-by: Stefan Beller Reviewed-by: Jonathan Tan --- submodule.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/submodule.c b/submodule.c index f93f0aff82..0c81aca6f2 100644 --- a/submodule.c +++ b/submodule.c @@ -1139,8 +1139,7 @@ static void calculate_changed_submodule_paths(struct repository *r, struct string_list *changed_submodule_names) { struct argv_array argv = ARGV_ARRAY_INIT; - struct string_list changed_submodules = STRING_LIST_INIT_DUP; - const struct string_list_item *name; + struct string_list_item *name; /* No need to check if there are no submodules configured */ if (!submodule_from_path(r, NULL, NULL)) @@ -1157,9 +1156,9 @@ static void calculate_changed_submodule_paths(struct repository *r, * Collect all submodules (whether checked out or not) for which new * commits have been recorded upstream in "changed_submodule_names". */ - collect_changed_submodules(r, &changed_submodules, &argv); + collect_changed_submodules(r, changed_submodule_names, &argv); - for_each_string_list_item(name, &changed_submodules) { + for_each_string_list_item(name, changed_submodule_names) { struct oid_array *commits = name->util; const struct submodule *submodule; const char *path = NULL; @@ -1173,12 +1172,14 @@ static void calculate_changed_submodule_paths(struct repository *r, if (!path) continue; - if (!submodule_has_commits(r, path, commits)) - string_list_append(changed_submodule_names, - name->string); + if (submodule_has_commits(r, path, commits)) { + oid_array_clear(commits); + *name->string = '\0'; + } } - free_submodules_oids(&changed_submodules); + string_list_remove_empty_items(changed_submodule_names, 1); + argv_array_clear(&argv); oid_array_clear(&ref_tips_before_fetch); oid_array_clear(&ref_tips_after_fetch); @@ -1389,7 +1390,7 @@ int fetch_populated_submodules(struct repository *r, argv_array_clear(&spf.args); out: - string_list_clear(&spf.changed_submodule_names, 1); + free_submodules_oids(&spf.changed_submodule_names); return spf.result; } From patchwork Thu Nov 29 00:27:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10703681 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3EC9D15A8 for ; Thu, 29 Nov 2018 00:28:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 321322DBBA for ; Thu, 29 Nov 2018 00:28:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 262B42DC05; Thu, 29 Nov 2018 00:28:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 612812DD03 for ; Thu, 29 Nov 2018 00:28:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727307AbeK2Lbm (ORCPT ); Thu, 29 Nov 2018 06:31:42 -0500 Received: from mail-qk1-f201.google.com ([209.85.222.201]:35961 "EHLO mail-qk1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727232AbeK2Lbm (ORCPT ); Thu, 29 Nov 2018 06:31:42 -0500 Received: by mail-qk1-f201.google.com with SMTP id h68so198181qke.3 for ; Wed, 28 Nov 2018 16:28:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=sotDYs7AyfhRYz1Gvn6FZfFqz6BJGgbwgPkk5yCvNEI=; b=VPtCuNj9ZKXSZFCJRC/uBtCmvV8TIQNep/3gtCo3ZwfvO/tESK1KYae0MC5bBpFVp4 PCeXhLYXlwnbN90JfmPvU/ru1z3rkRRuiCnhb76jWfd6fyhZ7b92J4P6+5ZcB+PL5DLL aRtqeD0tw1GryOXQaZpnVmWoNdXl0oyxy+AZ9Wqy5lV4pG3d1pgVi++QOof3JKbiRvuR Zn/JVQVl9R5+Gi7S/f4Mh3m301gpZnZP5ugX09aDxP539K/F+Df1opEaJ45jf3iCw81N hu80O4i3sxx+VzC42FPmsXbZivlnofKkzil2hAnNrHUWpVZSPsQAHo35vdrlR+ZMrIDe hWBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=sotDYs7AyfhRYz1Gvn6FZfFqz6BJGgbwgPkk5yCvNEI=; b=tKpvCXI6Khpyw/KGxbmVTPfrJhGrqoLJsa7M24cQVXlhjR5DWsIpZboG3/k7Ydm2bW M7lqFywzRNVNo5Iwmzg/tZEqipYAnaQ9skmY/FwhtvSqqwCDOPv7HSIx69BGe6W3OWNg EIC863FjJyGDiq5EaqR9/HZq2CNKhp1Uu/FC4hT1VDEhmaPtx2ihtm6vxN2Vt2M88jEm lCYtaDp6NyaK2jGfSj1vWjECnx0D+1uWbGK8RCl4Yy1urwXpUePcCkrPrN/Bnw68HYJr 3qCrvlSK5A3lGTUESDbIjqDOOxWBfBHoCogs/RLwH9idWUPlO7M08qiSGjZFO0elsBnd QoFA== X-Gm-Message-State: AA+aEWY8Dew7EM0fsqxfW8adJg4WuQBoP4RIG/KCsjjxZvzrVwLBJrY1 BmO4JL0mVhlHEGoNqRqV2/yzzUu7gTzoDTLA3cwDgfQpdqJrDPvMriSWBXQH7m3AYl0kE7xI0hy HurkES+2T7SbhLmqjaOtRlY9i+kSjD5ddTv3PIjeijXHL52BLpEA5MzVEh0Jb X-Google-Smtp-Source: AFSGD/X2THiJmd8dDx7h1nLk9JPGHWMRPlesIFAO9MG5D2ex1ZnxAFiiSALMcWzGQl2qddeJaghXbl3xMyNL X-Received: by 2002:a0c:9901:: with SMTP id h1mr5402242qvd.14.1543451294903; Wed, 28 Nov 2018 16:28:14 -0800 (PST) Date: Wed, 28 Nov 2018 16:27:53 -0800 In-Reply-To: <20181129002756.167615-1-sbeller@google.com> Message-Id: <20181129002756.167615-7-sbeller@google.com> Mime-Version: 1.0 References: <20181129002756.167615-1-sbeller@google.com> X-Mailer: git-send-email 2.20.0.rc1.387.gf8505762e3-goog Subject: [PATCH 6/9] repository: repo_submodule_init to take a submodule struct From: Stefan Beller To: git@vger.kernel.org Cc: jonathantanmy@google.com, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When constructing a struct repository for a submodule for some revision of the superproject where the submodule is not contained in the index, it may not be present in the working tree currently either. In that situation giving a 'path' argument is not useful. Upgrade the repo_submodule_init function to take a struct submodule instead. The submodule struct can be obtained via submodule_from_{path, name} or an artificial submodule struct can be passed in. While we are at it, rename the repository struct in the repo_submodule_init function, which is to be initialized, to a name that is not confused with the struct submodule as easily. Perform such renames in similar functions as well. Also move its documentation into the header file. Reviewed-by: Jonathan Tan Signed-off-by: Stefan Beller --- builtin/grep.c | 17 +++++++----- builtin/ls-files.c | 12 +++++---- builtin/submodule--helper.c | 2 +- repository.c | 27 ++++++++------------ repository.h | 12 +++++++-- t/helper/test-submodule-nested-repo-config.c | 8 +++--- 6 files changed, 43 insertions(+), 35 deletions(-) diff --git a/builtin/grep.c b/builtin/grep.c index 71df52a333..d6bd887b2d 100644 --- a/builtin/grep.c +++ b/builtin/grep.c @@ -404,7 +404,10 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, const struct object_id *oid, const char *filename, const char *path) { - struct repository submodule; + struct repository subrepo; + const struct submodule *sub = submodule_from_path(superproject, + &null_oid, path); + int hit; /* @@ -420,12 +423,12 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, return 0; } - if (repo_submodule_init(&submodule, superproject, path)) { + if (repo_submodule_init(&subrepo, superproject, sub)) { grep_read_unlock(); return 0; } - repo_read_gitmodules(&submodule); + repo_read_gitmodules(&subrepo); /* * NEEDSWORK: This adds the submodule's object directory to the list of @@ -437,7 +440,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, * store is no longer global and instead is a member of the repository * object. */ - add_to_alternates_memory(submodule.objects->objectdir); + add_to_alternates_memory(subrepo.objects->objectdir); grep_read_unlock(); if (oid) { @@ -462,14 +465,14 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, init_tree_desc(&tree, data, size); hit = grep_tree(opt, pathspec, &tree, &base, base.len, - object->type == OBJ_COMMIT, &submodule); + object->type == OBJ_COMMIT, &subrepo); strbuf_release(&base); free(data); } else { - hit = grep_cache(opt, &submodule, pathspec, 1); + hit = grep_cache(opt, &subrepo, pathspec, 1); } - repo_clear(&submodule); + repo_clear(&subrepo); return hit; } diff --git a/builtin/ls-files.c b/builtin/ls-files.c index c70a9c7158..583a0e1ca2 100644 --- a/builtin/ls-files.c +++ b/builtin/ls-files.c @@ -206,17 +206,19 @@ static void show_files(struct repository *repo, struct dir_struct *dir); static void show_submodule(struct repository *superproject, struct dir_struct *dir, const char *path) { - struct repository submodule; + struct repository subrepo; + const struct submodule *sub = submodule_from_path(superproject, + &null_oid, path); - if (repo_submodule_init(&submodule, superproject, path)) + if (repo_submodule_init(&subrepo, superproject, sub)) return; - if (repo_read_index(&submodule) < 0) + if (repo_read_index(&subrepo) < 0) die("index file corrupt"); - show_files(&submodule, dir); + show_files(&subrepo, dir); - repo_clear(&submodule); + repo_clear(&subrepo); } static void show_ce(struct repository *repo, struct dir_struct *dir, diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c index d38113a31a..4eceb8f040 100644 --- a/builtin/submodule--helper.c +++ b/builtin/submodule--helper.c @@ -2053,7 +2053,7 @@ static int ensure_core_worktree(int argc, const char **argv, const char *prefix) if (!sub) BUG("We could get the submodule handle before?"); - if (repo_submodule_init(&subrepo, the_repository, path)) + if (repo_submodule_init(&subrepo, the_repository, sub)) die(_("could not get a repository handle for submodule '%s'"), path); if (!repo_config_get_string(&subrepo, "core.worktree", &cw)) { diff --git a/repository.c b/repository.c index 5dd1486718..aabe64ee5d 100644 --- a/repository.c +++ b/repository.c @@ -166,30 +166,23 @@ int repo_init(struct repository *repo, return -1; } -/* - * Initialize 'submodule' as the submodule given by 'path' in parent repository - * 'superproject'. - * Return 0 upon success and a non-zero value upon failure. - */ -int repo_submodule_init(struct repository *submodule, +int repo_submodule_init(struct repository *subrepo, struct repository *superproject, - const char *path) + const struct submodule *sub) { - const struct submodule *sub; struct strbuf gitdir = STRBUF_INIT; struct strbuf worktree = STRBUF_INIT; int ret = 0; - sub = submodule_from_path(superproject, &null_oid, path); if (!sub) { ret = -1; goto out; } - strbuf_repo_worktree_path(&gitdir, superproject, "%s/.git", path); - strbuf_repo_worktree_path(&worktree, superproject, "%s", path); + strbuf_repo_worktree_path(&gitdir, superproject, "%s/.git", sub->path); + strbuf_repo_worktree_path(&worktree, superproject, "%s", sub->path); - if (repo_init(submodule, gitdir.buf, worktree.buf)) { + if (repo_init(subrepo, gitdir.buf, worktree.buf)) { /* * If initilization fails then it may be due to the submodule * not being populated in the superproject's worktree. Instead @@ -201,16 +194,16 @@ int repo_submodule_init(struct repository *submodule, strbuf_repo_git_path(&gitdir, superproject, "modules/%s", sub->name); - if (repo_init(submodule, gitdir.buf, NULL)) { + if (repo_init(subrepo, gitdir.buf, NULL)) { ret = -1; goto out; } } - submodule->submodule_prefix = xstrfmt("%s%s/", - superproject->submodule_prefix ? - superproject->submodule_prefix : - "", path); + subrepo->submodule_prefix = xstrfmt("%s%s/", + superproject->submodule_prefix ? + superproject->submodule_prefix : + "", sub->path); out: strbuf_release(&gitdir); diff --git a/repository.h b/repository.h index 9f16c42c1e..0e482b7d49 100644 --- a/repository.h +++ b/repository.h @@ -116,9 +116,17 @@ void repo_set_worktree(struct repository *repo, const char *path); void repo_set_hash_algo(struct repository *repo, int algo); void initialize_the_repository(void); int repo_init(struct repository *r, const char *gitdir, const char *worktree); -int repo_submodule_init(struct repository *submodule, + +/* + * Initialize the repository 'subrepo' as the submodule given by the + * struct submodule 'sub' in parent repository 'superproject'. + * Return 0 upon success and a non-zero value upon failure, which may happen + * if the submodule is not found, or 'sub' is NULL. + */ +struct submodule; +int repo_submodule_init(struct repository *subrepo, struct repository *superproject, - const char *path); + const struct submodule *sub); void repo_clear(struct repository *repo); /* diff --git a/t/helper/test-submodule-nested-repo-config.c b/t/helper/test-submodule-nested-repo-config.c index a31e2a9bea..bc97929bbc 100644 --- a/t/helper/test-submodule-nested-repo-config.c +++ b/t/helper/test-submodule-nested-repo-config.c @@ -10,19 +10,21 @@ static void die_usage(int argc, const char **argv, const char *msg) int cmd__submodule_nested_repo_config(int argc, const char **argv) { - struct repository submodule; + struct repository subrepo; + const struct submodule *sub; if (argc < 3) die_usage(argc, argv, "Wrong number of arguments."); setup_git_directory(); - if (repo_submodule_init(&submodule, the_repository, argv[1])) { + sub = submodule_from_path(the_repository, &null_oid, argv[1]); + if (repo_submodule_init(&subrepo, the_repository, sub)) { die_usage(argc, argv, "Submodule not found."); } /* Read the config of _child_ submodules. */ - print_config_from_gitmodules(&submodule, argv[2]); + print_config_from_gitmodules(&subrepo, argv[2]); submodule_free(the_repository); From patchwork Thu Nov 29 00:27:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10703683 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5B43A14D6 for ; Thu, 29 Nov 2018 00:28:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4EA562DD6E for ; Thu, 29 Nov 2018 00:28:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 436FB2DDB2; Thu, 29 Nov 2018 00:28:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 261982DD6E for ; Thu, 29 Nov 2018 00:28:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727315AbeK2Lbp (ORCPT ); Thu, 29 Nov 2018 06:31:45 -0500 Received: from mail-qk1-f201.google.com ([209.85.222.201]:53625 "EHLO mail-qk1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727232AbeK2Lbp (ORCPT ); Thu, 29 Nov 2018 06:31:45 -0500 Received: by mail-qk1-f201.google.com with SMTP id s19so153707qke.20 for ; Wed, 28 Nov 2018 16:28:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=IMRG8OwjQ6UrbAD3zjV+niybtLkNnzzjGnwtiwB748Q=; b=CQ6BdbOyB6+mdCVQ2QbfdPS4bpEPgbTyScJMtEmV4KVDmqgDNXDfWu9HmWoyUIFDIy gXYKlIuxW2W9QeGwh9SlKnH5XwPt4DxtJMqRDk132HblDC7ZFrK+LI6B76wJmxZwsgao lCrs2vwmjtdgdYdUB+QWR16452EL53b+kOYm9pHwEMRltchrQwUh939tFC18RPcTB9xH XlPN79SQEGHOtfxdtrk/4mkWmFNfDUrx7l/dUSl0DytnKZ6oXdKcPxm1eJzUTL0uP+D6 q0qtHZ3HdrzavjYBtKN1jlKLvhwTM7YRwdlQpu5Q7NcWyIv8tFcChVomIy5kYvb11H9Y S73w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=IMRG8OwjQ6UrbAD3zjV+niybtLkNnzzjGnwtiwB748Q=; b=e/Ei75hUoe+B2RAb3b7XLn7fHem5GsbXjIMUGVZGmYtf/kr7nzrm4du5GfCPeApBSY U0GYVg+Cd/fKOYDOUXxYZa2vUx8/Wi+YRDYx+NNQxzFcaS0m5L/fUklg3d7YciJ1X4gW xjDi9r0RXbEonv/T0d1QnvC9nt/JPf9eh1tU9pLRvEqpNHPs3Q+JNzpSvnmEkCs3Klzf T8HiItyZy704azS37bItaSasnwwPfi5lAaRI4djVpYG5PcbukbXsvxcx/mQXpNgsVAUz sbrX+KPgGZeMTEOBDaVZZK7KdITwLK0X7lxejJiguy/x8Kyn89ZBJtEIIDfKUi9qjQSS hW8w== X-Gm-Message-State: AA+aEWZtpOHYLjXknSRO1XvVYqr4/6bJnNg5Dw9PJGC0/0DPCleUkLGm faFlMDKMZ6XMxIY32jtrALI33MVDwe81CwB1LrVZLu6Xrz2JdEBlFZRiFElqeWsHbSk1/74zRYv gKH2exeM30SqmbRqzw2IeqVZ6fwjRE0CglrZbNZoNDYoB2ClLDJ7dPcSif3au X-Google-Smtp-Source: AFSGD/XtRyOFV5Q3IgbWJOcWDZol6MXc15XUpyMjtAwgKv/615wyZQEiEBGaobb17rWsY1R2QVV35dB6+Wl3 X-Received: by 2002:a37:6386:: with SMTP id x128mr5021615qkb.43.1543451297517; Wed, 28 Nov 2018 16:28:17 -0800 (PST) Date: Wed, 28 Nov 2018 16:27:54 -0800 In-Reply-To: <20181129002756.167615-1-sbeller@google.com> Message-Id: <20181129002756.167615-8-sbeller@google.com> Mime-Version: 1.0 References: <20181129002756.167615-1-sbeller@google.com> X-Mailer: git-send-email 2.20.0.rc1.387.gf8505762e3-goog Subject: [PATCH 7/9] submodule: migrate get_next_submodule to use repository structs From: Stefan Beller To: git@vger.kernel.org Cc: jonathantanmy@google.com, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We used to recurse into submodules, even if they were broken having only an objects directory. The child process executed in the submodule would fail though if the submodule was broken. This is tested via "fetching submodule into a broken repository" in t5526. This patch tightens the check upfront, such that we do not need to spawn a child process to find out if the submodule is broken. Signed-off-by: Stefan Beller --- submodule.c | 56 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 44 insertions(+), 12 deletions(-) diff --git a/submodule.c b/submodule.c index 0c81aca6f2..77ace5e784 100644 --- a/submodule.c +++ b/submodule.c @@ -1253,6 +1253,30 @@ static int get_fetch_recurse_config(const struct submodule *submodule, return spf->default_option; } +static struct repository *get_submodule_repo_for(struct repository *r, + const struct submodule *sub) +{ + struct repository *ret = xmalloc(sizeof(*ret)); + + if (repo_submodule_init(ret, r, sub)) { + /* + * No entry in .gitmodules? Technically not a submodule, + * but historically we supported repositories that happen to be + * in-place where a gitlink is. Keep supporting them. + */ + struct strbuf gitdir = STRBUF_INIT; + strbuf_repo_worktree_path(&gitdir, r, "%s/.git", sub->path); + if (repo_init(ret, gitdir.buf, NULL)) { + strbuf_release(&gitdir); + free(ret); + return NULL; + } + strbuf_release(&gitdir); + } + + return ret; +} + static int get_next_submodule(struct child_process *cp, struct strbuf *err, void *data, void **task_cb) { @@ -1260,12 +1284,11 @@ static int get_next_submodule(struct child_process *cp, struct submodule_parallel_fetch *spf = data; for (; spf->count < spf->r->index->cache_nr; spf->count++) { - struct strbuf submodule_path = STRBUF_INIT; - struct strbuf submodule_git_dir = STRBUF_INIT; struct strbuf submodule_prefix = STRBUF_INIT; const struct cache_entry *ce = spf->r->index->cache[spf->count]; - const char *git_dir, *default_argv; + const char *default_argv; const struct submodule *submodule; + struct repository *repo; struct submodule default_submodule = SUBMODULE_INIT; if (!S_ISGITLINK(ce->ce_mode)) @@ -1300,15 +1323,11 @@ static int get_next_submodule(struct child_process *cp, continue; } - strbuf_repo_worktree_path(&submodule_path, spf->r, "%s", ce->name); - strbuf_addf(&submodule_git_dir, "%s/.git", submodule_path.buf); strbuf_addf(&submodule_prefix, "%s%s/", spf->prefix, ce->name); - git_dir = read_gitfile(submodule_git_dir.buf); - if (!git_dir) - git_dir = submodule_git_dir.buf; - if (is_directory(git_dir)) { + repo = get_submodule_repo_for(spf->r, submodule); + if (repo) { child_process_init(cp); - cp->dir = strbuf_detach(&submodule_path, NULL); + cp->dir = xstrdup(repo->worktree); prepare_submodule_repo_env(&cp->env_array); cp->git_cmd = 1; if (!spf->quiet) @@ -1319,10 +1338,23 @@ static int get_next_submodule(struct child_process *cp, argv_array_push(&cp->args, default_argv); argv_array_push(&cp->args, "--submodule-prefix"); argv_array_push(&cp->args, submodule_prefix.buf); + + repo_clear(repo); + free(repo); ret = 1; + } else { + /* + * An empty directory is normal, + * the submodule is not initialized + */ + if (S_ISGITLINK(ce->ce_mode) && + !is_empty_dir(ce->name)) { + spf->result = 1; + strbuf_addf(err, + _("Could not access submodule '%s'"), + ce->name); + } } - strbuf_release(&submodule_path); - strbuf_release(&submodule_git_dir); strbuf_release(&submodule_prefix); if (ret) { spf->count++; From patchwork Thu Nov 29 00:27:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10703685 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E34F014D6 for ; Thu, 29 Nov 2018 00:28:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D73892DD98 for ; Thu, 29 Nov 2018 00:28:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CBAB42DD93; Thu, 29 Nov 2018 00:28:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7756C2DD6E for ; Thu, 29 Nov 2018 00:28:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727324AbeK2Lbr (ORCPT ); Thu, 29 Nov 2018 06:31:47 -0500 Received: from mail-qt1-f201.google.com ([209.85.160.201]:55105 "EHLO mail-qt1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727232AbeK2Lbr (ORCPT ); Thu, 29 Nov 2018 06:31:47 -0500 Received: by mail-qt1-f201.google.com with SMTP id z6so148667qtj.21 for ; Wed, 28 Nov 2018 16:28:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=tvU9dTNmDlKUlMg/Y4rezjD0vXRZgIt34r455l5zCyE=; b=FWKW8/EqLJ031jjrKzuUZWE6smI/Ybj0Yv3bHBLZ+OPIaHy80RkIQrW36nhSo1esfr aPY+xQwA3Bi4609IpgC7F47NonJVsU2qIVi37qicohlvEj0TV+I5gVCbWr3Otz6RY0mm bwE7TIIvpT3RnLPwUVz2t2urD8c1KH2vaAFjXAcndy6/1k3YxXH2HdboFP3dazVgH+bm sCy9JT0hYHykBy2ji3lli5oq7PzWVYxpwgnN91SnbyrCkyeghE/cSdszBPYzd8awo4Tj OAJdSs36uEB9KeCRuWlvlk7TdqU0ewhqBsUpEk44QmSMe4RRy39e3RTUBoL8HPCb9gyw oajg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=tvU9dTNmDlKUlMg/Y4rezjD0vXRZgIt34r455l5zCyE=; b=SQL7EnrNOaTzfR1d8BjNNKAOxZmOCGDez3Nclh0HO/R4/qlQY9ov6K7JWHgMHs/9Sr Zy0o1CjUblbtznOd8LJAuUFIk4PDv1nKQF0IkDGl0dP5WfAhXdIkre2f7mYz9YZSKEYu lHRJ8TEKiYvgViccxGkc/hiXpMFfvNMftg8pUnV2QlZuM5LnQ2OYcXQ4HO4NGfaRy5fm 3IBD2MO+KI3gKCntNg9gy1RosfRAOqEv3veQOeUYPwnwLXDlY7BEtoM9gfyrZ06YEykr FLdMcQs8Ac0xP07bVOqj3Ful5lvfzb7C+TXDMOAYv8DmSxrxuYEdTh2gcjr0MlMwSCqL XKaw== X-Gm-Message-State: AA+aEWbmUh4Vb1/j3jumlr+skzPAs0WciEyPdH9LEyZ4bTdT7KjbHF5Q AdB6bFTwp36f09FHbgbY9Zm5lzrZV//e+sauOn/fRBPO3xcEuXM1g0nWqmZzn9lG3xBBsfTyryy woSshQ/yYRfPoVQQynzWQcFJhcH0TS1VDSvvaLwydIJ4WrbyTtVx3nq2YnaxU X-Google-Smtp-Source: AFSGD/VI+dKezll33bi3SM34rJ5926stECI/mAC39WCa3gFVXRe0JQDu5KLPSKsHB3Q5LnAy4brR5jSEbkZu X-Received: by 2002:a37:1f62:: with SMTP id f95mr4955426qkf.46.1543451300008; Wed, 28 Nov 2018 16:28:20 -0800 (PST) Date: Wed, 28 Nov 2018 16:27:55 -0800 In-Reply-To: <20181129002756.167615-1-sbeller@google.com> Message-Id: <20181129002756.167615-9-sbeller@google.com> Mime-Version: 1.0 References: <20181129002756.167615-1-sbeller@google.com> X-Mailer: git-send-email 2.20.0.rc1.387.gf8505762e3-goog Subject: [PATCH 8/9] submodule.c: fetch in submodules git directory instead of in worktree From: Stefan Beller To: git@vger.kernel.org Cc: jonathantanmy@google.com, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Keep the properties introduced in 10f5c52656 (submodule: avoid auto-discovery in prepare_submodule_repo_env(), 2016-09-01), by fixating the git directory of the submodule. Signed-off-by: Stefan Beller --- submodule.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/submodule.c b/submodule.c index 77ace5e784..d1b6646f42 100644 --- a/submodule.c +++ b/submodule.c @@ -494,6 +494,12 @@ void prepare_submodule_repo_env(struct argv_array *out) DEFAULT_GIT_DIR_ENVIRONMENT); } +static void prepare_submodule_repo_env_in_gitdir(struct argv_array *out) +{ + prepare_submodule_repo_env_no_git_dir(out); + argv_array_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT); +} + /* Helper function to display the submodule header line prior to the full * summary output. If it can locate the submodule objects directory it will * attempt to lookup both the left and right commits and put them into the @@ -1327,8 +1333,8 @@ static int get_next_submodule(struct child_process *cp, repo = get_submodule_repo_for(spf->r, submodule); if (repo) { child_process_init(cp); - cp->dir = xstrdup(repo->worktree); - prepare_submodule_repo_env(&cp->env_array); + cp->dir = xstrdup(repo->gitdir); + prepare_submodule_repo_env_in_gitdir(&cp->env_array); cp->git_cmd = 1; if (!spf->quiet) strbuf_addf(err, "Fetching submodule %s%s\n", From patchwork Thu Nov 29 00:27:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10703687 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0C1EA15A8 for ; Thu, 29 Nov 2018 00:28:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F07C22D98A for ; Thu, 29 Nov 2018 00:28:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E4E2D2DD93; Thu, 29 Nov 2018 00:28:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CF8E82DD64 for ; Thu, 29 Nov 2018 00:28:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727327AbeK2Lbv (ORCPT ); Thu, 29 Nov 2018 06:31:51 -0500 Received: from mail-it1-f202.google.com ([209.85.166.202]:35773 "EHLO mail-it1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727232AbeK2Lbv (ORCPT ); Thu, 29 Nov 2018 06:31:51 -0500 Received: by mail-it1-f202.google.com with SMTP id c128so732384itc.0 for ; Wed, 28 Nov 2018 16:28:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ZLuqkGrLiTIYewnlFyaBCvPHouaz9uz0Q6SHtl0RwhQ=; b=NkUh+kdKcEA+4u4JPNfEBM+qDoApmbBjZowmOQ1Jf3ET/yCwzHr89gzepbeqbgZAze BucWXyNQCs3EkWp01GpiVRV59dtxQDyFZoBU+fRWpk1poZbpxYd1Tq3KMPjz4uC7jcOA HZFrhIM9hw5+0pzv+imK3TE0Ll3o7+tBKGzkYxllFjgROIUJvdXTfgEW30893C5imVuS XkS3zChnRLVUZchMysE5Owpv1gMntyQF9b24hgGk+Jeje6eQrbnehmpdaxx8RMs4HAE3 AyHSdInh3ZnxySqJj4Bz47A3xy1MMIbEg0yR4G8T/yikBGmDnIjoXWifeKnK5ircb76K 6PlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ZLuqkGrLiTIYewnlFyaBCvPHouaz9uz0Q6SHtl0RwhQ=; b=O01Ic75sM+P8PaD8S/nfTZ/MJT08uXLxzFtEv0etVZRy5egj1jNJoq715v1qBe4hWt CglvK0vHNBk4+P5YkQklKa57/UJzBGq+FiypsgIBcgGtWvqy8Oc4dVV0F2jqpqXCKw3s FtHmRyfTOfibpUz1XknPyhAsIxjXNFmR818qUXnbzsMkFsLiAYOFgBah6WJbHdsNBq/k 7ZFR3ZNkrTTFQ7B024epnQKCt4FQev4g021FXmHAmCpmbGEuTb5xJCdecRW8hzVA6xU1 O2R3prhcR8Pn/FnywM7wwZt87v4O2MVxykLyOEMqW9ucllv+pxfxpat/bOQQcKcgMaPk xzAg== X-Gm-Message-State: AA+aEWYbg1N2Zq2ajD2rhkKhwe5D2brCAoS6/dOoZ+aOcj4E8Ya0v1G4 XxrPVLgBCF0ZF+EuHsJ5eQdafQdP5JhUUIp37Lqi7fuWF++165ywKaOIHWlnVZsHHYNdSmeLXkf sT3pgxgjFCL95pBuujPiLRvG5unTtjHJu+0SGLr4Z4vFAwaPgOIMV+/s8M833 X-Google-Smtp-Source: AFSGD/W+AQyS+fD4DlV3dx04TFl0V7Me6u/x/K1XJTZgSiWN0HYDTkl8ypJq+HkGLfk3/G0s25DihTBcW/jP X-Received: by 2002:a24:6504:: with SMTP id u4mr4032527itb.25.1543451302147; Wed, 28 Nov 2018 16:28:22 -0800 (PST) Date: Wed, 28 Nov 2018 16:27:56 -0800 In-Reply-To: <20181129002756.167615-1-sbeller@google.com> Message-Id: <20181129002756.167615-10-sbeller@google.com> Mime-Version: 1.0 References: <20181129002756.167615-1-sbeller@google.com> X-Mailer: git-send-email 2.20.0.rc1.387.gf8505762e3-goog Subject: [PATCH 9/9] fetch: try fetching submodules if needed objects were not fetched From: Stefan Beller To: git@vger.kernel.org Cc: jonathantanmy@google.com, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Currently when git-fetch is asked to recurse into submodules, it dispatches a plain "git-fetch -C " (with some submodule related options such as prefix and recusing strategy, but) without any information of the remote or the tip that should be fetched. But this default fetch is not sufficient, as a newly fetched commit in the superproject could point to a commit in the submodule that is not in the default refspec. This is common in workflows like Gerrit's. When fetching a Gerrit change under review (from refs/changes/??), the commits in that change likely point to submodule commits that have not been merged to a branch yet. Try fetching a submodule by object id if the object id that the superproject points to, cannot be found. builtin/fetch used to only inspect submodules when they were fetched "on-demand", as in either on/off case it was clear whether the submodule needs to be fetched. However to know whether we need to try fetching the object ids, we need to identify the object names, which is done in this function check_for_new_submodule_commits(), so we'll also run that code in case the submodule recursion is set to "on". The submodule checks were done only when a ref in the superproject changed, these checks were extended to also be performed when fetching into FETCH_HEAD for completeness, and add a test for that too. Signed-off-by: Stefan Beller --- builtin/fetch.c | 11 +- submodule.c | 206 +++++++++++++++++++++++++++++++----- t/t5526-fetch-submodules.sh | 86 +++++++++++++++ 3 files changed, 265 insertions(+), 38 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index e0140327aa..91f9b7d9c8 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -763,9 +763,6 @@ static int update_local_ref(struct ref *ref, what = _("[new ref]"); } - if ((recurse_submodules != RECURSE_SUBMODULES_OFF) && - (recurse_submodules != RECURSE_SUBMODULES_ON)) - check_for_new_submodule_commits(&ref->new_oid); r = s_update_ref(msg, ref, 0); format_display(display, r ? '!' : '*', what, r ? _("unable to update local ref") : NULL, @@ -779,9 +776,6 @@ static int update_local_ref(struct ref *ref, strbuf_add_unique_abbrev(&quickref, ¤t->object.oid, DEFAULT_ABBREV); strbuf_addstr(&quickref, ".."); strbuf_add_unique_abbrev(&quickref, &ref->new_oid, DEFAULT_ABBREV); - if ((recurse_submodules != RECURSE_SUBMODULES_OFF) && - (recurse_submodules != RECURSE_SUBMODULES_ON)) - check_for_new_submodule_commits(&ref->new_oid); r = s_update_ref("fast-forward", ref, 1); format_display(display, r ? '!' : ' ', quickref.buf, r ? _("unable to update local ref") : NULL, @@ -794,9 +788,6 @@ static int update_local_ref(struct ref *ref, strbuf_add_unique_abbrev(&quickref, ¤t->object.oid, DEFAULT_ABBREV); strbuf_addstr(&quickref, "..."); strbuf_add_unique_abbrev(&quickref, &ref->new_oid, DEFAULT_ABBREV); - if ((recurse_submodules != RECURSE_SUBMODULES_OFF) && - (recurse_submodules != RECURSE_SUBMODULES_ON)) - check_for_new_submodule_commits(&ref->new_oid); r = s_update_ref("forced-update", ref, 1); format_display(display, r ? '!' : '+', quickref.buf, r ? _("unable to update local ref") : _("forced update"), @@ -892,6 +883,8 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, ref->force = rm->peer_ref->force; } + if (recurse_submodules != RECURSE_SUBMODULES_OFF) + check_for_new_submodule_commits(&rm->old_oid); if (!strcmp(rm->name, "HEAD")) { kind = ""; diff --git a/submodule.c b/submodule.c index d1b6646f42..1ce944a737 100644 --- a/submodule.c +++ b/submodule.c @@ -1231,8 +1231,14 @@ struct submodule_parallel_fetch { int result; struct string_list changed_submodule_names; + + /* The submodules to fetch in */ + struct fetch_task **oid_fetch_tasks; + int oid_fetch_tasks_nr, oid_fetch_tasks_alloc; }; -#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0, STRING_LIST_INIT_DUP } +#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0, \ + STRING_LIST_INIT_DUP, \ + NULL, 0, 0} static int get_fetch_recurse_config(const struct submodule *submodule, struct submodule_parallel_fetch *spf) @@ -1259,6 +1265,73 @@ static int get_fetch_recurse_config(const struct submodule *submodule, return spf->default_option; } +struct fetch_task { + struct repository *repo; + const struct submodule *sub; + unsigned free_sub : 1; /* Do we need to free the submodule? */ + + /* fetch specific oids if set, otherwise fetch default refspec */ + struct oid_array *commits; +}; + +/** + * When a submodule is not defined in .gitmodules, we cannot access it + * via the regular submodule-config. Create a fake submodule, which we can + * work on. + */ +static const struct submodule *get_non_gitmodules_submodule(const char *path) +{ + struct submodule *ret = NULL; + const char *name = default_name_or_path(path); + + if (!name) + return NULL; + + ret = xmalloc(sizeof(*ret)); + memset(ret, 0, sizeof(*ret)); + ret->path = name; + ret->name = name; + + return (const struct submodule *) ret; +} + +static struct fetch_task *fetch_task_create(struct repository *r, + const char *path) +{ + struct fetch_task *task = xmalloc(sizeof(*task)); + memset(task, 0, sizeof(*task)); + + task->sub = submodule_from_path(r, &null_oid, path); + if (!task->sub) { + /* + * No entry in .gitmodules? Technically not a submodule, + * but historically we supported repositories that happen to be + * in-place where a gitlink is. Keep supporting them. + */ + task->sub = get_non_gitmodules_submodule(path); + if (!task->sub) { + free(task); + return NULL; + } + + task->free_sub = 1; + } + + return task; +} + +static void fetch_task_release(struct fetch_task *p) +{ + if (p->free_sub) + free((void*)p->sub); + p->free_sub = 0; + p->sub = NULL; + + if (p->repo) + repo_clear(p->repo); + FREE_AND_NULL(p->repo); +} + static struct repository *get_submodule_repo_for(struct repository *r, const struct submodule *sub) { @@ -1286,39 +1359,32 @@ static struct repository *get_submodule_repo_for(struct repository *r, static int get_next_submodule(struct child_process *cp, struct strbuf *err, void *data, void **task_cb) { - int ret = 0; struct submodule_parallel_fetch *spf = data; for (; spf->count < spf->r->index->cache_nr; spf->count++) { - struct strbuf submodule_prefix = STRBUF_INIT; + int recurse_config; const struct cache_entry *ce = spf->r->index->cache[spf->count]; const char *default_argv; - const struct submodule *submodule; - struct repository *repo; - struct submodule default_submodule = SUBMODULE_INIT; + struct fetch_task *task; if (!S_ISGITLINK(ce->ce_mode)) continue; - submodule = submodule_from_path(spf->r, &null_oid, ce->name); - if (!submodule) { - const char *name = default_name_or_path(ce->name); - if (name) { - default_submodule.path = name; - default_submodule.name = name; - submodule = &default_submodule; - } - } + task = fetch_task_create(spf->r, ce->name); + if (!task) + continue; + + recurse_config = get_fetch_recurse_config(task->sub, spf); - switch (get_fetch_recurse_config(submodule, spf)) + switch (recurse_config) { default: case RECURSE_SUBMODULES_DEFAULT: case RECURSE_SUBMODULES_ON_DEMAND: - if (!submodule || + if (!task->sub || !string_list_lookup( &spf->changed_submodule_names, - submodule->name)) + task->sub->name)) continue; default_argv = "on-demand"; break; @@ -1329,11 +1395,11 @@ static int get_next_submodule(struct child_process *cp, continue; } - strbuf_addf(&submodule_prefix, "%s%s/", spf->prefix, ce->name); - repo = get_submodule_repo_for(spf->r, submodule); - if (repo) { + task->repo = get_submodule_repo_for(spf->r, task->sub); + if (task->repo) { + struct strbuf submodule_prefix = STRBUF_INIT; child_process_init(cp); - cp->dir = xstrdup(repo->gitdir); + cp->dir = task->repo->gitdir; prepare_submodule_repo_env_in_gitdir(&cp->env_array); cp->git_cmd = 1; if (!spf->quiet) @@ -1343,12 +1409,22 @@ static int get_next_submodule(struct child_process *cp, argv_array_pushv(&cp->args, spf->args.argv); argv_array_push(&cp->args, default_argv); argv_array_push(&cp->args, "--submodule-prefix"); + + strbuf_addf(&submodule_prefix, "%s%s/", + spf->prefix, + task->sub->path); argv_array_push(&cp->args, submodule_prefix.buf); - repo_clear(repo); - free(repo); - ret = 1; + spf->count++; + *task_cb = task; + + strbuf_release(&submodule_prefix); + return 1; } else { + + fetch_task_release(task); + free(task); + /* * An empty directory is normal, * the submodule is not initialized @@ -1361,12 +1437,38 @@ static int get_next_submodule(struct child_process *cp, ce->name); } } + } + + if (spf->oid_fetch_tasks_nr) { + struct fetch_task *task = + spf->oid_fetch_tasks[spf->oid_fetch_tasks_nr - 1]; + struct strbuf submodule_prefix = STRBUF_INIT; + spf->oid_fetch_tasks_nr--; + + strbuf_addf(&submodule_prefix, "%s%s/", + spf->prefix, task->sub->path); + + child_process_init(cp); + prepare_submodule_repo_env_in_gitdir(&cp->env_array); + cp->git_cmd = 1; + cp->dir = task->repo->gitdir; + + argv_array_init(&cp->args); + argv_array_pushv(&cp->args, spf->args.argv); + argv_array_push(&cp->args, "on-demand"); + argv_array_push(&cp->args, "--submodule-prefix"); + argv_array_push(&cp->args, submodule_prefix.buf); + + /* NEEDSWORK: have get_default_remote from submodule--helper */ + argv_array_push(&cp->args, "origin"); + oid_array_for_each_unique(task->commits, + append_oid_to_argv, &cp->args); + + *task_cb = task; strbuf_release(&submodule_prefix); - if (ret) { - spf->count++; - return 1; - } + return 1; } + return 0; } @@ -1374,20 +1476,66 @@ static int fetch_start_failure(struct strbuf *err, void *cb, void *task_cb) { struct submodule_parallel_fetch *spf = cb; + struct fetch_task *task = task_cb; spf->result = 1; + fetch_task_release(task); return 0; } +static int commit_exists_in_sub(const struct object_id *oid, void *data) +{ + struct repository *subrepo = data; + + enum object_type type = oid_object_info(subrepo, oid, NULL); + + return type != OBJ_COMMIT; +} + static int fetch_finish(int retvalue, struct strbuf *err, void *cb, void *task_cb) { struct submodule_parallel_fetch *spf = cb; + struct fetch_task *task = task_cb; + + struct string_list_item *it; + struct oid_array *commits; if (retvalue) spf->result = 1; + if (!task || !task->sub) + BUG("callback cookie bogus"); + + /* Is this the second time we process this submodule? */ + if (task->commits) + return 0; + + it = string_list_lookup(&spf->changed_submodule_names, task->sub->name); + if (!it) + /* Could be an unchanged submodule, not contained in the list */ + goto out; + + commits = it->util; + oid_array_filter(commits, + commit_exists_in_sub, + task->repo); + + /* Are there commits we want, but do not exist? */ + if (commits->nr) { + task->commits = commits; + ALLOC_GROW(spf->oid_fetch_tasks, + spf->oid_fetch_tasks_nr + 1, + spf->oid_fetch_tasks_alloc); + spf->oid_fetch_tasks[spf->oid_fetch_tasks_nr] = task; + spf->oid_fetch_tasks_nr++; + return 0; + } + +out: + fetch_task_release(task); + return 0; } diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh index 6c2f9b2ba2..8a016272bc 100755 --- a/t/t5526-fetch-submodules.sh +++ b/t/t5526-fetch-submodules.sh @@ -600,4 +600,90 @@ test_expect_success "fetch new commits when submodule got renamed" ' test_cmp expect actual ' +test_expect_success "fetch new submodule commits on-demand outside standard refspec" ' + # add a second submodule and ensure it is around in downstream first + git clone submodule sub1 && + git submodule add ./sub1 && + git commit -m "adding a second submodule" && + git -C downstream pull && + git -C downstream submodule update --init --recursive && + + git checkout --detach && + + C=$(git -C submodule commit-tree -m "new change outside refs/heads" HEAD^{tree}) && + git -C submodule update-ref refs/changes/1 $C && + git update-index --cacheinfo 160000 $C submodule && + test_tick && + + D=$(git -C sub1 commit-tree -m "new change outside refs/heads" HEAD^{tree}) && + git -C sub1 update-ref refs/changes/2 $D && + git update-index --cacheinfo 160000 $D sub1 && + + git commit -m "updated submodules outside of refs/heads" && + E=$(git rev-parse HEAD) && + git update-ref refs/changes/3 $E && + ( + cd downstream && + git fetch --recurse-submodules origin refs/changes/3:refs/heads/my_branch && + git -C submodule cat-file -t $C && + git -C sub1 cat-file -t $D && + git checkout --recurse-submodules FETCH_HEAD + ) +' + +test_expect_success 'fetch new submodule commits on-demand in FETCH_HEAD' ' + # depends on the previous test for setup + + C=$(git -C submodule commit-tree -m "another change outside refs/heads" HEAD^{tree}) && + git -C submodule update-ref refs/changes/4 $C && + git update-index --cacheinfo 160000 $C submodule && + test_tick && + + D=$(git -C sub1 commit-tree -m "another change outside refs/heads" HEAD^{tree}) && + git -C sub1 update-ref refs/changes/5 $D && + git update-index --cacheinfo 160000 $D sub1 && + + git commit -m "updated submodules outside of refs/heads" && + E=$(git rev-parse HEAD) && + git update-ref refs/changes/6 $E && + ( + cd downstream && + git fetch --recurse-submodules origin refs/changes/6 && + git -C submodule cat-file -t $C && + git -C sub1 cat-file -t $D && + git checkout --recurse-submodules FETCH_HEAD + ) +' + +test_expect_success 'fetch new submodule commits on-demand without .gitmodules entry' ' + # depends on the previous test for setup + + git config -f .gitmodules --remove-section submodule.sub1 && + git add .gitmodules && + git commit -m "delete gitmodules file" && + git checkout -B master && + git -C downstream fetch && + git -C downstream checkout origin/master && + + C=$(git -C submodule commit-tree -m "yet another change outside refs/heads" HEAD^{tree}) && + git -C submodule update-ref refs/changes/7 $C && + git update-index --cacheinfo 160000 $C submodule && + test_tick && + + D=$(git -C sub1 commit-tree -m "yet another change outside refs/heads" HEAD^{tree}) && + git -C sub1 update-ref refs/changes/8 $D && + git update-index --cacheinfo 160000 $D sub1 && + + git commit -m "updated submodules outside of refs/heads" && + E=$(git rev-parse HEAD) && + git update-ref refs/changes/9 $E && + ( + cd downstream && + git fetch --recurse-submodules origin refs/changes/9 && + git -C submodule cat-file -t $C && + git -C sub1 cat-file -t $D && + git checkout --recurse-submodules FETCH_HEAD + ) +' + test_done