From patchwork Wed Oct 10 21:50:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10635353 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6F3D316B1 for ; Wed, 10 Oct 2018 21:50:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5F9A82ABCD for ; Wed, 10 Oct 2018 21:50:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 544082ACD6; Wed, 10 Oct 2018 21:50:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E9DA42AC0C for ; Wed, 10 Oct 2018 21:50:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726007AbeJKFOT (ORCPT ); Thu, 11 Oct 2018 01:14:19 -0400 Received: from mail-yw1-f74.google.com ([209.85.161.74]:35158 "EHLO mail-yw1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725856AbeJKFOT (ORCPT ); Thu, 11 Oct 2018 01:14:19 -0400 Received: by mail-yw1-f74.google.com with SMTP id i20-v6so3781789ywe.2 for ; Wed, 10 Oct 2018 14:50:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=1iXIDOW89tjebeBqF/Oz1JJUe+ARiDAxPsK7dMrUv6o=; b=tL2z01TC/pgOV4o9+X1qB7QmFTDEtVcV4Q1bXakLRLMOpHWiCVGRYdSa566sxxYe4u Rq0zsuDIILWExCuY7dWVbUdDlXE0uj5h0nheka3DRvGx++lXGcMOa+SBrS0BkycEaY30 CpiIYIySa/52oHg7GXvu7rDDCqCiAjoT4LddzmFhrLNShc7zpmp7NAk96eUxQX1tdGq6 HAkmQfPgfTFP15UCSD3LQmScbW596ZpPNNjNam6HwD/NWtB2dqL2hiog3q9a1AUSr0PV UKafO7fi3o8kjhL8uLaryyHUDWQ/RPtpOdQSWvkDlpJ6JratLE0O5Y1ohEPv+qfY1Drj BJFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=1iXIDOW89tjebeBqF/Oz1JJUe+ARiDAxPsK7dMrUv6o=; b=Y218qlzDpNXESY04+o9eD3FWf8tIzv/PQoWcOJJ3/OPQvylYsdtWuVUH+w1b41ZxMg E67jIe0ElWFvmHS2Uc6AZvQCEjxOEk41t6HieDxAUkMjqX9WKn+APePAVyr1y2IfZ6+l n28rcdOv/63YY3t/0ngwqyhIwJTNU6BcOMwzVZ8YEH3MrprHhzs3Gdvva90l/acyhAoP aW6VR6CQl7hcqV9nbNtT43Hzj8RZ5tcnDjZ6aWVS0/6vKUyZp0/+gvtuWGy79dhMr9rA HxYFhsbCHeXysZqrSJmcnWOjD0OJfBXzAhWjHaofWLJXY3dYzL4NHERKkDjtLh6DTpfW M8xA== X-Gm-Message-State: ABuFfohHXNHDi+9UmUsgn4ib/+BlFdCpvLzXmj6C7f8Dno72WC66imDT y6tBRY67f0K3nGWwK+jc7s4ddsxB0iRjRJzJx/wC5iRW3/FCAqHZA5Swk+kHlcJYkltsM3Jpo+/ gcewkZK9lJ6a59c2+1v3bpZykQGKzKvS9F1vTVn5wCONK/smJ7YxkAZjK41sz X-Google-Smtp-Source: ACcGV62Vg4lCM+hdWWy+HwLUCkgEouLhPsfg3wGnEj1wLUPWp5dzYEWKGXhFy26CLBzAMXHb2kR52Yar2Oac X-Received: by 2002:a25:1286:: with SMTP id 128-v6mr17493997ybs.30.1539208213800; Wed, 10 Oct 2018 14:50:13 -0700 (PDT) Date: Wed, 10 Oct 2018 14:50:00 -0700 In-Reply-To: <20181010215008.233529-1-sbeller@google.com> Message-Id: <20181010215008.233529-2-sbeller@google.com> Mime-Version: 1.0 References: <20181010215008.233529-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH v5 1/9] sha1-array: provide oid_array_filter From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Helped-by: Junio C Hamano Signed-off-by: Stefan Beller --- Documentation/technical/api-oid-array.txt | 5 +++++ sha1-array.c | 17 +++++++++++++++++ sha1-array.h | 3 +++ 3 files changed, 25 insertions(+) diff --git a/Documentation/technical/api-oid-array.txt b/Documentation/technical/api-oid-array.txt index 9febfb1d52..c97428c2c3 100644 --- a/Documentation/technical/api-oid-array.txt +++ b/Documentation/technical/api-oid-array.txt @@ -48,6 +48,11 @@ Functions is not sorted, this function has the side effect of sorting it. +`oid_array_filter`:: + Apply the callback function `want` to each entry in the array, + retaining only the entries for which the function returns true. + Preserve the order of the entries that are retained. + Examples -------- diff --git a/sha1-array.c b/sha1-array.c index b94e0ec0f5..d922e94e3f 100644 --- a/sha1-array.c +++ b/sha1-array.c @@ -77,3 +77,20 @@ int oid_array_for_each_unique(struct oid_array *array, } return 0; } + +void oid_array_filter(struct oid_array *array, + for_each_oid_fn want, + void *cb_data) +{ + unsigned nr = array->nr, src, dst; + struct object_id *oids = array->oid; + + for (src = dst = 0; src < nr; src++) { + if (want(&oids[src], cb_data)) { + if (src != dst) + oidcpy(&oids[dst], &oids[src]); + dst++; + } + } + array->nr = dst; +} diff --git a/sha1-array.h b/sha1-array.h index 232bf95017..55d016c4bf 100644 --- a/sha1-array.h +++ b/sha1-array.h @@ -22,5 +22,8 @@ int oid_array_for_each(struct oid_array *array, int oid_array_for_each_unique(struct oid_array *array, for_each_oid_fn fn, void *data); +void oid_array_filter(struct oid_array *array, + for_each_oid_fn want, + void *cbdata); #endif /* SHA1_ARRAY_H */ From patchwork Wed Oct 10 21:50:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10635355 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8A1D746E4 for ; Wed, 10 Oct 2018 21:50:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 78E272AC23 for ; Wed, 10 Oct 2018 21:50:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6D37D2AD88; Wed, 10 Oct 2018 21:50:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1BAAD2AC2E for ; Wed, 10 Oct 2018 21:50:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726016AbeJKFOV (ORCPT ); Thu, 11 Oct 2018 01:14:21 -0400 Received: from mail-io1-f74.google.com ([209.85.166.74]:48347 "EHLO mail-io1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725856AbeJKFOV (ORCPT ); Thu, 11 Oct 2018 01:14:21 -0400 Received: by mail-io1-f74.google.com with SMTP id v12-v6so5939271iob.15 for ; Wed, 10 Oct 2018 14:50:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Iaw44RTBOK0AvMmAeksONK9xCLHcVKCwlD5k4BRnW9k=; b=DO3vpJomrLSBRSao049486nz1m4+cwBrCBaW6ec8UH0+qswzfYfkUWMYpqvht1M95R emfkxyxEGpbnpz9wpdLi6Yn7jyi4nOcI/DebZMRo73EQWudDS1B3xq62Ca2OaTty8LMh AkkDeZcgt020qjYJnF5TmqpLgOpISakSXpWaS3TGgQjY9I86Lls9/QuSKrAf67JdqGt9 9pEJO2fytDmDeqp+hdQiGvMoZt6BLSH9v8mMu/Nu9fybsImaX/1ERXOpM0T/cy0fVh3W pCo37C0ehWvGQJQ1OTu0uh2pXYqJewWV8lln05cV747Pe3IM+TAF+qJ9xVOQAv7pHdT4 8ZPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Iaw44RTBOK0AvMmAeksONK9xCLHcVKCwlD5k4BRnW9k=; b=hkY38Sfl300zswniNugBWbKGIge03+nVDbj0YHrIGR8aUrpcuQD+H+4sJE18YiOlPr Bwj0ofUvCSVzQCcjDvXqbP3gXDbxB9UwvrCkzWMNomRd+wtPmUzlEnSDpeKReCxYpspE jA482ilAsEbdYP8oMS72T7pT15jF55inxDcCC+JaIUSXLevadBDiwYnJGkvO7PRNifJs Wxs4g2s+GqY98ioKiAoQThM8eEumUktxjYtgFOhz/UZUVIvlX9p57uGkbeMnMmP/gkac +41C6EPOczP08hnWlDT7TfbFZgfc/hM/UKhDdSCSILIsKGDOOosu98MhA8tGZC8zQNHw fQ5Q== X-Gm-Message-State: ABuFfogJRzF/Rxn5t5rxX+Nd8C6HavaD15f1qDuF0ah+da3iKJKo66Nr qFjTVBO0BbuGadibqVXDwmHf4xn5pmLqQn577zhG34WD5eq6b+R6ri+7zdKwdkDO+/MXuCLjwNC KoLLrkop/+LOP+lhCkGp0NUFbAmaH7Ttn2o3L+gdmF4H02aRs5iiYrRk5FUKy X-Google-Smtp-Source: ACcGV61zHHNRmYrtGRCtXnHugdNnx1R0DBfxMTwoV6snSNCEKDWb5zYVbnM+YJJE9JmL910PezdIZxckf3zQ X-Received: by 2002:a24:6cc2:: with SMTP id w185-v6mr13462438itb.5.1539208216022; Wed, 10 Oct 2018 14:50:16 -0700 (PDT) Date: Wed, 10 Oct 2018 14:50:01 -0700 In-Reply-To: <20181010215008.233529-1-sbeller@google.com> Message-Id: <20181010215008.233529-3-sbeller@google.com> Mime-Version: 1.0 References: <20181010215008.233529-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH v5 2/9] submodule.c: fix indentation From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The submodule subsystem is really bad at staying within 80 characters. Fix it while we are here. Signed-off-by: Stefan Beller --- submodule.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/submodule.c b/submodule.c index b53cb6e9c4..0de9e2800a 100644 --- a/submodule.c +++ b/submodule.c @@ -1244,7 +1244,8 @@ static int get_next_submodule(struct child_process *cp, if (!submodule) { const char *name = default_name_or_path(ce->name); if (name) { - default_submodule.path = default_submodule.name = name; + default_submodule.path = name; + default_submodule.name = name; submodule = &default_submodule; } } @@ -1254,8 +1255,10 @@ static int get_next_submodule(struct child_process *cp, default: case RECURSE_SUBMODULES_DEFAULT: case RECURSE_SUBMODULES_ON_DEMAND: - if (!submodule || !unsorted_string_list_lookup(&changed_submodule_names, - submodule->name)) + if (!submodule || + !unsorted_string_list_lookup( + &changed_submodule_names, + submodule->name)) continue; default_argv = "on-demand"; break; From patchwork Wed Oct 10 21:50:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10635357 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A8E4146E4 for ; Wed, 10 Oct 2018 21:50:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 908D92AC2E for ; Wed, 10 Oct 2018 21:50:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 84E352AD11; Wed, 10 Oct 2018 21:50:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 39DC22AD17 for ; Wed, 10 Oct 2018 21:50:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726035AbeJKFOY (ORCPT ); Thu, 11 Oct 2018 01:14:24 -0400 Received: from mail-qt1-f202.google.com ([209.85.160.202]:35769 "EHLO mail-qt1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725856AbeJKFOX (ORCPT ); Thu, 11 Oct 2018 01:14:23 -0400 Received: by mail-qt1-f202.google.com with SMTP id s56-v6so6741590qtk.2 for ; Wed, 10 Oct 2018 14:50:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=hqZc7ihSQbUNrkq0Y9fju6ghoKYXJorGJuvYpPGIIwo=; b=dbSP+nNpT7BjYhYoIi8x4Blqkg9SRuekQg6RwUVC+t+Z5dNMcihxE2FlvRGviyK2hm RFOHdanhN+NHKWb6M6eXzcX04PYpKBYdtTsbNDU5p8kj4LBYNk6BqOvKZknC+BMBqBKw r32M+Lvqc4ycjh4EdrIbFqnrW2ApFAu2rv/B3PFgOAGF81Qm/t7NeVvIrUyNoLLIruzE s/QWS26ieWCdO6ypV2XHEWPULUTsFThAdVZflig5mOk16ZhCLbbI54jFMjk6h+qjc875 FcYt5nz/lZzc/TyfvbPzYQnNurLvgWMsfPVWoQgJ/LJsjTpQ0JPOD99dcYB3mAAYXQeK cY2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=hqZc7ihSQbUNrkq0Y9fju6ghoKYXJorGJuvYpPGIIwo=; b=VFgoOr3VINNNUewMrQiWlDx3oLbc9Vu7PCaaleOxTAoRilwyZeABRmvyUB/Axm+3O6 PvJpuZ4EB3I1j4ABdrleXbPWOsNyMuZDXIT8yDjucrb7AUQHMZPEwDkmaoKYGxE6JgTH dbdyb53+pV6EPV0nD3lmXiZGa3dbCDFPmKhE3C4F1h4dGzC2zFGyG2YO8J3yCTXXSilm m0lRLBQVXpgCkqA5sOVrvDPyVIS4kFoxaDOHuUKuDDu0hyzkfvbMXqBnhrsl0eAoJZ5L OQ+KpcrxrLDl4tGI+rbIucyr+3WwQ06ntQznJ3d/tTD84loa0RECeQqyKpspOkgdIrYO RWAQ== X-Gm-Message-State: ABuFfog1J1kxxK7Z6IjJH4qqij2IXQqObOxf6ATQ0C7RdR21wkdC3gEw LEaM0KvYYB6nu/YwLhzQuEYdSvG6yRXmOsWDsqu9CqIjUsMbhacPNNCpcbEzYWvUv2h51Cejr1M EYF1kXJG7iykgLI5zOgN/eybm7bG1BvMv2c94QeTAbJlQCjXscDCvf+RBVJoQ X-Google-Smtp-Source: ACcGV63p78eT3vGjMbNJT1RQCFppY3WZsxH6jwb9wRPQYRV7qHV14b9HvRuoXQdcSEgIft6Ysc/safAwRXy2 X-Received: by 2002:ac8:4108:: with SMTP id q8-v6mr634753qtl.17.1539208218171; Wed, 10 Oct 2018 14:50:18 -0700 (PDT) Date: Wed, 10 Oct 2018 14:50:02 -0700 In-Reply-To: <20181010215008.233529-1-sbeller@google.com> Message-Id: <20181010215008.233529-4-sbeller@google.com> Mime-Version: 1.0 References: <20181010215008.233529-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH v5 3/9] submodule.c: sort changed_submodule_names before searching it From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We can string_list_insert() to maintain sorted-ness of the list as we find new items, or we can string_list_append() to build an unsorted list and sort it at the end just once. As we do not rely on the sortedness while building the list, we pick the "append and sort at the end" as it has better worst case execution times. Signed-off-by: Stefan Beller --- submodule.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/submodule.c b/submodule.c index 0de9e2800a..22c64bd855 100644 --- a/submodule.c +++ b/submodule.c @@ -1256,7 +1256,7 @@ static int get_next_submodule(struct child_process *cp, case RECURSE_SUBMODULES_DEFAULT: case RECURSE_SUBMODULES_ON_DEMAND: if (!submodule || - !unsorted_string_list_lookup( + !string_list_lookup( &changed_submodule_names, submodule->name)) continue; @@ -1350,6 +1350,7 @@ int fetch_populated_submodules(struct repository *r, /* default value, "--submodule-prefix" and its value are added later */ calculate_changed_submodule_paths(); + string_list_sort(&changed_submodule_names); run_processes_parallel(max_parallel_jobs, get_next_submodule, fetch_start_failure, From patchwork Wed Oct 10 21:50:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10635359 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2129016B1 for ; Wed, 10 Oct 2018 21:50:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 105A92AC80 for ; Wed, 10 Oct 2018 21:50:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 04B042AD17; Wed, 10 Oct 2018 21:50:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 967432ACF8 for ; Wed, 10 Oct 2018 21:50:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726046AbeJKFO2 (ORCPT ); Thu, 11 Oct 2018 01:14:28 -0400 Received: from mail-qk1-f201.google.com ([209.85.222.201]:51823 "EHLO mail-qk1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725856AbeJKFO2 (ORCPT ); Thu, 11 Oct 2018 01:14:28 -0400 Received: by mail-qk1-f201.google.com with SMTP id x75-v6so6242599qka.18 for ; Wed, 10 Oct 2018 14:50:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=eZ+1/Zj9J2LmeHnKPnCj7udopPQcSJzE0rZTARKQ0OA=; b=oPjd27PvDHSHZ2R3YyN31C/Gceq5H+r4j/bR1ag5SGOJ76ybP8o+ftElhFnchySrVz qyQRjlJPEK544jGQhQIA5ZpdNlzqrfjmWpKdyMFGj0xk7VVHVMmeC/bNxUlGhUAahCrq ZBB6IwcT1nIaA5wnEpV1Drv8VSnsScHMVFvvEeDpXhF3wz/oT26/2wF++/h8maiNHUVY gPzmrlWlOlyjWLkiizOyNIrFS/aFkPYflN9HRYsSW763WchBITBd2B4IMXmPp+FCkHYC /DJOSgOzWTgrdvTliyes9zdLtAVsMRose/AbwFzWPpNhiiyZpTlWg3iK1/qwS5r8TEgU 4GPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=eZ+1/Zj9J2LmeHnKPnCj7udopPQcSJzE0rZTARKQ0OA=; b=fP1EeAyVQvGx0iKP3Me3c4Y99uT8PCRxqhwo8LyJtpb03ZFEcUGCtFWdCbXAQHdKSV hxCWQKq3mlBxJh/Z0lvhVcm+KYgd2vQU/JgxObyyPiHb7eOWKz1b4ZpEavyeqie9TiE4 KBylEKtwGFcVZRZIf85J4Ip4TynWdk8rACYNlbxylRCxV3gNq7ffU1v5YbPZf4gtFJej cIUdQo7PcfxyfiGAbne3t0evvmK4hLYbCpjcotfgKGtvqkkB0gKDzdAIHgit9acaxlzi gE4H3ZccRlgpSWTYZAO0bMSRWRFjWwh6oaTwN13mYTjsSsTv3KpX5VGSpf68bp+Tg7MM bbaQ== X-Gm-Message-State: ABuFfoiOa615lhVFbnAfG+BDR2ph53tvvbnA5jrCGGYrFkgEC6Wq4JNl lWLY+a5ke2ZgwkWdunPecYJYdsF0uiN6cgYmnM9J2fpuXg1l1qhFVrvsyaUtH3/o90889KJDkDc ggUcRfJjR8ie/pCijTsSEGbxhIAeUA+we+bxIhYk3Ud7lyeXB1gTzPhUUe7oR X-Google-Smtp-Source: ACcGV62hwK5LM6WEJROyvN3VBZ0mmiPtIHcK8k6TpLSc+4t8EY+MS4phJAMS43N4PzeXoWDaGA+axZhQBIlX X-Received: by 2002:a37:6687:: with SMTP id a129-v6mr1800624qkc.7.1539208221960; Wed, 10 Oct 2018 14:50:21 -0700 (PDT) Date: Wed, 10 Oct 2018 14:50:03 -0700 In-Reply-To: <20181010215008.233529-1-sbeller@google.com> Message-Id: <20181010215008.233529-5-sbeller@google.com> Mime-Version: 1.0 References: <20181010215008.233529-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH v5 4/9] submodule.c: move global changed_submodule_names into fetch submodule struct From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The `changed_submodule_names` are only used for fetching, so let's make it part of the struct that is passed around for fetching submodules. Signed-off-by: Stefan Beller --- submodule.c | 42 +++++++++++++++++++++++------------------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/submodule.c b/submodule.c index 22c64bd855..17103379ba 100644 --- a/submodule.c +++ b/submodule.c @@ -25,7 +25,7 @@ #include "commit-reach.h" static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF; -static struct string_list changed_submodule_names = STRING_LIST_INIT_DUP; + static int initialized_fetch_ref_tips; static struct oid_array ref_tips_before_fetch; static struct oid_array ref_tips_after_fetch; @@ -1110,7 +1110,22 @@ void check_for_new_submodule_commits(struct object_id *oid) oid_array_append(&ref_tips_after_fetch, oid); } -static void calculate_changed_submodule_paths(void) +struct submodule_parallel_fetch { + int count; + struct argv_array args; + struct repository *r; + const char *prefix; + int command_line_option; + int default_option; + int quiet; + int result; + + struct string_list changed_submodule_names; +}; +#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0, STRING_LIST_INIT_DUP } + +static void calculate_changed_submodule_paths( + struct submodule_parallel_fetch *spf) { struct argv_array argv = ARGV_ARRAY_INIT; struct string_list changed_submodules = STRING_LIST_INIT_DUP; @@ -1148,7 +1163,8 @@ static void calculate_changed_submodule_paths(void) continue; if (!submodule_has_commits(path, commits)) - string_list_append(&changed_submodule_names, name->string); + string_list_append(&spf->changed_submodule_names, + name->string); } free_submodules_oids(&changed_submodules); @@ -1185,18 +1201,6 @@ int submodule_touches_in_range(struct object_id *excl_oid, return ret; } -struct submodule_parallel_fetch { - int count; - struct argv_array args; - struct repository *r; - const char *prefix; - int command_line_option; - int default_option; - int quiet; - int result; -}; -#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0} - static int get_fetch_recurse_config(const struct submodule *submodule, struct submodule_parallel_fetch *spf) { @@ -1257,7 +1261,7 @@ static int get_next_submodule(struct child_process *cp, case RECURSE_SUBMODULES_ON_DEMAND: if (!submodule || !string_list_lookup( - &changed_submodule_names, + &spf->changed_submodule_names, submodule->name)) continue; default_argv = "on-demand"; @@ -1349,8 +1353,8 @@ int fetch_populated_submodules(struct repository *r, argv_array_push(&spf.args, "--recurse-submodules-default"); /* default value, "--submodule-prefix" and its value are added later */ - calculate_changed_submodule_paths(); - string_list_sort(&changed_submodule_names); + calculate_changed_submodule_paths(&spf); + string_list_sort(&spf.changed_submodule_names); run_processes_parallel(max_parallel_jobs, get_next_submodule, fetch_start_failure, @@ -1359,7 +1363,7 @@ int fetch_populated_submodules(struct repository *r, argv_array_clear(&spf.args); out: - string_list_clear(&changed_submodule_names, 1); + string_list_clear(&spf.changed_submodule_names, 1); return spf.result; } From patchwork Wed Oct 10 21:50:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10635361 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ABE3E46E4 for ; Wed, 10 Oct 2018 21:50:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 979E82AC4E for ; Wed, 10 Oct 2018 21:50:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8C4662AE0F; Wed, 10 Oct 2018 21:50:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 31A762AC4E for ; Wed, 10 Oct 2018 21:50:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726050AbeJKFOh (ORCPT ); Thu, 11 Oct 2018 01:14:37 -0400 Received: from mail-vs1-f73.google.com ([209.85.217.73]:48606 "EHLO mail-vs1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725856AbeJKFOg (ORCPT ); Thu, 11 Oct 2018 01:14:36 -0400 Received: by mail-vs1-f73.google.com with SMTP id t85so2625647vsc.15 for ; Wed, 10 Oct 2018 14:50:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=nhKbX5UMMokr0xuH5bisVy2OvQYMXSMU76Ag85Cxuxg=; b=tph//wk2j/vYllqeBndH5mVRxEiETThkC6mVnr262lTC2ea1mNDkZWw4b+oI8TVa9f 9veYESem9XR7C1DvwJVIczzirPl6b/JviOzLfnaxalCYOoPXnmFCE4nVWkWa0PlIyXNa Z523wMRFN98uk+ZqDNZMCfFrEngRtavZvGqb8U1iFiKexAUgpd602eFERQ73Gy7zCzyq qez9fGHsha285kn/CJ9m4fowDqSLMh6SAW6YqODyrhZotBUwmqOL2VQWMSYqYbwZMFFQ VU1gnZ+he9fLVlpJyAs1J3KcNpuXEmE4dxhtTbTe2/my8lxsHaGQanbB1ohn17kKmdzd N3fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=nhKbX5UMMokr0xuH5bisVy2OvQYMXSMU76Ag85Cxuxg=; b=j22vrfI4nPyxWGm57OHOAgTmXXDlSCZE+9JU5+55PAj398eWmiFOGTAUgflbld9AZ0 e7zJdVNxs/ZeSzD1Hgr891v6XHHBe+Dfnwd4eWWUDikyuhg1WncvSA+aXSpK1mmXyGaj RLwQen6pQw0BmwWJcQSb98r122NmrVBduowZG+uohn1jGr6iKrmXVj10p1hAffLVWRye 2I7cT4hV0QMqI/mLdHuJEAhsz2WtiFxWxvaj7fK1Y9o45UWxE+ImonTCAkL5yaRultTj AuVrTbAk/bxoR+ksCiZeVFN8eCT70D3kygDtvQh0dhYSqRbToPw2vnLfJq9e15Iz5YUy 9k/Q== X-Gm-Message-State: ABuFfogQTbvAx63/lylPfMyCTiagsxi1b5hwJzdcob+FdBEYE8AftCPc InyW4pJrlpfss74uBuoPdHyskurQKVvmtwgm2SWd7X2I7hZ5ages7vm0OggyaM/vtdK0IqNV8vC RmAbcq8guAvrDl62YpR1349w4sGbXtCb9BV9DoM3sC78piyZDqjoSTbQW/GPL X-Google-Smtp-Source: ACcGV60hVnJjIZ3pLNomCSIgARnwHNbqJzzs/NK9eHzP5aLrBHSVZrKgUbBhraIJf7sff+esUGgWUHu2AiSx X-Received: by 2002:a67:3644:: with SMTP id d65mr15658164vsa.35.1539208230464; Wed, 10 Oct 2018 14:50:30 -0700 (PDT) Date: Wed, 10 Oct 2018 14:50:04 -0700 In-Reply-To: <20181010215008.233529-1-sbeller@google.com> Message-Id: <20181010215008.233529-6-sbeller@google.com> Mime-Version: 1.0 References: <20181010215008.233529-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH v5 5/9] submodule.c: do not copy around submodule list From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP 'calculate_changed_submodule_paths' uses a local list to compute the changed submodules, and then produces the result by copying appropriate items into the result list. Instead use the result list directly and prune items afterwards using string_list_remove_empty_items. By doing so we'll have access to the util pointer for longer that contains the commits that we need to fetch, which will be useful in a later patch. Signed-off-by: Stefan Beller --- submodule.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/submodule.c b/submodule.c index 17103379ba..dd478ed70b 100644 --- a/submodule.c +++ b/submodule.c @@ -1128,8 +1128,7 @@ static void calculate_changed_submodule_paths( struct submodule_parallel_fetch *spf) { struct argv_array argv = ARGV_ARRAY_INIT; - struct string_list changed_submodules = STRING_LIST_INIT_DUP; - const struct string_list_item *name; + struct string_list_item *name; /* No need to check if there are no submodules configured */ if (!submodule_from_path(the_repository, NULL, NULL)) @@ -1146,9 +1145,9 @@ static void calculate_changed_submodule_paths( * Collect all submodules (whether checked out or not) for which new * commits have been recorded upstream in "changed_submodule_names". */ - collect_changed_submodules(&changed_submodules, &argv); + collect_changed_submodules(&spf->changed_submodule_names, &argv); - for_each_string_list_item(name, &changed_submodules) { + for_each_string_list_item(name, &spf->changed_submodule_names) { struct oid_array *commits = name->util; const struct submodule *submodule; const char *path = NULL; @@ -1162,12 +1161,14 @@ static void calculate_changed_submodule_paths( if (!path) continue; - if (!submodule_has_commits(path, commits)) - string_list_append(&spf->changed_submodule_names, - name->string); + if (submodule_has_commits(path, commits)) { + oid_array_clear(commits); + *name->string = '\0'; + } } - free_submodules_oids(&changed_submodules); + string_list_remove_empty_items(&spf->changed_submodule_names, 1); + argv_array_clear(&argv); oid_array_clear(&ref_tips_before_fetch); oid_array_clear(&ref_tips_after_fetch); @@ -1363,7 +1364,7 @@ int fetch_populated_submodules(struct repository *r, argv_array_clear(&spf.args); out: - string_list_clear(&spf.changed_submodule_names, 1); + free_submodules_oids(&spf.changed_submodule_names); return spf.result; } From patchwork Wed Oct 10 21:50:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10635363 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C71AA16B1 for ; Wed, 10 Oct 2018 21:50:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B23DF2AB51 for ; Wed, 10 Oct 2018 21:50:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A65062AE54; Wed, 10 Oct 2018 21:50:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F14702ADE2 for ; Wed, 10 Oct 2018 21:50:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725988AbeJKFOp (ORCPT ); Thu, 11 Oct 2018 01:14:45 -0400 Received: from mail-qt1-f201.google.com ([209.85.160.201]:45570 "EHLO mail-qt1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725856AbeJKFOp (ORCPT ); Thu, 11 Oct 2018 01:14:45 -0400 Received: by mail-qt1-f201.google.com with SMTP id l6-v6so6588641qtc.12 for ; Wed, 10 Oct 2018 14:50:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=z1QtzaQ4i/gEwNO2APR2R4QOqvRnv2ZwzWSyBwFXHUY=; b=EWqBtG+96y2iWO95vl+OMImgaPeWXFbYF7S0iXH4h6A6bOfUNkYD7rU+CkvLBK55aw DuKYeS6//g0wJJqN1qN7aKKdgN8wCSBBG83mVwga6tPXij/k43FKs98lbsIoQfzzbtHL NBMOy/Zhr2ygCT2WH92Moslwf/FOTYN/aDa9SDbFEGlUupuVilecYAkYRVgVlex38/x6 2/iKv26fJHAJLiDCexM4wRts8JP3Gkyo+AaeB3QqO9TkW9rn8KQnZ/fJ6v93D/xFGxoJ sPcv2U2K70XY9GYEQthXlMPFSolWXEtzLDNNgAYHTb35hTWPeKkWCScJg+bnHC9b2AvA zkug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=z1QtzaQ4i/gEwNO2APR2R4QOqvRnv2ZwzWSyBwFXHUY=; b=RoEIhbvZljRQ5X1cZ1EDMaEqkR55hbaAt3Ris5eMx0Bnz+KboC1JE+FJ5IN2078yam OeyDiIk+I9IVsKYEK9sXw9FDxkqROcr8JODuhW0EVOqNOsJCWb/Z5jmzSlK+L0CIFmka jG0W8Og9P2m5Y/c2izUVHBoz2CXTXDQf0U0/WzATz7Y5c4PAsfyZyUlg+79JsNi8sF5A 6Mw7dEGOTAG8TzEJ3u1WlarfYygx9cwlyrUCc5Xn1VxDOoSnl8KB3aECvubIrwUofcE9 zd88zUKxPoGqoyQN8u72bIFaYF9NfvvGU3fTL9F0fjxc1RVt+rokQacrqKWE1pzmJ6V0 jAcg== X-Gm-Message-State: ABuFfogcoKNbWtPL1deJzGhmwmVJiXaaHzkL6f4YcS5yh3UmTGL1HVKY oNaDeqadaQHOrayil5boT3Q+GcxBd1nqnAndoEMVivh0ntWUnPTyEuBsDREdsk2y+cVPqpqA/Wz 9KptuX7TPZwfEtojJnBdRISfyB2kF4txKLAlKYvIsxWMZFmrUBqMPEWJ4Gf/8 X-Google-Smtp-Source: ACcGV60NtSa20KGctTRokS9Atu3CWQOGhsJtevITIyHjtEY3urduMw6xleTtMR20/8kwqzG2iUmPAvbR4FMc X-Received: by 2002:a0c:8870:: with SMTP id 45mr1939223qvm.4.1539208239139; Wed, 10 Oct 2018 14:50:39 -0700 (PDT) Date: Wed, 10 Oct 2018 14:50:05 -0700 In-Reply-To: <20181010215008.233529-1-sbeller@google.com> Message-Id: <20181010215008.233529-7-sbeller@google.com> Mime-Version: 1.0 References: <20181010215008.233529-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH v5 6/9] repository: repo_submodule_init to take a submodule struct From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When constructing a struct repository for a submodule for some revision of the superproject where the submodule is not contained in the index, it may not be present in the working tree currently either. In that siutation giving a 'path' argument is not useful. Upgrade the repo_submodule_init function to take a struct submodule instead. While we are at it, overhaul the repo_submodule_init function by renaming the submodule repository struct, which is to be initialized, to a name that is not confused with the struct submodule as easily. Also move its documentation into the header file. Signed-off-by: Stefan Beller --- builtin/grep.c | 17 ++++++++++------- builtin/ls-files.c | 12 +++++++----- builtin/submodule--helper.c | 2 +- repository.c | 27 ++++++++++----------------- repository.h | 11 +++++++++-- 5 files changed, 37 insertions(+), 32 deletions(-) diff --git a/builtin/grep.c b/builtin/grep.c index 601f801158..81c53c862b 100644 --- a/builtin/grep.c +++ b/builtin/grep.c @@ -418,16 +418,19 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, const struct object_id *oid, const char *filename, const char *path) { - struct repository submodule; + struct repository subrepo; + const struct submodule *sub = submodule_from_path(superproject, + &null_oid, path); + int hit; if (!is_submodule_active(superproject, path)) return 0; - if (repo_submodule_init(&submodule, superproject, path)) + if (repo_submodule_init(&subrepo, superproject, sub)) return 0; - repo_read_gitmodules(&submodule); + repo_read_gitmodules(&subrepo); /* * NEEDSWORK: This adds the submodule's object directory to the list of @@ -440,7 +443,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, * object. */ grep_read_lock(); - add_to_alternates_memory(submodule.objects->objectdir); + add_to_alternates_memory(subrepo.objects->objectdir); grep_read_unlock(); if (oid) { @@ -465,14 +468,14 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, init_tree_desc(&tree, data, size); hit = grep_tree(opt, pathspec, &tree, &base, base.len, - object->type == OBJ_COMMIT, &submodule); + object->type == OBJ_COMMIT, &subrepo); strbuf_release(&base); free(data); } else { - hit = grep_cache(opt, &submodule, pathspec, 1); + hit = grep_cache(opt, &subrepo, pathspec, 1); } - repo_clear(&submodule); + repo_clear(&subrepo); return hit; } diff --git a/builtin/ls-files.c b/builtin/ls-files.c index 7f9919a362..4d1649c1b3 100644 --- a/builtin/ls-files.c +++ b/builtin/ls-files.c @@ -206,17 +206,19 @@ static void show_files(struct repository *repo, struct dir_struct *dir); static void show_submodule(struct repository *superproject, struct dir_struct *dir, const char *path) { - struct repository submodule; + struct repository subrepo; + const struct submodule *sub = submodule_from_path(superproject, + &null_oid, path); - if (repo_submodule_init(&submodule, superproject, path)) + if (repo_submodule_init(&subrepo, superproject, sub)) return; - if (repo_read_index(&submodule) < 0) + if (repo_read_index(&subrepo) < 0) die("index file corrupt"); - show_files(&submodule, dir); + show_files(&subrepo, dir); - repo_clear(&submodule); + repo_clear(&subrepo); } static void show_ce(struct repository *repo, struct dir_struct *dir, diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c index 247881189f..8214e77688 100644 --- a/builtin/submodule--helper.c +++ b/builtin/submodule--helper.c @@ -2038,7 +2038,7 @@ static int ensure_core_worktree(int argc, const char **argv, const char *prefix) if (!sub) BUG("We could get the submodule handle before?"); - if (repo_submodule_init(&subrepo, the_repository, path)) + if (repo_submodule_init(&subrepo, the_repository, sub)) die(_("could not get a repository handle for submodule '%s'"), path); if (!repo_config_get_string(&subrepo, "core.worktree", &cw)) { diff --git a/repository.c b/repository.c index 5dd1486718..aabe64ee5d 100644 --- a/repository.c +++ b/repository.c @@ -166,30 +166,23 @@ int repo_init(struct repository *repo, return -1; } -/* - * Initialize 'submodule' as the submodule given by 'path' in parent repository - * 'superproject'. - * Return 0 upon success and a non-zero value upon failure. - */ -int repo_submodule_init(struct repository *submodule, +int repo_submodule_init(struct repository *subrepo, struct repository *superproject, - const char *path) + const struct submodule *sub) { - const struct submodule *sub; struct strbuf gitdir = STRBUF_INIT; struct strbuf worktree = STRBUF_INIT; int ret = 0; - sub = submodule_from_path(superproject, &null_oid, path); if (!sub) { ret = -1; goto out; } - strbuf_repo_worktree_path(&gitdir, superproject, "%s/.git", path); - strbuf_repo_worktree_path(&worktree, superproject, "%s", path); + strbuf_repo_worktree_path(&gitdir, superproject, "%s/.git", sub->path); + strbuf_repo_worktree_path(&worktree, superproject, "%s", sub->path); - if (repo_init(submodule, gitdir.buf, worktree.buf)) { + if (repo_init(subrepo, gitdir.buf, worktree.buf)) { /* * If initilization fails then it may be due to the submodule * not being populated in the superproject's worktree. Instead @@ -201,16 +194,16 @@ int repo_submodule_init(struct repository *submodule, strbuf_repo_git_path(&gitdir, superproject, "modules/%s", sub->name); - if (repo_init(submodule, gitdir.buf, NULL)) { + if (repo_init(subrepo, gitdir.buf, NULL)) { ret = -1; goto out; } } - submodule->submodule_prefix = xstrfmt("%s%s/", - superproject->submodule_prefix ? - superproject->submodule_prefix : - "", path); + subrepo->submodule_prefix = xstrfmt("%s%s/", + superproject->submodule_prefix ? + superproject->submodule_prefix : + "", sub->path); out: strbuf_release(&gitdir); diff --git a/repository.h b/repository.h index 9f16c42c1e..a9c7a5baa5 100644 --- a/repository.h +++ b/repository.h @@ -116,9 +116,16 @@ void repo_set_worktree(struct repository *repo, const char *path); void repo_set_hash_algo(struct repository *repo, int algo); void initialize_the_repository(void); int repo_init(struct repository *r, const char *gitdir, const char *worktree); -int repo_submodule_init(struct repository *submodule, + +/* + * Initialize the repository 'subrepo' as the submodule given by the + * struct submodule 'sub' in parent repository 'superproject'. + * Return 0 upon success and a non-zero value upon failure. + */ +struct submodule; +int repo_submodule_init(struct repository *subrepo, struct repository *superproject, - const char *path); + const struct submodule *sub); void repo_clear(struct repository *repo); /* From patchwork Wed Oct 10 21:50:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10635365 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8E63916B1 for ; Wed, 10 Oct 2018 21:50:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7BB712AE62 for ; Wed, 10 Oct 2018 21:50:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7A01B2AE76; Wed, 10 Oct 2018 21:50:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E3B482AE68 for ; Wed, 10 Oct 2018 21:50:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725939AbeJKFOx (ORCPT ); Thu, 11 Oct 2018 01:14:53 -0400 Received: from mail-qt1-f202.google.com ([209.85.160.202]:55028 "EHLO mail-qt1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725856AbeJKFOx (ORCPT ); Thu, 11 Oct 2018 01:14:53 -0400 Received: by mail-qt1-f202.google.com with SMTP id i64-v6so6606152qtb.21 for ; Wed, 10 Oct 2018 14:50:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=5j5hT2IBgig73JxDXYFZtw/2y/EKtMsQ1QXXNgx+0qg=; b=RBatzevfjUiZSYPfXjmotUSJgw9M0qBRnuM+1vrX2VgxuXRDqCwtNgCCczBvyoNbWe /vMcoxbnR8VNZ8oPsi1oRdimImVux2X/ZDoMalNveOrRmKbeGwz+4XuS4DmGjAz79B29 j7Da/YB7O+x+TNoPusAkLWPzUs8yow4skJLr7W/SP7LJGe9Pi0ZQ8Twz3CESlXKhDriH qu44mhcaZ3yeAmXfLhFP6nx6ePYBIAdfr5PBn6WGKNeyBRGHLW37FpaMtsf/2Cx+eGfM VnIOcn6mUoHtgsX7Y9v1vmLxTUxAPgEiU+kNNKJzCbXqvFCevxbaoGhSNCDP651TOlCt Zb5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5j5hT2IBgig73JxDXYFZtw/2y/EKtMsQ1QXXNgx+0qg=; b=bIZFupZ1fCa0uISNMjWFkdLkjhrdAlwzSr29kTNnvzrqMp/Uo8QW3X9sSRSSnlU7Md bzoto0XrLNwmFK++PcZIiDeKNrOnkdVVdZVFd8JWmYCv7XcEMblJCgOLJC/Trv0zQSYi yHjakIZFB5OI3hSsBZ7oCFPiEZAfnKFz9PYse6dPNT0xy4sjjdLsNOxZNv3spt3y/r3n QUwmmc/oQlvA3O7Npt/loKse7Nlsph27BvLEsulhmrpRJQFU3He7RYJCKnPU9N2qN8qG rI1R11RFMDkg40c1mwqud246nYMHl+UYTeK7LNhssT8Nbwz3r2mr+h8RDJ+JvV+EW8Jn aZpw== X-Gm-Message-State: ABuFfojETYwXK8PGEVGQxsmsDqwcm72zClc265ojc3zZvcbYOo1h3vlP N5QS7PFTmYIZbn2MEEpNcPkuk5cMpiygpK5F/Scg3rIY+5ycoHqFhGdYifStZgJAS5onqs55kHO dqERbREUIkp1rNANiokeGcLuVdczewca2LLXrHzN73nbIRjot7RTwG8buiRwA X-Google-Smtp-Source: ACcGV61seMsOtim08TMHfIqVQgQtkSQpiZafxe+Qpi9+rpW+a4EkyGfoQZWzMRwqz6W79CqX5iPZ2Wi1SdEq X-Received: by 2002:ac8:2397:: with SMTP id q23-v6mr1895001qtq.55.1539208247465; Wed, 10 Oct 2018 14:50:47 -0700 (PDT) Date: Wed, 10 Oct 2018 14:50:06 -0700 In-Reply-To: <20181010215008.233529-1-sbeller@google.com> Message-Id: <20181010215008.233529-8-sbeller@google.com> Mime-Version: 1.0 References: <20181010215008.233529-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH v5 7/9] submodule: fetch in submodules git directory instead of in worktree From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch started as a refactoring to make 'get_next_submodule' more readable, but upon doing so, I realized that "git fetch" of the submodule actually doesn't need to be run in the submodules worktree. So let's run it in its git dir instead. That should pave the way towards fetching submodules that are currently not checked out. This patch leaks the cp->dir in get_next_submodule, as any further callback in run_processes_parallel doesn't have access to the child process any more. In an early iteration of this patch, the function get_submodule_repo_for directly returned the string containing the git directory, which would be a better design choice for this patch. However the next patch both fixes the memory leak of cp->dir and also has a use case for using the full repository handle of the submodule, so it makes sense to introduce the get_submodule_repo_for here already. Signed-off-by: Stefan Beller --- submodule.c | 51 +++++++++++++++++++++++++++---------- t/t5526-fetch-submodules.sh | 7 ++++- 2 files changed, 44 insertions(+), 14 deletions(-) diff --git a/submodule.c b/submodule.c index dd478ed70b..3f791f2277 100644 --- a/submodule.c +++ b/submodule.c @@ -481,6 +481,12 @@ void prepare_submodule_repo_env(struct argv_array *out) DEFAULT_GIT_DIR_ENVIRONMENT); } +static void prepare_submodule_repo_env_in_gitdir(struct argv_array *out) +{ + prepare_submodule_repo_env_no_git_dir(out); + argv_array_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT); +} + /* Helper function to display the submodule header line prior to the full * summary output. If it can locate the submodule objects directory it will * attempt to lookup both the left and right commits and put them into the @@ -1227,6 +1233,29 @@ static int get_fetch_recurse_config(const struct submodule *submodule, return spf->default_option; } +static struct repository *get_submodule_repo_for(struct repository *r, + const struct submodule *sub) +{ + struct repository *ret = xmalloc(sizeof(*ret)); + + if (repo_submodule_init(ret, r, sub)) { + /* + * No entry in .gitmodules? Technically not a submodule, + * but historically we supported repositories that happen to be + * in-place where a gitlink is. Keep supporting them. + */ + struct strbuf gitdir = STRBUF_INIT; + strbuf_repo_worktree_path(&gitdir, r, "%s/.git", sub->path); + if (repo_init(ret, gitdir.buf, NULL)) { + strbuf_release(&gitdir); + return NULL; + } + strbuf_release(&gitdir); + } + + return ret; +} + static int get_next_submodule(struct child_process *cp, struct strbuf *err, void *data, void **task_cb) { @@ -1234,12 +1263,11 @@ static int get_next_submodule(struct child_process *cp, struct submodule_parallel_fetch *spf = data; for (; spf->count < spf->r->index->cache_nr; spf->count++) { - struct strbuf submodule_path = STRBUF_INIT; - struct strbuf submodule_git_dir = STRBUF_INIT; struct strbuf submodule_prefix = STRBUF_INIT; const struct cache_entry *ce = spf->r->index->cache[spf->count]; - const char *git_dir, *default_argv; + const char *default_argv; const struct submodule *submodule; + struct repository *repo; struct submodule default_submodule = SUBMODULE_INIT; if (!S_ISGITLINK(ce->ce_mode)) @@ -1274,16 +1302,12 @@ static int get_next_submodule(struct child_process *cp, continue; } - strbuf_repo_worktree_path(&submodule_path, spf->r, "%s", ce->name); - strbuf_addf(&submodule_git_dir, "%s/.git", submodule_path.buf); strbuf_addf(&submodule_prefix, "%s%s/", spf->prefix, ce->name); - git_dir = read_gitfile(submodule_git_dir.buf); - if (!git_dir) - git_dir = submodule_git_dir.buf; - if (is_directory(git_dir)) { + repo = get_submodule_repo_for(spf->r, submodule); + if (repo) { child_process_init(cp); - cp->dir = strbuf_detach(&submodule_path, NULL); - prepare_submodule_repo_env(&cp->env_array); + prepare_submodule_repo_env_in_gitdir(&cp->env_array); + cp->dir = xstrdup(repo->gitdir); cp->git_cmd = 1; if (!spf->quiet) strbuf_addf(err, "Fetching submodule %s%s\n", @@ -1293,10 +1317,11 @@ static int get_next_submodule(struct child_process *cp, argv_array_push(&cp->args, default_argv); argv_array_push(&cp->args, "--submodule-prefix"); argv_array_push(&cp->args, submodule_prefix.buf); + + repo_clear(repo); + free(repo); ret = 1; } - strbuf_release(&submodule_path); - strbuf_release(&submodule_git_dir); strbuf_release(&submodule_prefix); if (ret) { spf->count++; diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh index 6c2f9b2ba2..42692219a1 100755 --- a/t/t5526-fetch-submodules.sh +++ b/t/t5526-fetch-submodules.sh @@ -566,7 +566,12 @@ test_expect_success 'fetching submodule into a broken repository' ' test_must_fail git -C dst status && test_must_fail git -C dst diff && - test_must_fail git -C dst fetch --recurse-submodules + + # git-fetch cannot find the git directory of the submodule, + # so it will do nothing, successfully, as it cannot distinguish between + # this broken submodule and a submodule that was just set active but + # not cloned yet + git -C dst fetch --recurse-submodules ' test_expect_success "fetch new commits when submodule got renamed" ' From patchwork Wed Oct 10 21:50:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10635367 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7C61E46E4 for ; Wed, 10 Oct 2018 21:50:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 67B142ABB9 for ; Wed, 10 Oct 2018 21:50:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5C1372AB51; Wed, 10 Oct 2018 21:50:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6AC682AE7C for ; Wed, 10 Oct 2018 21:50:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726052AbeJKFPC (ORCPT ); Thu, 11 Oct 2018 01:15:02 -0400 Received: from mail-qt1-f201.google.com ([209.85.160.201]:40130 "EHLO mail-qt1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725856AbeJKFPC (ORCPT ); Thu, 11 Oct 2018 01:15:02 -0400 Received: by mail-qt1-f201.google.com with SMTP id y23-v6so6562532qtc.7 for ; Wed, 10 Oct 2018 14:50:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=8+MnwU3zOWclwq8TrnY01lM/pbzmVmbeMUUf9s3ONxA=; b=D+0wA9an+P6xNpRfY8lVvZFpnYALqYBHGJKVF2XJoAVlG9rSwbchn2QzD0WYkQuPRw ad718G1THsbsx0HKrQ6bLCMLQ0DktkbQmytdhyXWgr4HUIHTSwXohdGP3Gbb4pZDfykY /lVqOX2vH+a3OjFDeychkku8jtLt5hOUU4bA5KJxJZVnBYWIPPsAykGsC2pg8PkQjkt8 xJ46NJZdsorSB2qSpjjhIsvDUD8JdAOs5/Y0ILkNNxYGcD8Q0qjmwEIK+pLjcL0T+x2j eQtrF9AHnEeLAez5I+qdbXc15/fRVi9BmKhBvL0ybzPIz1eZXpV7OvivowFnI/Zui5U3 6HSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=8+MnwU3zOWclwq8TrnY01lM/pbzmVmbeMUUf9s3ONxA=; b=Z/cp1KNR4xc+cDOC9P5k6uObvJXQ2E/GQLPocTVjF9yauQ/nCwZZbOKgnQ01INNsQc yM9398CbdP1tPqLgtScuAmDrWKIsXuRR98i/faBCSBnSgGst8qLbiehCekWNFVNl2lbf uxSPM6EYeHhRg4ZKmQ7Tdy8CNcVMJYwD/8o7gY1xemklFpI1YLTeBWs8blBli3UXPdNl DCdHP93idLEVS5WWPP6bcqh6hGsQWTQU7A026ypKqADUY+DxEyqtHCvj1rqN6ahyOGIp jc23DJlA+37Ze56lmXk87b3sZn488wvKKdpb1qoESRcd38A1ULih1FMcbDhFCuwpkYra ojSw== X-Gm-Message-State: ABuFfogVfNn5uW/0rLhQe49DpVXH7Kpz6VfqZl1eDYtfEIoYq8fDS3f/ kFDQMMBp4TG2WvmVMrPBXEgCWJkxPeYJknahpbRdyWU3gPatq3QIeiI/T2iP4v4fToGJtOXHDUZ xnQFM2C+Q7U0++qPyVSI57/WScIR6bMp3+PmIdZKyiAXrVyzIgC4YoKtLxcsu X-Google-Smtp-Source: ACcGV61GflzDNqndPEi2j1YmWOgSGOq1BJcWBmZSFxcwjLZP96j2d4H/7FU3iOPJdjYpbM3gsYtngb5L2Ci6 X-Received: by 2002:ac8:347d:: with SMTP id v58-v6mr1870557qtb.6.1539208255904; Wed, 10 Oct 2018 14:50:55 -0700 (PDT) Date: Wed, 10 Oct 2018 14:50:07 -0700 In-Reply-To: <20181010215008.233529-1-sbeller@google.com> Message-Id: <20181010215008.233529-9-sbeller@google.com> Mime-Version: 1.0 References: <20181010215008.233529-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH v5 8/9] fetch: retry fetching submodules if needed objects were not fetched From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Currently when git-fetch is asked to recurse into submodules, it dispatches a plain "git-fetch -C " (with some submodule related options such as prefix and recusing strategy, but) without any information of the remote or the tip that should be fetched. This works surprisingly well in some workflows (such as using submodules as a third party library), while not so well in other scenarios, such as in a Gerrit topic-based workflow, that can tie together changes (potentially across repositories) on the server side. One of the parts of such a Gerrit workflow is to download a change when wanting to examine it, and you'd want to have its submodule changes that are in the same topic downloaded as well. However these submodule changes reside in their own repository in their own ref (refs/changes/). Retry fetching a submodule by object name if the object id that the superproject points to, cannot be found. This retrying does not happen when the "git fetch" done at the superproject is not storing the fetched results in remote tracking branches (i.e. instead just recording them to FETCH_HEAD) in this step. A later patch will fix this. builtin/fetch used to only inspect submodules when they were fetched "on-demand", as in either on/off case it was clear whether the submodule needs to be fetched. However to know whether we need to try fetching the object ids, we need to identify the object names, which is done in this function check_for_new_submodule_commits(), so we'll also run that code in case the submodule recursion is set to "on". Signed-off-by: Stefan Beller --- builtin/fetch.c | 9 +- submodule.c | 185 ++++++++++++++++++++++++++++++------ t/t5526-fetch-submodules.sh | 16 ++++ 3 files changed, 177 insertions(+), 33 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 0696abfc2a..e3b03ad3bd 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -707,8 +707,7 @@ static int update_local_ref(struct ref *ref, what = _("[new ref]"); } - if ((recurse_submodules != RECURSE_SUBMODULES_OFF) && - (recurse_submodules != RECURSE_SUBMODULES_ON)) + if (recurse_submodules != RECURSE_SUBMODULES_OFF) check_for_new_submodule_commits(&ref->new_oid); r = s_update_ref(msg, ref, 0); format_display(display, r ? '!' : '*', what, @@ -723,8 +722,7 @@ static int update_local_ref(struct ref *ref, strbuf_add_unique_abbrev(&quickref, ¤t->object.oid, DEFAULT_ABBREV); strbuf_addstr(&quickref, ".."); strbuf_add_unique_abbrev(&quickref, &ref->new_oid, DEFAULT_ABBREV); - if ((recurse_submodules != RECURSE_SUBMODULES_OFF) && - (recurse_submodules != RECURSE_SUBMODULES_ON)) + if (recurse_submodules != RECURSE_SUBMODULES_OFF) check_for_new_submodule_commits(&ref->new_oid); r = s_update_ref("fast-forward", ref, 1); format_display(display, r ? '!' : ' ', quickref.buf, @@ -738,8 +736,7 @@ static int update_local_ref(struct ref *ref, strbuf_add_unique_abbrev(&quickref, ¤t->object.oid, DEFAULT_ABBREV); strbuf_addstr(&quickref, "..."); strbuf_add_unique_abbrev(&quickref, &ref->new_oid, DEFAULT_ABBREV); - if ((recurse_submodules != RECURSE_SUBMODULES_OFF) && - (recurse_submodules != RECURSE_SUBMODULES_ON)) + if (recurse_submodules != RECURSE_SUBMODULES_OFF) check_for_new_submodule_commits(&ref->new_oid); r = s_update_ref("forced-update", ref, 1); format_display(display, r ? '!' : '+', quickref.buf, diff --git a/submodule.c b/submodule.c index 3f791f2277..05799362e0 100644 --- a/submodule.c +++ b/submodule.c @@ -1127,8 +1127,12 @@ struct submodule_parallel_fetch { int result; struct string_list changed_submodule_names; + struct get_next_submodule_task **retry; + int retry_nr, retry_alloc; }; -#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0, STRING_LIST_INIT_DUP } +#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0, \ + STRING_LIST_INIT_DUP, \ + NULL, 0, 0} static void calculate_changed_submodule_paths( struct submodule_parallel_fetch *spf) @@ -1233,6 +1237,56 @@ static int get_fetch_recurse_config(const struct submodule *submodule, return spf->default_option; } +struct get_next_submodule_task { + struct repository *repo; + const struct submodule *sub; + unsigned free_sub : 1; /* Do we need to free the submodule? */ + struct oid_array *commits; +}; + +static const struct submodule *get_default_submodule(const char *path) +{ + struct submodule *ret = NULL; + const char *name = default_name_or_path(path); + + if (!name) + return NULL; + + ret = xmalloc(sizeof(*ret)); + memset(ret, 0, sizeof(*ret)); + ret->path = name; + ret->name = name; + + return (const struct submodule *) ret; +} + +static struct get_next_submodule_task *get_next_submodule_task_create( + struct repository *r, const char *path) +{ + struct get_next_submodule_task *task = xmalloc(sizeof(*task)); + memset(task, 0, sizeof(*task)); + + task->sub = submodule_from_path(r, &null_oid, path); + if (!task->sub) { + task->sub = get_default_submodule(path); + task->free_sub = 1; + } + + return task; +} + +static void get_next_submodule_task_release(struct get_next_submodule_task *p) +{ + if (p->free_sub) + free((void*)p->sub); + p->free_sub = 0; + p->sub = NULL; + + if (p->repo) + repo_clear(p->repo); + FREE_AND_NULL(p->repo); +} + static struct repository *get_submodule_repo_for(struct repository *r, const struct submodule *sub) { @@ -1259,39 +1313,35 @@ static struct repository *get_submodule_repo_for(struct repository *r, static int get_next_submodule(struct child_process *cp, struct strbuf *err, void *data, void **task_cb) { - int ret = 0; struct submodule_parallel_fetch *spf = data; for (; spf->count < spf->r->index->cache_nr; spf->count++) { - struct strbuf submodule_prefix = STRBUF_INIT; + int recurse_config; const struct cache_entry *ce = spf->r->index->cache[spf->count]; const char *default_argv; - const struct submodule *submodule; - struct repository *repo; - struct submodule default_submodule = SUBMODULE_INIT; + struct get_next_submodule_task *task; if (!S_ISGITLINK(ce->ce_mode)) continue; - submodule = submodule_from_path(spf->r, &null_oid, ce->name); - if (!submodule) { - const char *name = default_name_or_path(ce->name); - if (name) { - default_submodule.path = name; - default_submodule.name = name; - submodule = &default_submodule; - } + task = get_next_submodule_task_create(spf->r, ce->name); + + if (!task->sub) { + free(task); + continue; } - switch (get_fetch_recurse_config(submodule, spf)) + recurse_config = get_fetch_recurse_config(task->sub, spf); + + switch (recurse_config) { default: case RECURSE_SUBMODULES_DEFAULT: case RECURSE_SUBMODULES_ON_DEMAND: - if (!submodule || + if (!task->sub || !string_list_lookup( &spf->changed_submodule_names, - submodule->name)) + task->sub->name)) continue; default_argv = "on-demand"; break; @@ -1302,12 +1352,12 @@ static int get_next_submodule(struct child_process *cp, continue; } - strbuf_addf(&submodule_prefix, "%s%s/", spf->prefix, ce->name); - repo = get_submodule_repo_for(spf->r, submodule); - if (repo) { + task->repo = get_submodule_repo_for(spf->r, task->sub); + if (task->repo) { + struct strbuf submodule_prefix = STRBUF_INIT; child_process_init(cp); prepare_submodule_repo_env_in_gitdir(&cp->env_array); - cp->dir = xstrdup(repo->gitdir); + cp->dir = task->repo->gitdir; cp->git_cmd = 1; if (!spf->quiet) strbuf_addf(err, "Fetching submodule %s%s\n", @@ -1316,18 +1366,51 @@ static int get_next_submodule(struct child_process *cp, argv_array_pushv(&cp->args, spf->args.argv); argv_array_push(&cp->args, default_argv); argv_array_push(&cp->args, "--submodule-prefix"); + + strbuf_addf(&submodule_prefix, "%s%s/", + spf->prefix, + task->sub->path); argv_array_push(&cp->args, submodule_prefix.buf); - repo_clear(repo); - free(repo); - ret = 1; - } - strbuf_release(&submodule_prefix); - if (ret) { spf->count++; + *task_cb = task; + + strbuf_release(&submodule_prefix); return 1; + } else { + get_next_submodule_task_release(task); + free(task); } } + + if (spf->retry_nr) { + struct get_next_submodule_task *task = spf->retry[spf->retry_nr - 1]; + struct strbuf submodule_prefix = STRBUF_INIT; + spf->retry_nr--; + + strbuf_addf(&submodule_prefix, "%s%s/", spf->prefix, task->sub->path); + + child_process_init(cp); + prepare_submodule_repo_env_in_gitdir(&cp->env_array); + cp->git_cmd = 1; + cp->dir = task->repo->gitdir; + + argv_array_init(&cp->args); + argv_array_pushv(&cp->args, spf->args.argv); + argv_array_push(&cp->args, "on-demand"); + argv_array_push(&cp->args, "--submodule-prefix"); + argv_array_push(&cp->args, submodule_prefix.buf); + + /* NEEDSWORK: have get_default_remote from s--h */ + argv_array_push(&cp->args, "origin"); + oid_array_for_each_unique(task->commits, + append_oid_to_argv, &cp->args); + + *task_cb = task; + strbuf_release(&submodule_prefix); + return 1; + } + return 0; } @@ -1335,20 +1418,68 @@ static int fetch_start_failure(struct strbuf *err, void *cb, void *task_cb) { struct submodule_parallel_fetch *spf = cb; + struct get_next_submodule_task *task = task_cb; spf->result = 1; + get_next_submodule_task_release(task); return 0; } +static int commit_exists_in_sub(const struct object_id *oid, void *data) +{ + struct repository *subrepo = data; + + enum object_type type = oid_object_info(subrepo, oid, NULL); + + return type != OBJ_COMMIT; +} + static int fetch_finish(int retvalue, struct strbuf *err, void *cb, void *task_cb) { struct submodule_parallel_fetch *spf = cb; + struct get_next_submodule_task *task = task_cb; + const struct submodule *sub; + + struct string_list_item *it; + struct oid_array *commits; if (retvalue) spf->result = 1; + if (!task) + return 0; + + sub = task->sub; + if (!sub) + goto out; + + it = string_list_lookup(&spf->changed_submodule_names, sub->name); + if (!it) + goto out; + + commits = it->util; + oid_array_filter(commits, + commit_exists_in_sub, + task->repo); + + /* Are there commits that do not exist? */ + if (commits->nr) { + /* We already tried fetching them, do not try again. */ + if (task->commits) + return 0; + + task->commits = commits; + ALLOC_GROW(spf->retry, spf->retry_nr + 1, spf->retry_alloc); + spf->retry[spf->retry_nr] = task; + spf->retry_nr++; + return 0; + } + +out: + get_next_submodule_task_release(task); + return 0; } diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh index 42692219a1..af12c50e7d 100755 --- a/t/t5526-fetch-submodules.sh +++ b/t/t5526-fetch-submodules.sh @@ -605,4 +605,20 @@ test_expect_success "fetch new commits when submodule got renamed" ' test_cmp expect actual ' +test_expect_success "fetch new commits on-demand when they are not reachable" ' + git checkout --detach && + C=$(git -C submodule commit-tree -m "new change outside refs/heads" HEAD^{tree}) && + git -C submodule update-ref refs/changes/1 $C && + git update-index --cacheinfo 160000 $C submodule && + git commit -m "updated submodule outside of refs/heads" && + D=$(git rev-parse HEAD) && + git update-ref refs/changes/2 $D && + ( + cd downstream && + git fetch --recurse-submodules --recurse-submodules-default on-demand origin refs/changes/2:refs/heads/my_branch && + git -C submodule cat-file -t $C && + git checkout --recurse-submodules FETCH_HEAD + ) +' + test_done From patchwork Wed Oct 10 21:50:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10635369 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 89EB146E4 for ; Wed, 10 Oct 2018 21:51:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 680AD2AE8E for ; Wed, 10 Oct 2018 21:51:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 661742AE97; Wed, 10 Oct 2018 21:51:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 02D9A2AE90 for ; Wed, 10 Oct 2018 21:51:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726070AbeJKFPJ (ORCPT ); Thu, 11 Oct 2018 01:15:09 -0400 Received: from mail-vk1-f201.google.com ([209.85.221.201]:39077 "EHLO mail-vk1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725856AbeJKFPJ (ORCPT ); Thu, 11 Oct 2018 01:15:09 -0400 Received: by mail-vk1-f201.google.com with SMTP id y83-v6so2502282vky.6 for ; Wed, 10 Oct 2018 14:51:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=K2sUjtliowoUR75xLeTIpXUX4me2YUXUkwJgta0pQ0o=; b=hwNBL0DQD8km1VJ9ucQj4VXoY2WY5lbskZiSvTkV9RyZCxZBNeKQm5hvuYMijbdFDd AOnmPJZp8OQaHJUWIKd9wDn+O6pysG7gRWKmb+ibX6i/tk3QnatcwNXpshgj5aogz5Zk ok3sG97NbOANcHtacPzuA5CdTlFKrpcBTGJAUQyTp7nsbeN++4KFMkYoBmaRKJZEXOh2 nZsCy1sartp5My8Po8EG8ngYo5RITva+iTTuLa5s95iNUQ5WQkWSQCtlMtYiZ7bAVD8P MsBtOmLJpLP8PL8YRNXBGhjs+U3vw5qH1kByFx/EgUE0ryvmzfD+hdw6vBOpFf2yVJ26 oaRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=K2sUjtliowoUR75xLeTIpXUX4me2YUXUkwJgta0pQ0o=; b=hf0imcPhe0DzkaFlK1YcaFYLShXH32nnToDpqmdExqIhX0N2X4qSvvortLWYWf2ix+ SfQwIV3SWmJxYmXPORIUBI+XXpAfZYXiebYtekA3EPvaTxhp7PQfxrtTFYAORjuvS/t0 cMmVWbdjTHgBpDfpkdUciPgk0mUlqoiHZi5hD1feQECRj8inoCJbrcITZNMNUsLxzFlT tg/tRItTvFaM1JlLJirdkXetKf+OSpWk4XABsUZDrGYAR+BcM8m+oZ3JG5NtZp6KDENy RASGD0mdLBV6gxrw/gv5rGQqDKHoCQKN29k7bFNCPrTctLQ3sMFR9fNkTOXStjLD+bhi jrEw== X-Gm-Message-State: ABuFfogVaulaYC7T4dXUbeDqKOmD+lAzjpLepGhcQQXYHGpmDyzDSPg1 ZbGOjPHzhoxsh2/ZuD+YC+X5DGDSy8Q1fFOVPrnBrbS6DH48mzKi1CHXsMrcqj1N/ag8KEpWOAn T/Q5n7+5ZzP8/+KrDRqCG9j7RdbxxSJBYVUILRp3FjZkFY7EvFs5M+jYwu0jX X-Google-Smtp-Source: ACcGV632WP/B+O/spzl8jxgQFv3H0V9BpqQyK6rtlpL98YyEwNjHQFMKAyXOLBfDFyaH482fjhoKGa3YQWST X-Received: by 2002:ab0:446:: with SMTP id 64mr30277082uav.0.1539208263898; Wed, 10 Oct 2018 14:51:03 -0700 (PDT) Date: Wed, 10 Oct 2018 14:50:08 -0700 In-Reply-To: <20181010215008.233529-1-sbeller@google.com> Message-Id: <20181010215008.233529-10-sbeller@google.com> Mime-Version: 1.0 References: <20181010215008.233529-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH v5 9/9] builtin/fetch: check for submodule updates for non branch fetches From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Gerrit, the code review tool, has a different workflow than our mailing list based approach. Usually users upload changes to a Gerrit server and continuous integration and testing happens by bots. Sometimes however a user wants to checkout a change locally and look at it locally. For this use case, Gerrit offers a command line snippet to copy and paste to your terminal, which looks like git fetch https:///gerrit refs/changes/ && git checkout FETCH_HEAD For Gerrit changes that contain changing submodule gitlinks, it would be easy to extend both the fetch and checkout with the '--recurse-submodules' flag, such that this command line snippet would produce the state of a change locally. However the functionality added in the previous patch, which would ensure that we fetch the objects in the submodule that the gitlink pointed at, only works for remote tracking branches so far, not for FETCH_HEAD. Make sure that fetching a superproject to its FETCH_HEAD, also respects the existence checks for objects in the submodule recursion. Signed-off-by: Stefan Beller --- builtin/fetch.c | 5 ++++- t/t5526-fetch-submodules.sh | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index e3b03ad3bd..f2d9e548bf 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -894,11 +894,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, rc |= update_local_ref(ref, what, rm, ¬e, summary_width); free(ref); - } else + } else { + check_for_new_submodule_commits(&rm->old_oid); format_display(¬e, '*', *kind ? kind : "branch", NULL, *what ? what : "HEAD", "FETCH_HEAD", summary_width); + } + if (note.len) { if (verbosity >= 0 && !shown_url) { fprintf(stderr, _("From %.*s\n"), diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh index af12c50e7d..a509eabb04 100755 --- a/t/t5526-fetch-submodules.sh +++ b/t/t5526-fetch-submodules.sh @@ -615,7 +615,7 @@ test_expect_success "fetch new commits on-demand when they are not reachable" ' git update-ref refs/changes/2 $D && ( cd downstream && - git fetch --recurse-submodules --recurse-submodules-default on-demand origin refs/changes/2:refs/heads/my_branch && + git fetch --recurse-submodules origin refs/changes/2 && git -C submodule cat-file -t $C && git checkout --recurse-submodules FETCH_HEAD )