From patchwork Tue Oct 16 18:13:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10644019 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8E04017D2 for ; Tue, 16 Oct 2018 18:13:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7E7F420000 for ; Tue, 16 Oct 2018 18:13:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7247528ABE; Tue, 16 Oct 2018 18:13:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0EC3E20000 for ; Tue, 16 Oct 2018 18:13:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727249AbeJQCFM (ORCPT ); Tue, 16 Oct 2018 22:05:12 -0400 Received: from mail-ot1-f73.google.com ([209.85.210.73]:38230 "EHLO mail-ot1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727006AbeJQCFM (ORCPT ); Tue, 16 Oct 2018 22:05:12 -0400 Received: by mail-ot1-f73.google.com with SMTP id g2so5728816otb.5 for ; Tue, 16 Oct 2018 11:13:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+rOFzW5bJHQCIBM022kwARqDlluGnq5Nr5y183LRgpg=; b=T1q4kVryPBS+dqxSf8n1QEYB4NIvyUpVab6TEIvg5kD1aJ8gEw84il4zZWzRz6jfs5 xE2zc2e8CSbRiYzY+VeHalR2jhV5tOE7xHlLOwQtTVwkHxLyDvsgJ69JvOdZ/WYtBkRT 8u7S4LyJjVqEKBiD84DdprP0sAl3CI2L27qVbUccb1HOniOKb8CzMJLEHJOR9iABcGzG wYOWLIf2PVXUhxW1OI+wn36bnk1lJBN8izqrlJW76dWZpCsAc0ezLph9vgxwQdFFtn31 8mpBdLygbBdkjgNCnapKHLlVMQfAjPsm2QY9JxcQ8WI+tMQchQFZLOhELrUJYH1LSHVU KSWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+rOFzW5bJHQCIBM022kwARqDlluGnq5Nr5y183LRgpg=; b=bne98K241oj8JmXSeFT0Za5GcxMzR02R/7sW7kncdVsxUHcnNUuu1NsE6JTBeB5s4D rulfimnbekH1Tck2ayTqIPGB50Hr2kjZl0Jry+0Wky5b/3eyl9mW/j1HNIjNba7IwnTc hpM+0i84mNrMzi2CUA9UJX1r8D0fEv3U1QL7jTHI89c702PWJIqW5z17G3T1Y4Sp/Lbf 2ENhJlDk0siV0gpaCteO8WLYvA6EqIicP/t+sq3isb4pOtIesTHFGN6hF0VWlAqfWWq4 O/k3LWEyFx546A4yDKTbjbZO2w0U4m5KQWKz0YkQcgz4k41SkMGQw27dh3NZdYHrCHwh uKMg== X-Gm-Message-State: ABuFfojHsTjs9Aclsh4B5QB8R2eXUAkpxJEEU7jtEV8Gah4SiEl8E/Ya dXBqMyRi0fw1DSdRn4vOeQWJ/s8HDRE0 X-Google-Smtp-Source: ACcGV62vtyUaxsPQu6fcfwm0x/nJCt66TxuLr4ynO+fpgrpsfJiXeYVL9v8nhQ9HRaCZFuA08C7fstjN3Bm/ X-Received: by 2002:aca:d695:: with SMTP id n143-v6mr19642571oig.23.1539713612842; Tue, 16 Oct 2018 11:13:32 -0700 (PDT) Date: Tue, 16 Oct 2018 11:13:19 -0700 In-Reply-To: <20181016181327.107186-1-sbeller@google.com> Message-Id: <20181016181327.107186-2-sbeller@google.com> Mime-Version: 1.0 References: <20181016181327.107186-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH 1/9] sha1-array: provide oid_array_filter From: Stefan Beller To: gitster@pobox.com Cc: git@vger.kernel.org, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Helped-by: Junio C Hamano Signed-off-by: Stefan Beller Signed-off-by: Junio C Hamano --- Documentation/technical/api-oid-array.txt | 5 +++++ sha1-array.c | 17 +++++++++++++++++ sha1-array.h | 3 +++ 3 files changed, 25 insertions(+) diff --git a/Documentation/technical/api-oid-array.txt b/Documentation/technical/api-oid-array.txt index 9febfb1d52..c97428c2c3 100644 --- a/Documentation/technical/api-oid-array.txt +++ b/Documentation/technical/api-oid-array.txt @@ -48,6 +48,11 @@ Functions is not sorted, this function has the side effect of sorting it. +`oid_array_filter`:: + Apply the callback function `want` to each entry in the array, + retaining only the entries for which the function returns true. + Preserve the order of the entries that are retained. + Examples -------- diff --git a/sha1-array.c b/sha1-array.c index 265941fbf4..d505a004bb 100644 --- a/sha1-array.c +++ b/sha1-array.c @@ -77,3 +77,20 @@ int oid_array_for_each_unique(struct oid_array *array, } return 0; } + +void oid_array_filter(struct oid_array *array, + for_each_oid_fn want, + void *cb_data) +{ + unsigned nr = array->nr, src, dst; + struct object_id *oids = array->oid; + + for (src = dst = 0; src < nr; src++) { + if (want(&oids[src], cb_data)) { + if (src != dst) + oidcpy(&oids[dst], &oids[src]); + dst++; + } + } + array->nr = dst; +} diff --git a/sha1-array.h b/sha1-array.h index 232bf95017..55d016c4bf 100644 --- a/sha1-array.h +++ b/sha1-array.h @@ -22,5 +22,8 @@ int oid_array_for_each(struct oid_array *array, int oid_array_for_each_unique(struct oid_array *array, for_each_oid_fn fn, void *data); +void oid_array_filter(struct oid_array *array, + for_each_oid_fn want, + void *cbdata); #endif /* SHA1_ARRAY_H */ From patchwork Tue Oct 16 18:13:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10644021 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ACDE015E2 for ; Tue, 16 Oct 2018 18:13:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A1D3520000 for ; Tue, 16 Oct 2018 18:13:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9537628ABE; Tue, 16 Oct 2018 18:13:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 29E2C20000 for ; Tue, 16 Oct 2018 18:13:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727285AbeJQCFO (ORCPT ); Tue, 16 Oct 2018 22:05:14 -0400 Received: from mail-pg1-f201.google.com ([209.85.215.201]:38203 "EHLO mail-pg1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727006AbeJQCFO (ORCPT ); Tue, 16 Oct 2018 22:05:14 -0400 Received: by mail-pg1-f201.google.com with SMTP id e6-v6so17931122pge.5 for ; Tue, 16 Oct 2018 11:13:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=B00n941xueuBWwdXzRKPIBl5kroqyU94J3aafKflOrg=; b=ZCkKjCz4QQ9Fr9WT3kO6outpxHmJERqg8Y0hkmUQTJPg677rE4uTsBERtEDQgqhUWR Fonl1ySNHHS+205zHhUvUEpDrY+Plo5IUvKy8ggf9b2c/d785wl7BZCGnKsstBo75C5g uJcagjTsEU/ymz90DoSyEjO4E0kT1BxM21KdZcocbX8Dm/yF7YKy75scoAa/XI12WW/z mcSVqygeXwlikE+qNlKXTBc9D8ZN4liHslHGTfWc8NqRDmNANo+39BKvj6YHNN9PWC52 ooOYAXb+IiKko8pGG2F5IxEo/uAjtnxu5zVuqx3kWTowc6rC8N+SYaQC3E1FdgrE3RPD f7sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=B00n941xueuBWwdXzRKPIBl5kroqyU94J3aafKflOrg=; b=p4+7TVFYAyJyILF0gMQn6R4yCIR1iaWo1NuaQOBZaXG/qYAoNb2we9wDk1up+g16Ss c8mr9r843tXU7isM9bz5OZTKlESsLMt8JQWShoIv5mF48VuD4G8+9XW9hSgeqTVOzz7x Cr2XyLXHFHgU1VvcHEOt+uE+q2EOwOPU3Uj7h7zo+HBnm31RJ/LfZfVfzd4Ex7jQKP6R QZbvCJQo9eqpj/15WtpBoCNjuUlfcGLVl6qz02/kZbAy1870D5LfPoJtVXzDDulFz2dd ud/8WiNAXtbw2OZqmp96KP5m18D3qcBpgpjXG1dJkwJHM4zw0N4xibqNBC1RzrO0MUn9 eouQ== X-Gm-Message-State: ABuFfohLx7Vg99lzu5IWwmO6sTokNiZ0+yETxtTEDdLXjli2TjyxEMVA D71ylCiCuXFBOArFzwcN0rrhPvvzvk1t X-Google-Smtp-Source: ACcGV62K2oJ/RFl+Suvul/+pGWLZVTMRLweuEIndFdHDLX/39FN4h0wUtAt1E/xKoYUiFHPnDXmdjUUG2a4W X-Received: by 2002:a62:4285:: with SMTP id h5-v6mr11038147pfd.37.1539713615071; Tue, 16 Oct 2018 11:13:35 -0700 (PDT) Date: Tue, 16 Oct 2018 11:13:20 -0700 In-Reply-To: <20181016181327.107186-1-sbeller@google.com> Message-Id: <20181016181327.107186-3-sbeller@google.com> Mime-Version: 1.0 References: <20181016181327.107186-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH 2/9] submodule.c: fix indentation From: Stefan Beller To: gitster@pobox.com Cc: git@vger.kernel.org, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The submodule subsystem is really bad at staying within 80 characters. Fix it while we are here. Signed-off-by: Stefan Beller Signed-off-by: Junio C Hamano --- submodule.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/submodule.c b/submodule.c index 2b7082b2db..e145ebbb16 100644 --- a/submodule.c +++ b/submodule.c @@ -1258,7 +1258,8 @@ static int get_next_submodule(struct child_process *cp, if (!submodule) { const char *name = default_name_or_path(ce->name); if (name) { - default_submodule.path = default_submodule.name = name; + default_submodule.path = name; + default_submodule.name = name; submodule = &default_submodule; } } @@ -1268,8 +1269,10 @@ static int get_next_submodule(struct child_process *cp, default: case RECURSE_SUBMODULES_DEFAULT: case RECURSE_SUBMODULES_ON_DEMAND: - if (!submodule || !unsorted_string_list_lookup(&changed_submodule_names, - submodule->name)) + if (!submodule || + !unsorted_string_list_lookup( + &changed_submodule_names, + submodule->name)) continue; default_argv = "on-demand"; break; From patchwork Tue Oct 16 18:13:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10644023 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A614315E2 for ; Tue, 16 Oct 2018 18:13:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 999D120000 for ; Tue, 16 Oct 2018 18:13:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8DE8A28ABE; Tue, 16 Oct 2018 18:13:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3158420000 for ; Tue, 16 Oct 2018 18:13:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727289AbeJQCFR (ORCPT ); Tue, 16 Oct 2018 22:05:17 -0400 Received: from mail-vs1-f73.google.com ([209.85.217.73]:35351 "EHLO mail-vs1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727006AbeJQCFQ (ORCPT ); Tue, 16 Oct 2018 22:05:16 -0400 Received: by mail-vs1-f73.google.com with SMTP id x7so10997498vsf.2 for ; Tue, 16 Oct 2018 11:13:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=dKkiRNzxRqU0laEc6xb/t6nsn4iaEPm7vic1aGSiMvA=; b=DXIa48EPSO7bbzBQf4WudNQW5ctiug9ibPYc6RMiRxqW6jrRghbqbfKjuPOdDhsnf0 08/K8NQXNLHNMgoM+S+1BeWuXjRTnOqXXq5fNDH9lQRhUukN03gmJgw5nqKLTr4cRElq 8/j3lW6TnhOLFGPOsf+/OAhinAaojG9ELcJ1GziNMC0qNXHXH/9XSeV7oFNJa4vi1wBi LBJdZdtf78auEgaDYjEIFdsb11VhvW0rA34n09akwor74g/CWw3aZT7/JDKnu4ON7KHS knl8z6CrDUipJZCz6WxsVZ/juHVmbVaUF1S4g4533Is6OWrde1QgoFN6qBTseZBT/C7d y6ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=dKkiRNzxRqU0laEc6xb/t6nsn4iaEPm7vic1aGSiMvA=; b=t61+nkI34IoYg3gjeVmTkq+GrKF7UevRPuVyhzZQCXrv3XnFXzzyfbImXzL5k+LHmf 5a4Wp+9vzdZfDwcrFBQD1FbAPcf+52jeUoeHSY1gyRYgZBfJP0d7Z+uMQJHEy4tDxdtq DeKgXQHfO3oceQnkMaTc94fS4Xe3cNiBdU8vM57p/N8cQmQI2r6ITIwg1tf9rOl07d7Q iKesIl5ZWzmUaqx1MuO7lLDJk6uTQyrsrNNUiBhVbEs7V+gGYbKuAo8s+dPjN/tZ9Tfs p/sfzLFI+neFNUI4OqOL0u2URlWm4repX8BhqZZXKj3OaxYiicRMYz8A46J6yMN+Chc0 HtLA== X-Gm-Message-State: ABuFfogX3VWAMJi9tth3e4i3WQaZqxDqHhTcPpdmd75g1oV2GRg1y3XP w114kFsyTTuiyWSUq1eskCynu+5CdxYK X-Google-Smtp-Source: ACcGV60WyOJitA6ePTZZk31SQbzp3oedHkR5Ll/pYX14cOc0jmHUgTXJocURrXwvO90PMM6jZD67oeOYJY74 X-Received: by 2002:a67:854b:: with SMTP id h72-v6mr22352149vsd.31.1539713617384; Tue, 16 Oct 2018 11:13:37 -0700 (PDT) Date: Tue, 16 Oct 2018 11:13:21 -0700 In-Reply-To: <20181016181327.107186-1-sbeller@google.com> Message-Id: <20181016181327.107186-4-sbeller@google.com> Mime-Version: 1.0 References: <20181016181327.107186-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH 3/9] submodule.c: sort changed_submodule_names before searching it From: Stefan Beller To: gitster@pobox.com Cc: git@vger.kernel.org, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We can string_list_insert() to maintain sorted-ness of the list as we find new items, or we can string_list_append() to build an unsorted list and sort it at the end just once. As we do not rely on the sortedness while building the list, we pick the "append and sort at the end" as it has better worst case execution times. Signed-off-by: Stefan Beller Signed-off-by: Junio C Hamano --- submodule.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/submodule.c b/submodule.c index e145ebbb16..9fbfcfcfe1 100644 --- a/submodule.c +++ b/submodule.c @@ -1270,7 +1270,7 @@ static int get_next_submodule(struct child_process *cp, case RECURSE_SUBMODULES_DEFAULT: case RECURSE_SUBMODULES_ON_DEMAND: if (!submodule || - !unsorted_string_list_lookup( + !string_list_lookup( &changed_submodule_names, submodule->name)) continue; @@ -1364,6 +1364,7 @@ int fetch_populated_submodules(struct repository *r, /* default value, "--submodule-prefix" and its value are added later */ calculate_changed_submodule_paths(); + string_list_sort(&changed_submodule_names); run_processes_parallel(max_parallel_jobs, get_next_submodule, fetch_start_failure, From patchwork Tue Oct 16 18:13:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10644025 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8B03A109C for ; Tue, 16 Oct 2018 18:13:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7BDAE2A587 for ; Tue, 16 Oct 2018 18:13:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6FB9D28ABE; Tue, 16 Oct 2018 18:13:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0837220000 for ; Tue, 16 Oct 2018 18:13:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727292AbeJQCFW (ORCPT ); Tue, 16 Oct 2018 22:05:22 -0400 Received: from mail-qk1-f202.google.com ([209.85.222.202]:47122 "EHLO mail-qk1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727006AbeJQCFW (ORCPT ); Tue, 16 Oct 2018 22:05:22 -0400 Received: by mail-qk1-f202.google.com with SMTP id f81-v6so24862102qkb.14 for ; Tue, 16 Oct 2018 11:13:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=x62wfanp9xsLqfARQY2I/omzlwaVx2Vc9lal1QDVfGA=; b=gJTiUcDHjANozH4x2pgczogGZgMQN+GqwFJdEhD0Aj3w5hK1utyTsYUofqGlo31yWd xZ1DqKwtY+jJgxIxwYRcfP2SmXqzWNcE/ALZivxdZqJpfUVZc5ZuUuHf4HCWcCOhSGZ7 /ExnJ0fIKBXu89o/6IA6xMLlo4zpWTTG+8mwOnsENHz1PEdeHs1yVp64UT2uZWHESxlG 4KfhKRu4UQmlZUMQbjLqJB5A2DxPcl8OXS/Tmq9bRbPYx138jMALBU2ni//+ssYow+gt hdWjeTViOkTJ+hM4sFVLVGh3YxvUMxTdvU5eBeO/TiS0G4R1RUp7fLS6yTpoeBqgJ/ti soPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=x62wfanp9xsLqfARQY2I/omzlwaVx2Vc9lal1QDVfGA=; b=L/oTLhmjKUML7WCmb3Yt7aGcyZzQyKmJtyrD8il3vZMpteIXPoK6+voASeJ8guWQCS LEOM1gR0IMzgzBvgRm0QTNEuMSggC0yW6ECgaSdLCx9Fw5d3C4xMX01ejwWysr+LARIM wKZ/brzJM+qm+9Stthj8fOK0KPsHixRF0PwJ4UsMabiosJPpY2wr9Ugw+qYJLnCCOYS8 Zl1jz3Hw402ZvfpjczQ8+fV5xDDTi4JRcYeEex5xhEHjv4If7FBU0gtrfhGJEaibNy3f 0+8R30cKyJTrHyfCf7wXs9NqKLKwsfzW+DLmtgyd9dsnAT+/cbfsS7VqQDf/sNdoE0SW D0qQ== X-Gm-Message-State: ABuFfohd/LoGfA1aF/iDRa57ooZbWe74odEwltQxWXZJi30gnuQ6s1Ey cmC/Q9IrrKYmpaPu1Dkkrv739bWCPT4B X-Google-Smtp-Source: ACcGV63fuQEBSxsk/ie511WDb8o/ZT3s9gvce0ePTXRRuhhqb22G6wCOA9BPteNyr9aFIq2/WZZjqseewwxR X-Received: by 2002:a37:100a:: with SMTP id a10-v6mr18531214qkh.15.1539713622530; Tue, 16 Oct 2018 11:13:42 -0700 (PDT) Date: Tue, 16 Oct 2018 11:13:22 -0700 In-Reply-To: <20181016181327.107186-1-sbeller@google.com> Message-Id: <20181016181327.107186-5-sbeller@google.com> Mime-Version: 1.0 References: <20181016181327.107186-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH 4/9] submodule.c: move global changed_submodule_names into fetch submodule struct From: Stefan Beller To: gitster@pobox.com Cc: git@vger.kernel.org, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The `changed_submodule_names` are only used for fetching, so let's make it part of the struct that is passed around for fetching submodules. Signed-off-by: Stefan Beller Signed-off-by: Junio C Hamano --- submodule.c | 42 +++++++++++++++++++++++------------------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/submodule.c b/submodule.c index 9fbfcfcfe1..6b4cee82bf 100644 --- a/submodule.c +++ b/submodule.c @@ -24,7 +24,7 @@ #include "object-store.h" static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF; -static struct string_list changed_submodule_names = STRING_LIST_INIT_DUP; + static int initialized_fetch_ref_tips; static struct oid_array ref_tips_before_fetch; static struct oid_array ref_tips_after_fetch; @@ -1124,7 +1124,22 @@ void check_for_new_submodule_commits(struct object_id *oid) oid_array_append(&ref_tips_after_fetch, oid); } -static void calculate_changed_submodule_paths(void) +struct submodule_parallel_fetch { + int count; + struct argv_array args; + struct repository *r; + const char *prefix; + int command_line_option; + int default_option; + int quiet; + int result; + + struct string_list changed_submodule_names; +}; +#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0, STRING_LIST_INIT_DUP } + +static void calculate_changed_submodule_paths( + struct submodule_parallel_fetch *spf) { struct argv_array argv = ARGV_ARRAY_INIT; struct string_list changed_submodules = STRING_LIST_INIT_DUP; @@ -1162,7 +1177,8 @@ static void calculate_changed_submodule_paths(void) continue; if (!submodule_has_commits(path, commits)) - string_list_append(&changed_submodule_names, name->string); + string_list_append(&spf->changed_submodule_names, + name->string); } free_submodules_oids(&changed_submodules); @@ -1199,18 +1215,6 @@ int submodule_touches_in_range(struct object_id *excl_oid, return ret; } -struct submodule_parallel_fetch { - int count; - struct argv_array args; - struct repository *r; - const char *prefix; - int command_line_option; - int default_option; - int quiet; - int result; -}; -#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0} - static int get_fetch_recurse_config(const struct submodule *submodule, struct submodule_parallel_fetch *spf) { @@ -1271,7 +1275,7 @@ static int get_next_submodule(struct child_process *cp, case RECURSE_SUBMODULES_ON_DEMAND: if (!submodule || !string_list_lookup( - &changed_submodule_names, + &spf->changed_submodule_names, submodule->name)) continue; default_argv = "on-demand"; @@ -1363,8 +1367,8 @@ int fetch_populated_submodules(struct repository *r, argv_array_push(&spf.args, "--recurse-submodules-default"); /* default value, "--submodule-prefix" and its value are added later */ - calculate_changed_submodule_paths(); - string_list_sort(&changed_submodule_names); + calculate_changed_submodule_paths(&spf); + string_list_sort(&spf.changed_submodule_names); run_processes_parallel(max_parallel_jobs, get_next_submodule, fetch_start_failure, @@ -1373,7 +1377,7 @@ int fetch_populated_submodules(struct repository *r, argv_array_clear(&spf.args); out: - string_list_clear(&changed_submodule_names, 1); + string_list_clear(&spf.changed_submodule_names, 1); return spf.result; } From patchwork Tue Oct 16 18:13:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10644027 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7492915E2 for ; Tue, 16 Oct 2018 18:13:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 684C02A587 for ; Tue, 16 Oct 2018 18:13:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5C8CA2A5F1; Tue, 16 Oct 2018 18:13:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F41012A587 for ; Tue, 16 Oct 2018 18:13:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727294AbeJQCFb (ORCPT ); Tue, 16 Oct 2018 22:05:31 -0400 Received: from mail-ua1-f73.google.com ([209.85.222.73]:43871 "EHLO mail-ua1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727006AbeJQCFb (ORCPT ); Tue, 16 Oct 2018 22:05:31 -0400 Received: by mail-ua1-f73.google.com with SMTP id f10so1652046uan.10 for ; Tue, 16 Oct 2018 11:13:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ZK0RzDWS+tk5X176hsxHIhH9aiBzhWKC4hPlHVauUH0=; b=AbYB1+D0Tf9HbeJhKFciQ6aBGuXhdESY/SxveHcqq08MLcBD0/rVgpCibs5FwhrdsZ FKg/nRDbabro3tjSzw183feC220BKokLDIQdIvyn8Ngw9gwOsHGXDWQmHa+h1WwnDHHB AA82tGj3Q7Je99TMmG7FInByTPTzk1gQH0pC+KZDkPvrt6mwyY9ifuSNMsHW0b7inOBx LL/x7oRe1eHD8IxV0taOhstbfb1whRwxeAGlKHghOIPPRot2y/VJ+mTNODR/sX5NbEG1 kHYPOpN4G9LVVe3rqHQfqvDIyBECsT7Av5uks+tbCzntv5Yibxxm2M8acXfXRnL1pRNS Gm/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ZK0RzDWS+tk5X176hsxHIhH9aiBzhWKC4hPlHVauUH0=; b=SFqik8FP3LkvSZjxoWIoPN+sFgFoupJQcAH8YbrASyUqMPEEuHCulKJK7F9bx948VW wqJ5eGKjVZVz46yBk+oahUh5FTog9bF4NUpM+PC4cp40eBt/bCxGg9rb4tyW6+VObvkM zfezQ8IZ6wpI/hBt0FzUlRF0V7NBcPhOR6L7w1SOzL17JkYzNzSp6zlRlfK7OR+nWLi8 iqDITJERCY1dRrFLnsYjVnX6FKNTMCui2SQW8OR8z5Sync6KYupD0Z0CQrGNjlVtPDa8 NeF6AHQYKimdgX+s5gVA6/GgB0mNCwQbte9yrl+VUJtXSKcnlKRRGEDkSWJociZ9GBlb ZvLg== X-Gm-Message-State: ABuFfogonXB2QBg0ELtsm4Pp+pss/OxR/VwLIhMhnYpF67gUxzNsLjL8 4g+5nACZAvRrOrIQy0llfK4ASk5O1xwp X-Google-Smtp-Source: ACcGV61PdcDhNobaY0Ky9xm9gmzxl2oglmbAxgQfLW+Qpvs0Au7bhEUrx4ihV2KIsDd3emhV8i2AgU27RW/1 X-Received: by 2002:a1f:2c49:: with SMTP id s70-v6mr20690353vks.7.1539713631285; Tue, 16 Oct 2018 11:13:51 -0700 (PDT) Date: Tue, 16 Oct 2018 11:13:23 -0700 In-Reply-To: <20181016181327.107186-1-sbeller@google.com> Message-Id: <20181016181327.107186-6-sbeller@google.com> Mime-Version: 1.0 References: <20181016181327.107186-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH 5/9] submodule.c: do not copy around submodule list From: Stefan Beller To: gitster@pobox.com Cc: git@vger.kernel.org, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP 'calculate_changed_submodule_paths' uses a local list to compute the changed submodules, and then produces the result by copying appropriate items into the result list. Instead use the result list directly and prune items afterwards using string_list_remove_empty_items. By doing so we'll have access to the util pointer for longer that contains the commits that we need to fetch, which will be useful in a later patch. Signed-off-by: Stefan Beller Signed-off-by: Junio C Hamano --- submodule.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/submodule.c b/submodule.c index 6b4cee82bf..cbefe5f54d 100644 --- a/submodule.c +++ b/submodule.c @@ -1142,8 +1142,7 @@ static void calculate_changed_submodule_paths( struct submodule_parallel_fetch *spf) { struct argv_array argv = ARGV_ARRAY_INIT; - struct string_list changed_submodules = STRING_LIST_INIT_DUP; - const struct string_list_item *name; + struct string_list_item *name; /* No need to check if there are no submodules configured */ if (!submodule_from_path(the_repository, NULL, NULL)) @@ -1160,9 +1159,9 @@ static void calculate_changed_submodule_paths( * Collect all submodules (whether checked out or not) for which new * commits have been recorded upstream in "changed_submodule_names". */ - collect_changed_submodules(&changed_submodules, &argv); + collect_changed_submodules(&spf->changed_submodule_names, &argv); - for_each_string_list_item(name, &changed_submodules) { + for_each_string_list_item(name, &spf->changed_submodule_names) { struct oid_array *commits = name->util; const struct submodule *submodule; const char *path = NULL; @@ -1176,12 +1175,14 @@ static void calculate_changed_submodule_paths( if (!path) continue; - if (!submodule_has_commits(path, commits)) - string_list_append(&spf->changed_submodule_names, - name->string); + if (submodule_has_commits(path, commits)) { + oid_array_clear(commits); + *name->string = '\0'; + } } - free_submodules_oids(&changed_submodules); + string_list_remove_empty_items(&spf->changed_submodule_names, 1); + argv_array_clear(&argv); oid_array_clear(&ref_tips_before_fetch); oid_array_clear(&ref_tips_after_fetch); @@ -1377,7 +1378,7 @@ int fetch_populated_submodules(struct repository *r, argv_array_clear(&spf.args); out: - string_list_clear(&spf.changed_submodule_names, 1); + free_submodules_oids(&spf.changed_submodule_names); return spf.result; } From patchwork Tue Oct 16 18:13:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10644033 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 43589109C for ; Tue, 16 Oct 2018 18:14:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 36EB82A587 for ; Tue, 16 Oct 2018 18:14:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2B1952A5F9; Tue, 16 Oct 2018 18:14:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7B1412A587 for ; Tue, 16 Oct 2018 18:14:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727330AbeJQCFk (ORCPT ); Tue, 16 Oct 2018 22:05:40 -0400 Received: from mail-io1-f73.google.com ([209.85.166.73]:48574 "EHLO mail-io1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727006AbeJQCFj (ORCPT ); Tue, 16 Oct 2018 22:05:39 -0400 Received: by mail-io1-f73.google.com with SMTP id v12-v6so22359031iob.15 for ; Tue, 16 Oct 2018 11:14:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=eKZosNxwNfmoclJwZLY9ORZilz4Ns8/QFzalhLMUAAs=; b=tjVRzI//rXF4N29OiYQS4Ev8+e3DpMpj3NwQUnzL3FRNMmo61IRPbAy2JC/JXz14ZB BvH4wmTb0/K2T6a+9z9da33rXUuxiIPD02wSdNpviPUu5QDpmZIOe3nfZ9+pIt0B7jzN 6WaFPQ3dJsuVGng04icOez8TqH7OGS5QKs1DeVRWvY5X82LZRMBe7bpoYTBofaS87HuB pCGEUFrfBFxs3+KnEnUu/P2rYE9PGBWc3iKAudfTIk9wL60edCDrnkkmuKUcsS0uTSXX yT8A64lLmnL8jsn6dJXckS/7c92wYenyRKQUlpG1BakKznXRA8yVcyCzk/A6rIegoIpp GR+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=eKZosNxwNfmoclJwZLY9ORZilz4Ns8/QFzalhLMUAAs=; b=HVolLpl/XjbQ9rcncbup7xTO/TAEk9tfXavNTR+w/ZiKSx9vjLzL1LiZhBpJ4WIjPA dMhKdG2wS9jnxt9Qg7l8MmwfQ2HJV5MKVaigG9hKeRLVdEtIy5fp/oWCjMb2cy+p8hfE RmboSlJI6iVrMt0LKroeDapWk1n6gztfm7Z6YqW2PS6p3Efg3ealc+D6Wm3ltA4fUswT 8Q8GiyEqkGrvsOcsxYKxgqW41dQRcs+TbV4rLuuzDZKyEMqFX5+yoVO47XM/OlcKyLdB bRIjv+4SRsWbOWoj5qglPEtfwxw6zeSfYGOQjhOjaiqQUsTH/ekOys2x3YHzV+Spsm79 XzzQ== X-Gm-Message-State: ABuFfohNuxXqgtkYBX7yGrkRA2eJw6w8xgZXHYosGp0851pK06lxwAMR 1sZAZrwZKinSWr6l8AJI3H7BadyTwZqG X-Google-Smtp-Source: ACcGV625ktWiwLU0U3WuBD4WgTJsvF4k25FRLCX08GvpcElkLwpPo5yMB3PjNKauFYPZkphQEiSi5CaEp4oi X-Received: by 2002:a24:6cc2:: with SMTP id w185-v6mr16037836itb.5.1539713640103; Tue, 16 Oct 2018 11:14:00 -0700 (PDT) Date: Tue, 16 Oct 2018 11:13:24 -0700 In-Reply-To: <20181016181327.107186-1-sbeller@google.com> Message-Id: <20181016181327.107186-7-sbeller@google.com> Mime-Version: 1.0 References: <20181016181327.107186-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH 6/9] repository: repo_submodule_init to take a submodule struct From: Stefan Beller To: gitster@pobox.com Cc: git@vger.kernel.org, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When constructing a struct repository for a submodule for some revision of the superproject where the submodule is not contained in the index, it may not be present in the working tree currently either. In that siutation giving a 'path' argument is not useful. Upgrade the repo_submodule_init function to take a struct submodule instead. While we are at it, overhaul the repo_submodule_init function by renaming the submodule repository struct, which is to be initialized, to a name that is not confused with the struct submodule as easily. Also move its documentation into the header file. Signed-off-by: Stefan Beller --- builtin/grep.c | 17 +++++++----- builtin/ls-files.c | 12 +++++---- repository.c | 27 ++++++++------------ repository.h | 11 ++++++-- t/helper/test-submodule-nested-repo-config.c | 8 +++--- 5 files changed, 41 insertions(+), 34 deletions(-) diff --git a/builtin/grep.c b/builtin/grep.c index 7da8fef31a..ba7634258a 100644 --- a/builtin/grep.c +++ b/builtin/grep.c @@ -418,7 +418,10 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, const struct object_id *oid, const char *filename, const char *path) { - struct repository submodule; + struct repository subrepo; + const struct submodule *sub = submodule_from_path(superproject, + &null_oid, path); + int hit; /* @@ -434,12 +437,12 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, return 0; } - if (repo_submodule_init(&submodule, superproject, path)) { + if (repo_submodule_init(&subrepo, superproject, sub)) { grep_read_unlock(); return 0; } - repo_read_gitmodules(&submodule); + repo_read_gitmodules(&subrepo); /* * NEEDSWORK: This adds the submodule's object directory to the list of @@ -451,7 +454,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, * store is no longer global and instead is a member of the repository * object. */ - add_to_alternates_memory(submodule.objects->objectdir); + add_to_alternates_memory(subrepo.objects->objectdir); grep_read_unlock(); if (oid) { @@ -476,14 +479,14 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, init_tree_desc(&tree, data, size); hit = grep_tree(opt, pathspec, &tree, &base, base.len, - object->type == OBJ_COMMIT, &submodule); + object->type == OBJ_COMMIT, &subrepo); strbuf_release(&base); free(data); } else { - hit = grep_cache(opt, &submodule, pathspec, 1); + hit = grep_cache(opt, &subrepo, pathspec, 1); } - repo_clear(&submodule); + repo_clear(&subrepo); return hit; } diff --git a/builtin/ls-files.c b/builtin/ls-files.c index 7f9919a362..4d1649c1b3 100644 --- a/builtin/ls-files.c +++ b/builtin/ls-files.c @@ -206,17 +206,19 @@ static void show_files(struct repository *repo, struct dir_struct *dir); static void show_submodule(struct repository *superproject, struct dir_struct *dir, const char *path) { - struct repository submodule; + struct repository subrepo; + const struct submodule *sub = submodule_from_path(superproject, + &null_oid, path); - if (repo_submodule_init(&submodule, superproject, path)) + if (repo_submodule_init(&subrepo, superproject, sub)) return; - if (repo_read_index(&submodule) < 0) + if (repo_read_index(&subrepo) < 0) die("index file corrupt"); - show_files(&submodule, dir); + show_files(&subrepo, dir); - repo_clear(&submodule); + repo_clear(&subrepo); } static void show_ce(struct repository *repo, struct dir_struct *dir, diff --git a/repository.c b/repository.c index 5dd1486718..aabe64ee5d 100644 --- a/repository.c +++ b/repository.c @@ -166,30 +166,23 @@ int repo_init(struct repository *repo, return -1; } -/* - * Initialize 'submodule' as the submodule given by 'path' in parent repository - * 'superproject'. - * Return 0 upon success and a non-zero value upon failure. - */ -int repo_submodule_init(struct repository *submodule, +int repo_submodule_init(struct repository *subrepo, struct repository *superproject, - const char *path) + const struct submodule *sub) { - const struct submodule *sub; struct strbuf gitdir = STRBUF_INIT; struct strbuf worktree = STRBUF_INIT; int ret = 0; - sub = submodule_from_path(superproject, &null_oid, path); if (!sub) { ret = -1; goto out; } - strbuf_repo_worktree_path(&gitdir, superproject, "%s/.git", path); - strbuf_repo_worktree_path(&worktree, superproject, "%s", path); + strbuf_repo_worktree_path(&gitdir, superproject, "%s/.git", sub->path); + strbuf_repo_worktree_path(&worktree, superproject, "%s", sub->path); - if (repo_init(submodule, gitdir.buf, worktree.buf)) { + if (repo_init(subrepo, gitdir.buf, worktree.buf)) { /* * If initilization fails then it may be due to the submodule * not being populated in the superproject's worktree. Instead @@ -201,16 +194,16 @@ int repo_submodule_init(struct repository *submodule, strbuf_repo_git_path(&gitdir, superproject, "modules/%s", sub->name); - if (repo_init(submodule, gitdir.buf, NULL)) { + if (repo_init(subrepo, gitdir.buf, NULL)) { ret = -1; goto out; } } - submodule->submodule_prefix = xstrfmt("%s%s/", - superproject->submodule_prefix ? - superproject->submodule_prefix : - "", path); + subrepo->submodule_prefix = xstrfmt("%s%s/", + superproject->submodule_prefix ? + superproject->submodule_prefix : + "", sub->path); out: strbuf_release(&gitdir); diff --git a/repository.h b/repository.h index 9f16c42c1e..a9c7a5baa5 100644 --- a/repository.h +++ b/repository.h @@ -116,9 +116,16 @@ void repo_set_worktree(struct repository *repo, const char *path); void repo_set_hash_algo(struct repository *repo, int algo); void initialize_the_repository(void); int repo_init(struct repository *r, const char *gitdir, const char *worktree); -int repo_submodule_init(struct repository *submodule, + +/* + * Initialize the repository 'subrepo' as the submodule given by the + * struct submodule 'sub' in parent repository 'superproject'. + * Return 0 upon success and a non-zero value upon failure. + */ +struct submodule; +int repo_submodule_init(struct repository *subrepo, struct repository *superproject, - const char *path); + const struct submodule *sub); void repo_clear(struct repository *repo); /* diff --git a/t/helper/test-submodule-nested-repo-config.c b/t/helper/test-submodule-nested-repo-config.c index a31e2a9bea..bc97929bbc 100644 --- a/t/helper/test-submodule-nested-repo-config.c +++ b/t/helper/test-submodule-nested-repo-config.c @@ -10,19 +10,21 @@ static void die_usage(int argc, const char **argv, const char *msg) int cmd__submodule_nested_repo_config(int argc, const char **argv) { - struct repository submodule; + struct repository subrepo; + const struct submodule *sub; if (argc < 3) die_usage(argc, argv, "Wrong number of arguments."); setup_git_directory(); - if (repo_submodule_init(&submodule, the_repository, argv[1])) { + sub = submodule_from_path(the_repository, &null_oid, argv[1]); + if (repo_submodule_init(&subrepo, the_repository, sub)) { die_usage(argc, argv, "Submodule not found."); } /* Read the config of _child_ submodules. */ - print_config_from_gitmodules(&submodule, argv[2]); + print_config_from_gitmodules(&subrepo, argv[2]); submodule_free(the_repository); From patchwork Tue Oct 16 18:13:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10644041 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 767CD15E2 for ; Tue, 16 Oct 2018 18:14:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6A3C32A587 for ; Tue, 16 Oct 2018 18:14:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5E4CB2A5F1; Tue, 16 Oct 2018 18:14:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D66642A587 for ; Tue, 16 Oct 2018 18:14:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727392AbeJQCFt (ORCPT ); Tue, 16 Oct 2018 22:05:49 -0400 Received: from mail-it1-f202.google.com ([209.85.166.202]:38790 "EHLO mail-it1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727006AbeJQCFt (ORCPT ); Tue, 16 Oct 2018 22:05:49 -0400 Received: by mail-it1-f202.google.com with SMTP id d10so26860754itk.3 for ; Tue, 16 Oct 2018 11:14:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=cwEaVOQsZKeLZkIk0ya0bf1ynrWZWRCKcmUMzgnv1dc=; b=gELAxh+6BFQrzX2XKmAcAU8tePVnMlzt0vRtu1R9sbYcMZwu4r6X6VLRnyypn5W7De uR5HqawZyZuMcyK7ouAdVZecSY8y2Q08AZBEbuuKaDMZxhkv5etJsosv2J6EZfse48g5 gISYa8dWEIHR7ja1zPtaArg+fIvqzuhij6crgw0AzUHvkye+7xv2ex/apBeRIX2TPx6R CroF47rsBJ9s9I74jU5vuEdRoo3USwIeD+lPgi+8zAU8YiZV9UUZZVqMvCnIdPbzaVw9 sWPXZ4GxtheSKPtBT3fNpiYdSyYi+yufuutwB0Xa+IGGMD8ge1/qsocPfOhT9x1V5Los X4Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=cwEaVOQsZKeLZkIk0ya0bf1ynrWZWRCKcmUMzgnv1dc=; b=lY7f18SqbhkExo38MJCrCAxFi2Zl0nGd2S9t05uJdqof+gLBecFrEfhACNUjaZk9ms EmkU69cher7GrJe03zVlwEmchaq7Y7bJJrJE99AMatMSYZ7xS+jtJSC0GRg9Fynru479 BgGWDjCv7Npeig5NnrrAA2qJzGZUiXNwYOo/yeoIGs2EFRsWylLMFSDM7enSwjqCFnYm rx3itjAUlaFzXbfFOV2k0j9MM0Nvl85iqEciaAKD87tDq7WlbLjmLpB8bphD+RiD4V+G +OlFQGreDXV4XyxPB1PHjqnwHTPl3wR1IPfENhvk46LXfwO5IyStIHtIOKcYDax8ukXi 0UsQ== X-Gm-Message-State: ABuFfogKVmlSoVQVXAUern/jxXUIKMdAvOHzTYf6skpX4pVgy5zuNZdf 9+p3SO/NHhEbBkGNbY9lsf48dFp/4Q9B X-Google-Smtp-Source: ACcGV62lDM5CoAmXiCN51Z5IivJr+sQ/KbZAOy4jM87ynX+B/Ru70uyj2UELlb2YccXzlusfEYea5V8QeHjp X-Received: by 2002:a24:8903:: with SMTP id s3-v6mr17738771itd.26.1539713648969; Tue, 16 Oct 2018 11:14:08 -0700 (PDT) Date: Tue, 16 Oct 2018 11:13:25 -0700 In-Reply-To: <20181016181327.107186-1-sbeller@google.com> Message-Id: <20181016181327.107186-8-sbeller@google.com> Mime-Version: 1.0 References: <20181016181327.107186-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH 7/9] submodule: fetch in submodules git directory instead of in worktree From: Stefan Beller To: gitster@pobox.com Cc: git@vger.kernel.org, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch started as a refactoring to make 'get_next_submodule' more readable, but upon doing so, I realized that "git fetch" of the submodule actually doesn't need to be run in the submodules worktree. So let's run it in its git dir instead. That should pave the way towards fetching submodules that are currently not checked out. This patch leaks the cp->dir in get_next_submodule, as any further callback in run_processes_parallel doesn't have access to the child process any more. In an early iteration of this patch, the function get_submodule_repo_for directly returned the string containing the git directory, which would be a better design choice for this patch. However the next patch both fixes the memory leak of cp->dir and also has a use case for using the full repository handle of the submodule, so it makes sense to introduce the get_submodule_repo_for here already. Signed-off-by: Stefan Beller Signed-off-by: Junio C Hamano --- submodule.c | 51 +++++++++++++++++++++++++++---------- t/t5526-fetch-submodules.sh | 7 ++++- 2 files changed, 44 insertions(+), 14 deletions(-) diff --git a/submodule.c b/submodule.c index cbefe5f54d..30c06507e3 100644 --- a/submodule.c +++ b/submodule.c @@ -495,6 +495,12 @@ void prepare_submodule_repo_env(struct argv_array *out) DEFAULT_GIT_DIR_ENVIRONMENT); } +static void prepare_submodule_repo_env_in_gitdir(struct argv_array *out) +{ + prepare_submodule_repo_env_no_git_dir(out); + argv_array_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT); +} + /* Helper function to display the submodule header line prior to the full * summary output. If it can locate the submodule objects directory it will * attempt to lookup both the left and right commits and put them into the @@ -1241,6 +1247,29 @@ static int get_fetch_recurse_config(const struct submodule *submodule, return spf->default_option; } +static struct repository *get_submodule_repo_for(struct repository *r, + const struct submodule *sub) +{ + struct repository *ret = xmalloc(sizeof(*ret)); + + if (repo_submodule_init(ret, r, sub)) { + /* + * No entry in .gitmodules? Technically not a submodule, + * but historically we supported repositories that happen to be + * in-place where a gitlink is. Keep supporting them. + */ + struct strbuf gitdir = STRBUF_INIT; + strbuf_repo_worktree_path(&gitdir, r, "%s/.git", sub->path); + if (repo_init(ret, gitdir.buf, NULL)) { + strbuf_release(&gitdir); + return NULL; + } + strbuf_release(&gitdir); + } + + return ret; +} + static int get_next_submodule(struct child_process *cp, struct strbuf *err, void *data, void **task_cb) { @@ -1248,12 +1277,11 @@ static int get_next_submodule(struct child_process *cp, struct submodule_parallel_fetch *spf = data; for (; spf->count < spf->r->index->cache_nr; spf->count++) { - struct strbuf submodule_path = STRBUF_INIT; - struct strbuf submodule_git_dir = STRBUF_INIT; struct strbuf submodule_prefix = STRBUF_INIT; const struct cache_entry *ce = spf->r->index->cache[spf->count]; - const char *git_dir, *default_argv; + const char *default_argv; const struct submodule *submodule; + struct repository *repo; struct submodule default_submodule = SUBMODULE_INIT; if (!S_ISGITLINK(ce->ce_mode)) @@ -1288,16 +1316,12 @@ static int get_next_submodule(struct child_process *cp, continue; } - strbuf_repo_worktree_path(&submodule_path, spf->r, "%s", ce->name); - strbuf_addf(&submodule_git_dir, "%s/.git", submodule_path.buf); strbuf_addf(&submodule_prefix, "%s%s/", spf->prefix, ce->name); - git_dir = read_gitfile(submodule_git_dir.buf); - if (!git_dir) - git_dir = submodule_git_dir.buf; - if (is_directory(git_dir)) { + repo = get_submodule_repo_for(spf->r, submodule); + if (repo) { child_process_init(cp); - cp->dir = strbuf_detach(&submodule_path, NULL); - prepare_submodule_repo_env(&cp->env_array); + prepare_submodule_repo_env_in_gitdir(&cp->env_array); + cp->dir = xstrdup(repo->gitdir); cp->git_cmd = 1; if (!spf->quiet) strbuf_addf(err, "Fetching submodule %s%s\n", @@ -1307,10 +1331,11 @@ static int get_next_submodule(struct child_process *cp, argv_array_push(&cp->args, default_argv); argv_array_push(&cp->args, "--submodule-prefix"); argv_array_push(&cp->args, submodule_prefix.buf); + + repo_clear(repo); + free(repo); ret = 1; } - strbuf_release(&submodule_path); - strbuf_release(&submodule_git_dir); strbuf_release(&submodule_prefix); if (ret) { spf->count++; diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh index 6c2f9b2ba2..42692219a1 100755 --- a/t/t5526-fetch-submodules.sh +++ b/t/t5526-fetch-submodules.sh @@ -566,7 +566,12 @@ test_expect_success 'fetching submodule into a broken repository' ' test_must_fail git -C dst status && test_must_fail git -C dst diff && - test_must_fail git -C dst fetch --recurse-submodules + + # git-fetch cannot find the git directory of the submodule, + # so it will do nothing, successfully, as it cannot distinguish between + # this broken submodule and a submodule that was just set active but + # not cloned yet + git -C dst fetch --recurse-submodules ' test_expect_success "fetch new commits when submodule got renamed" ' From patchwork Tue Oct 16 18:13:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10644043 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 53E4015E2 for ; Tue, 16 Oct 2018 18:14:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 43EDD2A587 for ; Tue, 16 Oct 2018 18:14:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 377862A5F1; Tue, 16 Oct 2018 18:14:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 62CCC2A587 for ; Tue, 16 Oct 2018 18:14:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727398AbeJQCF5 (ORCPT ); Tue, 16 Oct 2018 22:05:57 -0400 Received: from mail-qk1-f202.google.com ([209.85.222.202]:43773 "EHLO mail-qk1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727374AbeJQCF5 (ORCPT ); Tue, 16 Oct 2018 22:05:57 -0400 Received: by mail-qk1-f202.google.com with SMTP id n64-v6so24521507qkd.10 for ; Tue, 16 Oct 2018 11:14:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=2x1aCinTAd7uZZxZ5ghUhVczgLCHwB3MUQ5qGGy6Nk8=; b=Rfqg0EEcoelYl+IcYM+hpBX03YTxKRfeIktgmYqUAdx++WfrTYNbMPaP83HMf0txfF 2JdpRzzvxP9/NhAPLCYYroFqhjcoE5l7CjJz21Bqkm/y/IGck0blVSMGh99M/S5RIoQQ FtvWegt9Wzrdh02AxdLTZTYwdowkoBduAzYRxHuLh6ZqCwh2/DWRO8Al82v8u5QpbFy7 jurHKL4AZqfiqPM/j+wK7bujtBxHHyfd7HuBzDJEDXRuG5G4tzWd9JXHnKE4hDQjMs8p Fi2tZPLWobWtO5p1qTACFStVttTEP+ZThElB3B6kh3mpG9BEa3tV5tCJJR5/lkaJ5ony jNFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=2x1aCinTAd7uZZxZ5ghUhVczgLCHwB3MUQ5qGGy6Nk8=; b=I9A4X8EAZCefKrVnXmcyQ/zM1jBfLblcTrvjZWE+MlVTpp7cS6zQoY3j2vk4k2iIAf rjvb4XqNVHRfh9er9ExiSyl60ykBfSaZl2swP6haChLkmslhlK3inbEW4qeobl2kpL/Y zYLOzUKJkOKzcDoAgudl4Osh/aaT26H8VJGCD8ssJxQXViqblplE68WxT/NTizWY3MsX itnp0e0lvZoAnisJIR2tv3UL1kDE4Ex27UaZbK6Vcd1e9X5NOcqvE4oRtOpsMMgWgt2z bUrvF8AW3eTBG2Bzva8TenjUn76fM1tyLp6xbTauLSw8/d+URj9HHY7raR1Rv4zAnGBP Trqg== X-Gm-Message-State: ABuFfohlPwzUJdlTUan6zyIqEVwE62xtYkKyuFfuGpuNjkWA75frmPcc /RuqB3x7poQOwYHwyNkZ3hB8WX+SCwjI X-Google-Smtp-Source: ACcGV60F2W98TCUdqWG3D7f+RK+GDjy10f+bK0MGBA5TTBKvPkVLUAF6swayiau9fwLln3Tl2LseKNQsIQag X-Received: by 2002:a37:4ed5:: with SMTP id c204-v6mr17925726qkb.24.1539713657647; Tue, 16 Oct 2018 11:14:17 -0700 (PDT) Date: Tue, 16 Oct 2018 11:13:26 -0700 In-Reply-To: <20181016181327.107186-1-sbeller@google.com> Message-Id: <20181016181327.107186-9-sbeller@google.com> Mime-Version: 1.0 References: <20181016181327.107186-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH 8/9] fetch: retry fetching submodules if needed objects were not fetched From: Stefan Beller To: gitster@pobox.com Cc: git@vger.kernel.org, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Currently when git-fetch is asked to recurse into submodules, it dispatches a plain "git-fetch -C " (with some submodule related options such as prefix and recusing strategy, but) without any information of the remote or the tip that should be fetched. This works surprisingly well in some workflows (such as using submodules as a third party library), while not so well in other scenarios, such as in a Gerrit topic-based workflow, that can tie together changes (potentially across repositories) on the server side. One of the parts of such a Gerrit workflow is to download a change when wanting to examine it, and you'd want to have its submodule changes that are in the same topic downloaded as well. However these submodule changes reside in their own repository in their own ref (refs/changes/). Retry fetching a submodule by object name if the object id that the superproject points to, cannot be found. This retrying does not happen when the "git fetch" done at the superproject is not storing the fetched results in remote tracking branches (i.e. instead just recording them to FETCH_HEAD) in this step. A later patch will fix this. builtin/fetch used to only inspect submodules when they were fetched "on-demand", as in either on/off case it was clear whether the submodule needs to be fetched. However to know whether we need to try fetching the object ids, we need to identify the object names, which is done in this function check_for_new_submodule_commits(), so we'll also run that code in case the submodule recursion is set to "on". Signed-off-by: Stefan Beller Signed-off-by: Junio C Hamano --- builtin/fetch.c | 9 +- submodule.c | 185 ++++++++++++++++++++++++++++++------ t/t5526-fetch-submodules.sh | 16 ++++ 3 files changed, 177 insertions(+), 33 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 61bec5d213..95c44bf6ff 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -700,8 +700,7 @@ static int update_local_ref(struct ref *ref, what = _("[new ref]"); } - if ((recurse_submodules != RECURSE_SUBMODULES_OFF) && - (recurse_submodules != RECURSE_SUBMODULES_ON)) + if (recurse_submodules != RECURSE_SUBMODULES_OFF) check_for_new_submodule_commits(&ref->new_oid); r = s_update_ref(msg, ref, 0); format_display(display, r ? '!' : '*', what, @@ -716,8 +715,7 @@ static int update_local_ref(struct ref *ref, strbuf_add_unique_abbrev(&quickref, ¤t->object.oid, DEFAULT_ABBREV); strbuf_addstr(&quickref, ".."); strbuf_add_unique_abbrev(&quickref, &ref->new_oid, DEFAULT_ABBREV); - if ((recurse_submodules != RECURSE_SUBMODULES_OFF) && - (recurse_submodules != RECURSE_SUBMODULES_ON)) + if (recurse_submodules != RECURSE_SUBMODULES_OFF) check_for_new_submodule_commits(&ref->new_oid); r = s_update_ref("fast-forward", ref, 1); format_display(display, r ? '!' : ' ', quickref.buf, @@ -731,8 +729,7 @@ static int update_local_ref(struct ref *ref, strbuf_add_unique_abbrev(&quickref, ¤t->object.oid, DEFAULT_ABBREV); strbuf_addstr(&quickref, "..."); strbuf_add_unique_abbrev(&quickref, &ref->new_oid, DEFAULT_ABBREV); - if ((recurse_submodules != RECURSE_SUBMODULES_OFF) && - (recurse_submodules != RECURSE_SUBMODULES_ON)) + if (recurse_submodules != RECURSE_SUBMODULES_OFF) check_for_new_submodule_commits(&ref->new_oid); r = s_update_ref("forced-update", ref, 1); format_display(display, r ? '!' : '+', quickref.buf, diff --git a/submodule.c b/submodule.c index 30c06507e3..7246b776f3 100644 --- a/submodule.c +++ b/submodule.c @@ -1141,8 +1141,12 @@ struct submodule_parallel_fetch { int result; struct string_list changed_submodule_names; + struct get_next_submodule_task **retry; + int retry_nr, retry_alloc; }; -#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0, STRING_LIST_INIT_DUP } +#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0, \ + STRING_LIST_INIT_DUP, \ + NULL, 0, 0} static void calculate_changed_submodule_paths( struct submodule_parallel_fetch *spf) @@ -1247,6 +1251,56 @@ static int get_fetch_recurse_config(const struct submodule *submodule, return spf->default_option; } +struct get_next_submodule_task { + struct repository *repo; + const struct submodule *sub; + unsigned free_sub : 1; /* Do we need to free the submodule? */ + struct oid_array *commits; +}; + +static const struct submodule *get_default_submodule(const char *path) +{ + struct submodule *ret = NULL; + const char *name = default_name_or_path(path); + + if (!name) + return NULL; + + ret = xmalloc(sizeof(*ret)); + memset(ret, 0, sizeof(*ret)); + ret->path = name; + ret->name = name; + + return (const struct submodule *) ret; +} + +static struct get_next_submodule_task *get_next_submodule_task_create( + struct repository *r, const char *path) +{ + struct get_next_submodule_task *task = xmalloc(sizeof(*task)); + memset(task, 0, sizeof(*task)); + + task->sub = submodule_from_path(r, &null_oid, path); + if (!task->sub) { + task->sub = get_default_submodule(path); + task->free_sub = 1; + } + + return task; +} + +static void get_next_submodule_task_release(struct get_next_submodule_task *p) +{ + if (p->free_sub) + free((void*)p->sub); + p->free_sub = 0; + p->sub = NULL; + + if (p->repo) + repo_clear(p->repo); + FREE_AND_NULL(p->repo); +} + static struct repository *get_submodule_repo_for(struct repository *r, const struct submodule *sub) { @@ -1273,39 +1327,35 @@ static struct repository *get_submodule_repo_for(struct repository *r, static int get_next_submodule(struct child_process *cp, struct strbuf *err, void *data, void **task_cb) { - int ret = 0; struct submodule_parallel_fetch *spf = data; for (; spf->count < spf->r->index->cache_nr; spf->count++) { - struct strbuf submodule_prefix = STRBUF_INIT; + int recurse_config; const struct cache_entry *ce = spf->r->index->cache[spf->count]; const char *default_argv; - const struct submodule *submodule; - struct repository *repo; - struct submodule default_submodule = SUBMODULE_INIT; + struct get_next_submodule_task *task; if (!S_ISGITLINK(ce->ce_mode)) continue; - submodule = submodule_from_path(spf->r, &null_oid, ce->name); - if (!submodule) { - const char *name = default_name_or_path(ce->name); - if (name) { - default_submodule.path = name; - default_submodule.name = name; - submodule = &default_submodule; - } + task = get_next_submodule_task_create(spf->r, ce->name); + + if (!task->sub) { + free(task); + continue; } - switch (get_fetch_recurse_config(submodule, spf)) + recurse_config = get_fetch_recurse_config(task->sub, spf); + + switch (recurse_config) { default: case RECURSE_SUBMODULES_DEFAULT: case RECURSE_SUBMODULES_ON_DEMAND: - if (!submodule || + if (!task->sub || !string_list_lookup( &spf->changed_submodule_names, - submodule->name)) + task->sub->name)) continue; default_argv = "on-demand"; break; @@ -1316,12 +1366,12 @@ static int get_next_submodule(struct child_process *cp, continue; } - strbuf_addf(&submodule_prefix, "%s%s/", spf->prefix, ce->name); - repo = get_submodule_repo_for(spf->r, submodule); - if (repo) { + task->repo = get_submodule_repo_for(spf->r, task->sub); + if (task->repo) { + struct strbuf submodule_prefix = STRBUF_INIT; child_process_init(cp); prepare_submodule_repo_env_in_gitdir(&cp->env_array); - cp->dir = xstrdup(repo->gitdir); + cp->dir = task->repo->gitdir; cp->git_cmd = 1; if (!spf->quiet) strbuf_addf(err, "Fetching submodule %s%s\n", @@ -1330,18 +1380,51 @@ static int get_next_submodule(struct child_process *cp, argv_array_pushv(&cp->args, spf->args.argv); argv_array_push(&cp->args, default_argv); argv_array_push(&cp->args, "--submodule-prefix"); + + strbuf_addf(&submodule_prefix, "%s%s/", + spf->prefix, + task->sub->path); argv_array_push(&cp->args, submodule_prefix.buf); - repo_clear(repo); - free(repo); - ret = 1; - } - strbuf_release(&submodule_prefix); - if (ret) { spf->count++; + *task_cb = task; + + strbuf_release(&submodule_prefix); return 1; + } else { + get_next_submodule_task_release(task); + free(task); } } + + if (spf->retry_nr) { + struct get_next_submodule_task *task = spf->retry[spf->retry_nr - 1]; + struct strbuf submodule_prefix = STRBUF_INIT; + spf->retry_nr--; + + strbuf_addf(&submodule_prefix, "%s%s/", spf->prefix, task->sub->path); + + child_process_init(cp); + prepare_submodule_repo_env_in_gitdir(&cp->env_array); + cp->git_cmd = 1; + cp->dir = task->repo->gitdir; + + argv_array_init(&cp->args); + argv_array_pushv(&cp->args, spf->args.argv); + argv_array_push(&cp->args, "on-demand"); + argv_array_push(&cp->args, "--submodule-prefix"); + argv_array_push(&cp->args, submodule_prefix.buf); + + /* NEEDSWORK: have get_default_remote from s--h */ + argv_array_push(&cp->args, "origin"); + oid_array_for_each_unique(task->commits, + append_oid_to_argv, &cp->args); + + *task_cb = task; + strbuf_release(&submodule_prefix); + return 1; + } + return 0; } @@ -1349,20 +1432,68 @@ static int fetch_start_failure(struct strbuf *err, void *cb, void *task_cb) { struct submodule_parallel_fetch *spf = cb; + struct get_next_submodule_task *task = task_cb; spf->result = 1; + get_next_submodule_task_release(task); return 0; } +static int commit_exists_in_sub(const struct object_id *oid, void *data) +{ + struct repository *subrepo = data; + + enum object_type type = oid_object_info(subrepo, oid, NULL); + + return type != OBJ_COMMIT; +} + static int fetch_finish(int retvalue, struct strbuf *err, void *cb, void *task_cb) { struct submodule_parallel_fetch *spf = cb; + struct get_next_submodule_task *task = task_cb; + const struct submodule *sub; + + struct string_list_item *it; + struct oid_array *commits; if (retvalue) spf->result = 1; + if (!task) + return 0; + + sub = task->sub; + if (!sub) + goto out; + + it = string_list_lookup(&spf->changed_submodule_names, sub->name); + if (!it) + goto out; + + commits = it->util; + oid_array_filter(commits, + commit_exists_in_sub, + task->repo); + + /* Are there commits that do not exist? */ + if (commits->nr) { + /* We already tried fetching them, do not try again. */ + if (task->commits) + return 0; + + task->commits = commits; + ALLOC_GROW(spf->retry, spf->retry_nr + 1, spf->retry_alloc); + spf->retry[spf->retry_nr] = task; + spf->retry_nr++; + return 0; + } + +out: + get_next_submodule_task_release(task); + return 0; } diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh index 42692219a1..af12c50e7d 100755 --- a/t/t5526-fetch-submodules.sh +++ b/t/t5526-fetch-submodules.sh @@ -605,4 +605,20 @@ test_expect_success "fetch new commits when submodule got renamed" ' test_cmp expect actual ' +test_expect_success "fetch new commits on-demand when they are not reachable" ' + git checkout --detach && + C=$(git -C submodule commit-tree -m "new change outside refs/heads" HEAD^{tree}) && + git -C submodule update-ref refs/changes/1 $C && + git update-index --cacheinfo 160000 $C submodule && + git commit -m "updated submodule outside of refs/heads" && + D=$(git rev-parse HEAD) && + git update-ref refs/changes/2 $D && + ( + cd downstream && + git fetch --recurse-submodules --recurse-submodules-default on-demand origin refs/changes/2:refs/heads/my_branch && + git -C submodule cat-file -t $C && + git checkout --recurse-submodules FETCH_HEAD + ) +' + test_done From patchwork Tue Oct 16 18:13:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10644051 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C709B109C for ; Tue, 16 Oct 2018 18:14:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BA85A2A587 for ; Tue, 16 Oct 2018 18:14:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AE9C72A5F9; Tue, 16 Oct 2018 18:14:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 47DEA2A587 for ; Tue, 16 Oct 2018 18:14:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727478AbeJQCGG (ORCPT ); Tue, 16 Oct 2018 22:06:06 -0400 Received: from mail-it1-f202.google.com ([209.85.166.202]:42526 "EHLO mail-it1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727395AbeJQCGG (ORCPT ); Tue, 16 Oct 2018 22:06:06 -0400 Received: by mail-it1-f202.google.com with SMTP id v125-v6so26843640ita.7 for ; Tue, 16 Oct 2018 11:14:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Nfr1jpOQmzwQFPfOqjexMqzJUD/hwxx48KXiGXgBQ3s=; b=ZUq2OTYcUEW9NT/yB3Ka2Mo2EZ0sA7Vjhaz+9b+eYZ2C7m5m9Yyo/tX0vX1q0BFtDI 4Q1tBBnQRCKGi5aOzFoRGyNEERxkiudutcXfPeEnRWS/wABtpFcQuF+9x7ZxYLdTeisM 5QPi9iSGJmIgokyKYRAXPWHbJ3CGhcij9k52kG/zB0UDzp20VJHfqnbnvoaDqSqVSfuW N5sfPmST37pPc5q3Ds7DyPUzKilueWXwtUXJUuVgks8rbp8DARyYHuCU7Q1KPOTe6fhe VSXN4KSTDtq5tj2O8oNBnv3QN7D1nFiK2rpf9uAMi13fmYo00rL6FF2M9o0dfhzWE/AT D6sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Nfr1jpOQmzwQFPfOqjexMqzJUD/hwxx48KXiGXgBQ3s=; b=dipPLVITuJAu6tM70Zo+eSYUJJCrVGXlX9ZRttzQ77s/bDXpPRZXdsk1ZvzGvb3cHX 2f95u2f+W8uO0ZOJMIoHil4we2jqrY8gXSCN8letleFa7B8rUvUGXjQPubA4TcSv9n+C CPmqg1MO+maWvnz2JLetuva3mRALTtPFmQH9HmXG6zF/ZaTE2wPmQDs4kGuD2qCeGiuk LCunxx5artFtmLag8hep1p1Na592AJF8UpWwwWtSYJBq1/uX31Cxiu5BJ8HhFS54kGWy GpreWHGcjnKg5jALKWd7LNgiAxjV+RqRqNlMxeNjp50/RWYesAp9BRJG58BNZgY5kSDS 0iUg== X-Gm-Message-State: ABuFfohSshP3T2t0/RaeYikccSa5/hh0Uy28R2zps/TlR8dsSNsHMceG +Ck+W5iJJeNf9y7N1IDRqvla/3xpAbkr X-Google-Smtp-Source: ACcGV6145UWlF8+qnjZm8F6CuXM40BIqxP9+Ubvb5sI/UKFoh8Ia0MRmJGcQZrfykDHi5iCuCmdS+1HDVj9m X-Received: by 2002:a24:5517:: with SMTP id e23-v6mr20995909itb.1.1539713666437; Tue, 16 Oct 2018 11:14:26 -0700 (PDT) Date: Tue, 16 Oct 2018 11:13:27 -0700 In-Reply-To: <20181016181327.107186-1-sbeller@google.com> Message-Id: <20181016181327.107186-10-sbeller@google.com> Mime-Version: 1.0 References: <20181016181327.107186-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0 Subject: [PATCH 9/9] builtin/fetch: check for submodule updates for non branch fetches From: Stefan Beller To: gitster@pobox.com Cc: git@vger.kernel.org, Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Gerrit, the code review tool, has a different workflow than our mailing list based approach. Usually users upload changes to a Gerrit server and continuous integration and testing happens by bots. Sometimes however a user wants to checkout a change locally and look at it locally. For this use case, Gerrit offers a command line snippet to copy and paste to your terminal, which looks like git fetch https:///gerrit refs/changes/ && git checkout FETCH_HEAD For Gerrit changes that contain changing submodule gitlinks, it would be easy to extend both the fetch and checkout with the '--recurse-submodules' flag, such that this command line snippet would produce the state of a change locally. However the functionality added in the previous patch, which would ensure that we fetch the objects in the submodule that the gitlink pointed at, only works for remote tracking branches so far, not for FETCH_HEAD. Make sure that fetching a superproject to its FETCH_HEAD, also respects the existence checks for objects in the submodule recursion. Signed-off-by: Stefan Beller Signed-off-by: Junio C Hamano --- builtin/fetch.c | 5 ++++- t/t5526-fetch-submodules.sh | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 95c44bf6ff..ea6ecd123e 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -887,11 +887,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, rc |= update_local_ref(ref, what, rm, ¬e, summary_width); free(ref); - } else + } else { + check_for_new_submodule_commits(&rm->old_oid); format_display(¬e, '*', *kind ? kind : "branch", NULL, *what ? what : "HEAD", "FETCH_HEAD", summary_width); + } + if (note.len) { if (verbosity >= 0 && !shown_url) { fprintf(stderr, _("From %.*s\n"), diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh index af12c50e7d..a509eabb04 100755 --- a/t/t5526-fetch-submodules.sh +++ b/t/t5526-fetch-submodules.sh @@ -615,7 +615,7 @@ test_expect_success "fetch new commits on-demand when they are not reachable" ' git update-ref refs/changes/2 $D && ( cd downstream && - git fetch --recurse-submodules --recurse-submodules-default on-demand origin refs/changes/2:refs/heads/my_branch && + git fetch --recurse-submodules origin refs/changes/2 && git -C submodule cat-file -t $C && git checkout --recurse-submodules FETCH_HEAD )