From patchwork Mon Feb 22 19:20:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 12099383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA93AC433E0 for ; Mon, 22 Feb 2021 19:26:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 97D5A64E15 for ; Mon, 22 Feb 2021 19:26:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231612AbhBVTZg (ORCPT ); Mon, 22 Feb 2021 14:25:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233193AbhBVTU5 (ORCPT ); Mon, 22 Feb 2021 14:20:57 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9943C061797 for ; Mon, 22 Feb 2021 11:20:15 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id s18so7808172pfe.10 for ; Mon, 22 Feb 2021 11:20:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=vfKfOGTi+pc7bqsX5Z/70a0qR04Cc4O6Wnb9CLDZLOM=; b=WsSm+ceU4ksyen5b33ELJ9/x83C7VssAHxfMSTBrdIeXYaYdNRTVSUxD4ezO65A7eW Fi0Rc++WlFgU7GcBI387fn7smNqYYU9rCz47T969CLgw6Zhpjvic4+dsYqfrsSRQn+QW fAHqSgdk4HYC7JEG0d/nCkWPk/8y/8nDzCnu67niOGwDB0hL1JuzFLByeiIqrEEM3jJc j80YxdgS3hJ8wiCdOvHLF+9DIE9TVGzeWBHdK59T2UQnAa86L4ahfSSaOefFvL3orWhQ DDVegBzqs/zS7idaeSJdRULWv1rETQByo7FC8WlWcIyY5qMNM7hNcDowEuEu2ZYXfAhE Ll1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=vfKfOGTi+pc7bqsX5Z/70a0qR04Cc4O6Wnb9CLDZLOM=; b=l5RxL93A9R0AbynaQY2L1kz6lzhtFsJbkcFprItH8c6A95PtBOVV4Mu0/RywFfCTyi xe3vXzaxe0HzGPJilKyaIMOS8bvGLFlfh4BY1aZQX2OhtC0Pc/odCiSVoU9kKG2BY6JF X+bttr0m+RN/mzPydaLH2cGnLsImLvhvtyDVOoXY4ldk7FKGuNBBe04It9hpI7KtCm5b 3HR6OqIvonfJh1D6+kuqYRy68xIGaXA5YlyoY1eXTVWQ4r65R/a46gp8v1r2ravYOKMC s349f880jxuhIJ1oTg7fCXuLH21g7H/+qZIDlVffj/eH0+SMHkz01tn2+On7439/BBkd +LFA== X-Gm-Message-State: AOAM5319f+UCIFEvWIjFk5eVSCgMaN/KLwcaI33XDzhQXeEDsjjwsfLP nJRwl9SBsT7uxF6BekndGQKu+DPiMUhOJKt7uqrtRNTo5QuPJ2h1qO9pIKrQ+QPlaDFHVUsHSqq vfKHMXU/X4H1mTUzQmzBdHSIZUHCP/pTVQ75X3umgnl9QFsyYFwabrkT3SRm4YyjA8uXqpmECWR JB X-Google-Smtp-Source: ABdhPJxnGm8vPumLDuNcFEVdkNSn8+JCHmY7OY1olv7xtGyV8uRQ5m4Pn1xtvqyBunM3h3aRmtzwbeJ+4o9z7l29EkA/ Sender: "jonathantanmy via sendgmr" X-Received: from twelve4.c.googlers.com ([fda3:e722:ac3:10:24:72f4:c0a8:437a]) (user=jonathantanmy job=sendgmr) by 2002:aa7:8d8e:0:b029:1d1:f9c9:cff6 with SMTP id i14-20020aa78d8e0000b02901d1f9c9cff6mr23713478pfr.31.1614021615062; Mon, 22 Feb 2021 11:20:15 -0800 (PST) Date: Mon, 22 Feb 2021 11:20:06 -0800 In-Reply-To: Message-Id: Mime-Version: 1.0 References: <20210115234300.350442-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.30.0.617.g56c4b15f3c-goog Subject: [PATCH v2 1/4] http: allow custom index-pack args From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , avarab@gmail.com, gitster@pobox.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Currently, when fetching, packfiles referenced by URIs are run through index-pack without any arguments other than --stdin and --keep, no matter what arguments are used for the packfile that is inline in the fetch response. As a preparation for ensuring that all packs (whether inline or not) use the same index-pack arguments, teach the http subsystem to allow custom index-pack arguments. http-fetch has been updated to use the new API. For now, it passes --keep alone instead of --keep with a process ID, but this is only temporary because http-fetch itself will be taught to accept index-pack parameters (instead of using a hardcoded constant) in a subsequent commit. Signed-off-by: Jonathan Tan Signed-off-by: Junio C Hamano --- http-fetch.c | 6 +++++- http.c | 15 ++++++++------- http.h | 10 +++++----- 3 files changed, 18 insertions(+), 13 deletions(-) diff --git a/http-fetch.c b/http-fetch.c index c4ccc5fea9..2d1d9d054f 100644 --- a/http-fetch.c +++ b/http-fetch.c @@ -43,6 +43,9 @@ static int fetch_using_walker(const char *raw_url, int get_verbosely, return rc; } +static const char *index_pack_args[] = + {"index-pack", "--stdin", "--keep", NULL}; + static void fetch_single_packfile(struct object_id *packfile_hash, const char *url) { struct http_pack_request *preq; @@ -55,7 +58,8 @@ static void fetch_single_packfile(struct object_id *packfile_hash, if (preq == NULL) die("couldn't create http pack request"); preq->slot->results = &results; - preq->generate_keep = 1; + preq->index_pack_args = index_pack_args; + preq->preserve_index_pack_stdout = 1; if (start_active_slot(preq->slot)) { run_active_slot(preq->slot); diff --git a/http.c b/http.c index 8b23a546af..f8ea28bb2e 100644 --- a/http.c +++ b/http.c @@ -2259,6 +2259,9 @@ void release_http_pack_request(struct http_pack_request *preq) free(preq); } +static const char *default_index_pack_args[] = + {"index-pack", "--stdin", NULL}; + int finish_http_pack_request(struct http_pack_request *preq) { struct child_process ip = CHILD_PROCESS_INIT; @@ -2270,17 +2273,15 @@ int finish_http_pack_request(struct http_pack_request *preq) tmpfile_fd = xopen(preq->tmpfile.buf, O_RDONLY); - strvec_push(&ip.args, "index-pack"); - strvec_push(&ip.args, "--stdin"); ip.git_cmd = 1; ip.in = tmpfile_fd; - if (preq->generate_keep) { - strvec_pushf(&ip.args, "--keep=git %"PRIuMAX, - (uintmax_t)getpid()); + ip.argv = preq->index_pack_args ? preq->index_pack_args + : default_index_pack_args; + + if (preq->preserve_index_pack_stdout) ip.out = 0; - } else { + else ip.no_stdout = 1; - } if (run_command(&ip)) { ret = -1; diff --git a/http.h b/http.h index 5de792ef3f..bf3d1270ad 100644 --- a/http.h +++ b/http.h @@ -218,12 +218,12 @@ struct http_pack_request { char *url; /* - * If this is true, finish_http_pack_request() will pass "--keep" to - * index-pack, resulting in the creation of a keep file, and will not - * suppress its stdout (that is, the "keep\t\n" line will be - * printed to stdout). + * index-pack command to run. Must be terminated by NULL. + * + * If NULL, defaults to {"index-pack", "--stdin", NULL}. */ - unsigned generate_keep : 1; + const char **index_pack_args; + unsigned preserve_index_pack_stdout : 1; FILE *packfile; struct strbuf tmpfile; From patchwork Mon Feb 22 19:20:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 12099385 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A59DEC433E6 for ; Mon, 22 Feb 2021 19:26:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7786364DB1 for ; Mon, 22 Feb 2021 19:26:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233227AbhBVT0M (ORCPT ); Mon, 22 Feb 2021 14:26:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232966AbhBVTU7 (ORCPT ); Mon, 22 Feb 2021 14:20:59 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 06802C0617A7 for ; Mon, 22 Feb 2021 11:20:18 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id o9so17449187yba.18 for ; Mon, 22 Feb 2021 11:20:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=2KDCcKW0fwfWkW20Is9cUgfK+oTUu8kLssxVdaq2Tsg=; b=dq53jb7EO4g+qDjUo96wFaQBm7riCwI0o0rOs1h5/ieopMywGnAzgTV4rnDmjdzyhx 9hdYE0Omi4bQfsLaVT+jFoYKx84G0HKnquN5EryDBOS1DBhwnhmk26eJ0cqnEd4rjo89 fq/ltqgrhsEwOeBkSwXoLaW81JN9AQH6Fb63SLhIw7sPrqNKeT4T3qcAFgLXWHX4LiBV q+e60yMHsNCkWwkclFgGgLJsBJ8laFm/zkUIzoYmlDQEmSJp23eMB9hPhDJ4ioqqJDKP UfLaLFDmRFmxdIYASE0tndZrz1MSt4T5cVNV/kWqiH7lcT/5e5RzbVym+FdYxabojReU Hp1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=2KDCcKW0fwfWkW20Is9cUgfK+oTUu8kLssxVdaq2Tsg=; b=NRYo8BKdbtAh+yMivf3CE/Sf9+XPgKU53/o+GN4Wz214XwOUPx0rfEF4cQrrjEdnnG qVzmJOItTlkLcvtSVp3GA8FEI5h+4mPJij5J89tBh9vj3mJyF/QHu/4+decYQ5G9amjX FdLBwfAXDxfvc/krPVOjt7uAiZy3Sn1Vjt1HT+bavj8CBPlfZPH2BbIkTN9iisA5c2uK j3NYAFJmj0G0xGfJgR/DBzzamSm/hzo+JfyzSsKqix2hIBW1Y0n8RKU5kyuamydXs6Xx I3W0vDa4g6Cmp97lBrL91iBAs82Cx1XvxVvioAeFZ2VeDQkG6DQxTWiV53HRiag3QA8/ RdZg== X-Gm-Message-State: AOAM5313BLkqH/HoV5gLZu87i/awh0mwNjOq56DlBjkxWK44+SjH7ABf QwguiZrHaMGnfNhJmbOQoyzHJSvm92tipsA/GIyZd6+rbD5fXB7POb6/GxUqEewISUsjzUqFJhN DliK5+jg6BH6fSXvJfvTYqwZCxYtOMdS3KL5g0/I0LIoDasdIPzO9qqF3jWu/QW6o2MSjRer4T6 gf X-Google-Smtp-Source: ABdhPJz69XeEwed89HTMPiKARt6YJC4yakXjychXBLRjwmTitx1ywd00eiASZ7ftnBkPTaXh8eAU8r+P/aRnFQ6cbEmJ Sender: "jonathantanmy via sendgmr" X-Received: from twelve4.c.googlers.com ([fda3:e722:ac3:10:24:72f4:c0a8:437a]) (user=jonathantanmy job=sendgmr) by 2002:a25:8b8b:: with SMTP id j11mr36860148ybl.305.1614021617184; Mon, 22 Feb 2021 11:20:17 -0800 (PST) Date: Mon, 22 Feb 2021 11:20:07 -0800 In-Reply-To: Message-Id: <57220ceb841056aade08705ca0ac73ccc69f05ab.1614021093.git.jonathantanmy@google.com> Mime-Version: 1.0 References: <20210115234300.350442-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.30.0.617.g56c4b15f3c-goog Subject: [PATCH v2 2/4] http-fetch: allow custom index-pack args From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , avarab@gmail.com, gitster@pobox.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This is the next step in teaching fetch-pack to pass its index-pack arguments when processing packfiles referenced by URIs. The "--keep" in fetch-pack.c will be replaced with a full message in a subsequent commit. Signed-off-by: Jonathan Tan Signed-off-by: Junio C Hamano --- Documentation/git-http-fetch.txt | 10 ++++++++-- fetch-pack.c | 3 +++ http-fetch.c | 20 +++++++++++++++----- t/t5550-http-fetch-dumb.sh | 5 ++++- 4 files changed, 30 insertions(+), 8 deletions(-) diff --git a/Documentation/git-http-fetch.txt b/Documentation/git-http-fetch.txt index 4deb4893f5..9fa17b60e4 100644 --- a/Documentation/git-http-fetch.txt +++ b/Documentation/git-http-fetch.txt @@ -41,11 +41,17 @@ commit-id:: ['\t'] --packfile=:: - Instead of a commit id on the command line (which is not expected in + For internal use only. Instead of a commit id on the command + line (which is not expected in this case), 'git http-fetch' fetches the packfile directly at the given URL and uses index-pack to generate corresponding .idx and .keep files. The hash is used to determine the name of the temporary file and is - arbitrary. The output of index-pack is printed to stdout. + arbitrary. The output of index-pack is printed to stdout. Requires + --index-pack-args. + +--index-pack-args=:: + For internal use only. The command to run on the contents of the + downloaded pack. Arguments are URL-encoded separated by spaces. --recover:: Verify that everything reachable from target is fetched. Used after diff --git a/fetch-pack.c b/fetch-pack.c index 876f90c759..aeac010b0b 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1645,6 +1645,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, strvec_pushf(&cmd.args, "--packfile=%.*s", (int) the_hash_algo->hexsz, packfile_uris.items[i].string); + strvec_push(&cmd.args, "--index-pack-arg=index-pack"); + strvec_push(&cmd.args, "--index-pack-arg=--stdin"); + strvec_push(&cmd.args, "--index-pack-arg=--keep"); strvec_push(&cmd.args, uri); cmd.git_cmd = 1; cmd.no_stdin = 1; diff --git a/http-fetch.c b/http-fetch.c index 2d1d9d054f..fa642462a9 100644 --- a/http-fetch.c +++ b/http-fetch.c @@ -3,6 +3,7 @@ #include "exec-cmd.h" #include "http.h" #include "walker.h" +#include "strvec.h" static const char http_fetch_usage[] = "git http-fetch " "[-c] [-t] [-a] [-v] [--recover] [-w ref] [--stdin | --packfile=hash | commit-id] url"; @@ -43,11 +44,9 @@ static int fetch_using_walker(const char *raw_url, int get_verbosely, return rc; } -static const char *index_pack_args[] = - {"index-pack", "--stdin", "--keep", NULL}; - static void fetch_single_packfile(struct object_id *packfile_hash, - const char *url) { + const char *url, + const char **index_pack_args) { struct http_pack_request *preq; struct slot_results results; int ret; @@ -90,6 +89,7 @@ int cmd_main(int argc, const char **argv) int packfile = 0; int nongit; struct object_id packfile_hash; + struct strvec index_pack_args = STRVEC_INIT; setup_git_directory_gently(&nongit); @@ -116,6 +116,8 @@ int cmd_main(int argc, const char **argv) packfile = 1; if (parse_oid_hex(p, &packfile_hash, &end) || *end) die(_("argument to --packfile must be a valid hash (got '%s')"), p); + } else if (skip_prefix(argv[arg], "--index-pack-arg=", &p)) { + strvec_push(&index_pack_args, p); } arg++; } @@ -128,10 +130,18 @@ int cmd_main(int argc, const char **argv) git_config(git_default_config, NULL); if (packfile) { - fetch_single_packfile(&packfile_hash, argv[arg]); + if (!index_pack_args.nr) + die(_("--packfile requires --index-pack-args")); + + fetch_single_packfile(&packfile_hash, argv[arg], + index_pack_args.v); + return 0; } + if (index_pack_args.nr) + die(_("--index-pack-args can only be used with --packfile")); + if (commits_on_stdin) { commits = walker_targets_stdin(&commit_id, &write_ref); } else { diff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh index 483578b2d7..358b322e05 100755 --- a/t/t5550-http-fetch-dumb.sh +++ b/t/t5550-http-fetch-dumb.sh @@ -224,7 +224,10 @@ test_expect_success 'http-fetch --packfile' ' git init packfileclient && p=$(cd "$HTTPD_DOCUMENT_ROOT_PATH"/repo_pack.git && ls objects/pack/pack-*.pack) && - git -C packfileclient http-fetch --packfile=$ARBITRARY "$HTTPD_URL"/dumb/repo_pack.git/$p >out && + git -C packfileclient http-fetch --packfile=$ARBITRARY \ + --index-pack-arg=index-pack --index-pack-arg=--stdin \ + --index-pack-arg=--keep \ + "$HTTPD_URL"/dumb/repo_pack.git/$p >out && grep "^keep.[0-9a-f]\{16,\}$" out && cut -c6- out >packhash && From patchwork Mon Feb 22 19:20:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 12099387 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0CB9C433DB for ; Mon, 22 Feb 2021 19:26:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BCCB864E32 for ; Mon, 22 Feb 2021 19:26:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233022AbhBVT0a (ORCPT ); Mon, 22 Feb 2021 14:26:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233219AbhBVTU7 (ORCPT ); Mon, 22 Feb 2021 14:20:59 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87574C0617A9 for ; Mon, 22 Feb 2021 11:20:19 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id v62so17535466ybb.15 for ; Mon, 22 Feb 2021 11:20:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=H14SKziskvi0fDy74D291MDC23rTB7aIJmAsEUUk8wk=; b=f0tQTmHXoGva6B8jNLyiDYKWmON1OfLRqD/CmuLwmCpOafbAZr8YVEeFKrlw+KX4j8 tmXGaD+vF1O7gLVAopM8OOaUF5K+XDqccopqx2eHwOCcTQNL9fNul9KwoUahIZwfAoCi eIj6fSCUoPjx8DZL5aXwGUAlN6Isnb/ftsvOLi15n96Hpu540LP7cFD28JfZtto6RFE7 ng0e7Ytuypf5s216ljqXw58rKiiUBoKxtp310dlswB3126XCqM3ctUJBFUrS6uJsm5rp KskIsfKezmrb2kWPympXp4gNHvlIB2E3SltnJGWB/yB+TiJnXOWeLbeuZMvjuWFpZQSG K63w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=H14SKziskvi0fDy74D291MDC23rTB7aIJmAsEUUk8wk=; b=BQ2s+jxvMBula+z+EizC/o9L8uwTG4R19KHvPRc47YzLMVYYyS3EYJ/kXqwAVB2gUj NZF/ppFDJf6irWgzaJUgUsyhZSlY6X9n7Szc/ljKMiaZviTJTxEivGvRk8ZDjou6aa9R 3FMe9BCNQAfkbRzSQ1o3wVQFjmtgQDMGPdXs/BlwnuHUFw7+UazLLSjFlHlJbjD2d0ma 5KP+AgOwnyu6BBZFzrOCCqkgZjxGZLRbisJSuJBz2y15qHn+deUZX/z/iZFxqazRdINg 80LZzghnwYqfyYxUoh4syr/gnq6DdoXRPWJ/V5pNc75nPKoHpyHYqN6e2a8uUaT7gNAI Gqew== X-Gm-Message-State: AOAM531WCUzgrOs/if6BgVCIFmGkZ9hxCWzqdzM6qIoczCZJ7PnXWERG xyrHhOYvrW8aCfPIiaWrLx47/FniS+k5vXUqn4zsqs6iPlioM5D8/evGNynFzZFSiw8SxHgPqvA 6RdHiqROZodA3WM6k75Z/pCHTq7hCdPcf/WVH0V9LogW+MRq8x9mGyOKyEcXAd51O2qNKzyzMxW Rj X-Google-Smtp-Source: ABdhPJz7IV5mgVMMRJ8rjZY+COZlldsQGSfLlh7M6kvJ4AvhSmTdQGnRpY9OwUqBPbsGaCUVy687Vmq6BQ/s2IE+Giaf Sender: "jonathantanmy via sendgmr" X-Received: from twelve4.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:437a]) (user=jonathantanmy job=sendgmr) by 2002:a25:730b:: with SMTP id o11mr34889344ybc.415.1614021618748; Mon, 22 Feb 2021 11:20:18 -0800 (PST) Date: Mon, 22 Feb 2021 11:20:08 -0800 In-Reply-To: Message-Id: Mime-Version: 1.0 References: <20210115234300.350442-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.30.0.617.g56c4b15f3c-goog Subject: [PATCH v2 3/4] fetch-pack: with packfile URIs, use index-pack arg From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , avarab@gmail.com, gitster@pobox.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Unify the index-pack arguments used when processing the inline pack and when downloading packfiles referenced by URIs. This is done by teaching get_pack() to also store the index-pack arguments whenever at least one packfile URI is given, and then when processing the packfile URI(s), using the stored arguments. Signed-off-by: Jonathan Tan Signed-off-by: Junio C Hamano --- fetch-pack.c | 34 +++++++++++++++++++++++----------- 1 file changed, 23 insertions(+), 11 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index aeac010b0b..dd0a6c4b34 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -797,12 +797,13 @@ static void write_promisor_file(const char *keep_name, } /* - * Pass 1 as "only_packfile" if the pack received is the only pack in this - * fetch request (that is, if there were no packfile URIs provided). + * If packfile URIs were provided, pass a non-NULL pointer to index_pack_args. + * The strings to pass as the --index-pack-arg arguments to http-fetch will be + * stored there. (It must be freed by the caller.) */ static int get_pack(struct fetch_pack_args *args, int xd[2], struct string_list *pack_lockfiles, - int only_packfile, + struct strvec *index_pack_args, struct ref **sought, int nr_sought) { struct async demux; @@ -845,7 +846,7 @@ static int get_pack(struct fetch_pack_args *args, strvec_push(&cmd.args, alternate_shallow_file); } - if (do_keep || args->from_promisor) { + if (do_keep || args->from_promisor || index_pack_args) { if (pack_lockfiles) cmd.out = -1; cmd_name = "index-pack"; @@ -863,7 +864,7 @@ static int get_pack(struct fetch_pack_args *args, "--keep=fetch-pack %"PRIuMAX " on %s", (uintmax_t)getpid(), hostname); } - if (only_packfile && args->check_self_contained_and_connected) + if (!index_pack_args && args->check_self_contained_and_connected) strvec_push(&cmd.args, "--check-self-contained-and-connected"); else /* @@ -901,7 +902,7 @@ static int get_pack(struct fetch_pack_args *args, : transfer_fsck_objects >= 0 ? transfer_fsck_objects : 0) { - if (args->from_promisor || !only_packfile) + if (args->from_promisor || index_pack_args) /* * We cannot use --strict in index-pack because it * checks both broken objects and links, but we only @@ -913,6 +914,13 @@ static int get_pack(struct fetch_pack_args *args, fsck_msg_types.buf); } + if (index_pack_args) { + int i; + + for (i = 0; i < cmd.args.nr; i++) + strvec_push(index_pack_args, cmd.args.v[i]); + } + cmd.in = demux.out; cmd.git_cmd = 1; if (start_command(&cmd)) @@ -1084,7 +1092,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args, alternate_shallow_file = setup_temporary_shallow(si->shallow); else alternate_shallow_file = NULL; - if (get_pack(args, fd, pack_lockfiles, 1, sought, nr_sought)) + if (get_pack(args, fd, pack_lockfiles, NULL, sought, nr_sought)) die(_("git fetch-pack: fetch failed.")); all_done: @@ -1535,6 +1543,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, int seen_ack = 0; struct string_list packfile_uris = STRING_LIST_INIT_DUP; int i; + struct strvec index_pack_args = STRVEC_INIT; negotiator = &negotiator_alloc; fetch_negotiator_init(r, negotiator); @@ -1624,7 +1633,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, receive_packfile_uris(&reader, &packfile_uris); process_section_header(&reader, "packfile", 0); if (get_pack(args, fd, pack_lockfiles, - !packfile_uris.nr, sought, nr_sought)) + packfile_uris.nr ? &index_pack_args : NULL, + sought, nr_sought)) die(_("git fetch-pack: fetch failed.")); do_check_stateless_delimiter(args, &reader); @@ -1636,6 +1646,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, } for (i = 0; i < packfile_uris.nr; i++) { + int j; struct child_process cmd = CHILD_PROCESS_INIT; char packname[GIT_MAX_HEXSZ + 1]; const char *uri = packfile_uris.items[i].string + @@ -1645,9 +1656,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, strvec_pushf(&cmd.args, "--packfile=%.*s", (int) the_hash_algo->hexsz, packfile_uris.items[i].string); - strvec_push(&cmd.args, "--index-pack-arg=index-pack"); - strvec_push(&cmd.args, "--index-pack-arg=--stdin"); - strvec_push(&cmd.args, "--index-pack-arg=--keep"); + for (j = 0; j < index_pack_args.nr; j++) + strvec_pushf(&cmd.args, "--index-pack-arg=%s", + index_pack_args.v[j]); strvec_push(&cmd.args, uri); cmd.git_cmd = 1; cmd.no_stdin = 1; @@ -1683,6 +1694,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, packname)); } string_list_clear(&packfile_uris, 0); + strvec_clear(&index_pack_args); if (negotiator) negotiator->release(negotiator); From patchwork Mon Feb 22 19:20:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 12099389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-31.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF134C433DB for ; Mon, 22 Feb 2021 19:26:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 90A4264E12 for ; Mon, 22 Feb 2021 19:26:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233242AbhBVT0k (ORCPT ); Mon, 22 Feb 2021 14:26:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233230AbhBVTVB (ORCPT ); Mon, 22 Feb 2021 14:21:01 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A481C0617AA for ; Mon, 22 Feb 2021 11:20:21 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id d8so8805680plo.23 for ; Mon, 22 Feb 2021 11:20:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=xitqUwzrxhQSO3dxHXeN6x3qwhk1jd2vlyo92hbbClQ=; b=tTWwM9ehk2GwihQkO2GWGCAlkkhuw28uMPa5UEWYVxZS1smYYKBohgTEeXzyG9y2yG AMb1fEhtA0RqJXXx+oVAmJb0wwY+A6GaxvPp5G4lqT5vNErA6TW98ad0r+xJf5jzlp/+ LlaX6AxrMd397RDg6mmBH3aF1lSNkEPNHbc1zbEppVfXPZnQDRVqJwN9lx9F314aN6xB 1kVBH5AD/Wrq/+OK6FTktcI8Rwwe9H4UHCaAE8HZkXACRTmT21L+CfNR+X8/7bWwQtCk g1FINPI8e8FEEGChndAsJYzYnkbGdCwYMjSVcZbWgnMMF9l5BkLqmO10qMddLeY38mIT jZvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=xitqUwzrxhQSO3dxHXeN6x3qwhk1jd2vlyo92hbbClQ=; b=h2g2yC2sHD6vWBXYkktdLEvEL8ucdt/HiRZlmghJjNRTnm98Jh4LpT3xadnhrYchZu QYFYs8Z+/g1DrdeeUZHSZPF3l1K0KjRSU6OEaSQQV6hyhpbpFigBguUlwFQfekslG8BR ZRaqwioJoo3LR39WwXaEm6coXrL8fkm6xX69VpyuVUgsDtWFxa0P+FRe7tDSYbCx6QHk AkaduPAnyQ8fuc3IJU1zmPoHc2muCAOp0CgPPrnjpqxGnCiiEAGCO4SU2vO4wyu9wxMs auLDclzqE8wdO54RLydb9YqUgY2ZEc92uGky2Zx6cEpRrzWYDHR7dU+3MCdimqVg7guq OL7A== X-Gm-Message-State: AOAM533vqeaKqBQO8zwNDmeUst0bk5BXm/rzmfXrefaIWnXLpQIndlru F1HZa+/Ga/CWR6oTzXtSvTKTWtp0KSjD1/9zezGUGqreIu74+DNOuOIfnZ+R1pT0g4I/gqYip9W sPW/wHdSymCHsTMILuK3WKymyBVL1qsi7Sno6rKSfZ0bgaLwNdStExD3QdLyQGnr3eJro/e/o9j vv X-Google-Smtp-Source: ABdhPJywtzCFnxnvDr8OvEhErZq3zVOuRr3mAeGD0llZ7KbBJ60yJBWEsIcOdyaIcBRhFPc41gXoawwAGE9hORjZMIW7 Sender: "jonathantanmy via sendgmr" X-Received: from twelve4.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:437a]) (user=jonathantanmy job=sendgmr) by 2002:a65:5bc5:: with SMTP id o5mr20704556pgr.17.1614021620473; Mon, 22 Feb 2021 11:20:20 -0800 (PST) Date: Mon, 22 Feb 2021 11:20:09 -0800 In-Reply-To: Message-Id: Mime-Version: 1.0 References: <20210115234300.350442-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.30.0.617.g56c4b15f3c-goog Subject: [PATCH v2 4/4] fetch-pack: print and use dangling .gitmodules From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , avarab@gmail.com, gitster@pobox.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Teach index-pack to print dangling .gitmodules links after its "keep" or "pack" line instead of declaring an error, and teach fetch-pack to check such lines printed. This allows the tree side of the .gitmodules link to be in one packfile and the blob side to be in another without failing the fsck check, because it is now fetch-pack which checks such objects after all packfiles have been downloaded and indexed (and not index-pack on an individual packfile, as it is before this commit). Signed-off-by: Jonathan Tan Signed-off-by: Junio C Hamano --- Documentation/git-index-pack.txt | 7 ++- builtin/index-pack.c | 25 +++++++++- builtin/receive-pack.c | 2 +- fetch-pack.c | 78 +++++++++++++++++++++++++++----- fsck.c | 5 ++ fsck.h | 2 + pack-write.c | 8 +++- pack.h | 2 +- t/t5702-protocol-v2.sh | 58 ++++++++++++++++++++++-- 9 files changed, 165 insertions(+), 22 deletions(-) diff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt index af0c26232c..e74a4a1eda 100644 --- a/Documentation/git-index-pack.txt +++ b/Documentation/git-index-pack.txt @@ -78,7 +78,12 @@ OPTIONS Die if the pack contains broken links. For internal use only. --fsck-objects:: - Die if the pack contains broken objects. For internal use only. + For internal use only. ++ +Die if the pack contains broken objects. If the pack contains a tree +pointing to a .gitmodules blob that does not exist, prints the hash of +that blob (for the caller to check) after the hash that goes into the +name of the pack/idx file (see "Notes"). --threads=:: Specifies the number of threads to spawn when resolving diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 557bd2f348..0444febeee 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1693,6 +1693,22 @@ static void show_pack_info(int stat_only) } } +static int print_dangling_gitmodules(struct fsck_options *o, + const struct object_id *oid, + enum object_type object_type, + int msg_type, const char *message) +{ + /* + * NEEDSWORK: Plumb the MSG_ID (from fsck.c) here and use it + * instead of relying on this string check. + */ + if (starts_with(message, "gitmodulesMissing")) { + printf("%s\n", oid_to_hex(oid)); + return 0; + } + return fsck_error_function(o, oid, object_type, msg_type, message); +} + int cmd_index_pack(int argc, const char **argv, const char *prefix) { int i, fix_thin_pack = 0, verify = 0, stat_only = 0; @@ -1888,8 +1904,13 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix) else close(input_fd); - if (do_fsck_object && fsck_finish(&fsck_options)) - die(_("fsck error in pack objects")); + if (do_fsck_object) { + struct fsck_options fo = fsck_options; + + fo.error_func = print_dangling_gitmodules; + if (fsck_finish(&fo)) + die(_("fsck error in pack objects")); + } free(objects); strbuf_release(&index_name_buf); diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index d49d050e6e..ed2c9b42e9 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -2275,7 +2275,7 @@ static const char *unpack(int err_fd, struct shallow_info *si) status = start_command(&child); if (status) return "index-pack fork failed"; - pack_lockfile = index_pack_lockfile(child.out); + pack_lockfile = index_pack_lockfile(child.out, NULL); close(child.out); status = finish_command(&child); if (status) diff --git a/fetch-pack.c b/fetch-pack.c index dd0a6c4b34..f9def5ac74 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -796,6 +796,26 @@ static void write_promisor_file(const char *keep_name, strbuf_release(&promisor_name); } +static void parse_gitmodules_oids(int fd, struct oidset *gitmodules_oids) +{ + int len = the_hash_algo->hexsz + 1; /* hash + NL */ + + do { + char hex_hash[GIT_MAX_HEXSZ + 1]; + int read_len = read_in_full(fd, hex_hash, len); + struct object_id oid; + const char *end; + + if (!read_len) + return; + if (read_len != len) + die("invalid length read %d", read_len); + if (parse_oid_hex(hex_hash, &oid, &end) || *end != '\n') + die("invalid hash"); + oidset_insert(gitmodules_oids, &oid); + } while (1); +} + /* * If packfile URIs were provided, pass a non-NULL pointer to index_pack_args. * The strings to pass as the --index-pack-arg arguments to http-fetch will be @@ -804,7 +824,8 @@ static void write_promisor_file(const char *keep_name, static int get_pack(struct fetch_pack_args *args, int xd[2], struct string_list *pack_lockfiles, struct strvec *index_pack_args, - struct ref **sought, int nr_sought) + struct ref **sought, int nr_sought, + struct oidset *gitmodules_oids) { struct async demux; int do_keep = args->keep_pack; @@ -812,6 +833,7 @@ static int get_pack(struct fetch_pack_args *args, struct pack_header header; int pass_header = 0; struct child_process cmd = CHILD_PROCESS_INIT; + int fsck_objects = 0; int ret; memset(&demux, 0, sizeof(demux)); @@ -846,8 +868,15 @@ static int get_pack(struct fetch_pack_args *args, strvec_push(&cmd.args, alternate_shallow_file); } - if (do_keep || args->from_promisor || index_pack_args) { - if (pack_lockfiles) + if (fetch_fsck_objects >= 0 + ? fetch_fsck_objects + : transfer_fsck_objects >= 0 + ? transfer_fsck_objects + : 0) + fsck_objects = 1; + + if (do_keep || args->from_promisor || index_pack_args || fsck_objects) { + if (pack_lockfiles || fsck_objects) cmd.out = -1; cmd_name = "index-pack"; strvec_push(&cmd.args, cmd_name); @@ -897,11 +926,7 @@ static int get_pack(struct fetch_pack_args *args, strvec_pushf(&cmd.args, "--pack_header=%"PRIu32",%"PRIu32, ntohl(header.hdr_version), ntohl(header.hdr_entries)); - if (fetch_fsck_objects >= 0 - ? fetch_fsck_objects - : transfer_fsck_objects >= 0 - ? transfer_fsck_objects - : 0) { + if (fsck_objects) { if (args->from_promisor || index_pack_args) /* * We cannot use --strict in index-pack because it @@ -925,10 +950,15 @@ static int get_pack(struct fetch_pack_args *args, cmd.git_cmd = 1; if (start_command(&cmd)) die(_("fetch-pack: unable to fork off %s"), cmd_name); - if (do_keep && pack_lockfiles) { - char *pack_lockfile = index_pack_lockfile(cmd.out); + if (do_keep && (pack_lockfiles || fsck_objects)) { + int is_well_formed; + char *pack_lockfile = index_pack_lockfile(cmd.out, &is_well_formed); + + if (!is_well_formed) + die(_("fetch-pack: invalid index-pack output")); if (pack_lockfile) string_list_append_nodup(pack_lockfiles, pack_lockfile); + parse_gitmodules_oids(cmd.out, gitmodules_oids); close(cmd.out); } @@ -963,6 +993,22 @@ static int cmp_ref_by_name(const void *a_, const void *b_) return strcmp(a->name, b->name); } +static void fsck_gitmodules_oids(struct oidset *gitmodules_oids) +{ + struct oidset_iter iter; + const struct object_id *oid; + struct fsck_options fo = FSCK_OPTIONS_STRICT; + + if (!oidset_size(gitmodules_oids)) + return; + + oidset_iter_init(gitmodules_oids, &iter); + while ((oid = oidset_iter_next(&iter))) + register_found_gitmodules(oid); + if (fsck_finish(&fo)) + die("fsck failed"); +} + static struct ref *do_fetch_pack(struct fetch_pack_args *args, int fd[2], const struct ref *orig_ref, @@ -977,6 +1023,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args, int agent_len; struct fetch_negotiator negotiator_alloc; struct fetch_negotiator *negotiator; + struct oidset gitmodules_oids = OIDSET_INIT; negotiator = &negotiator_alloc; fetch_negotiator_init(r, negotiator); @@ -1092,8 +1139,10 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args, alternate_shallow_file = setup_temporary_shallow(si->shallow); else alternate_shallow_file = NULL; - if (get_pack(args, fd, pack_lockfiles, NULL, sought, nr_sought)) + if (get_pack(args, fd, pack_lockfiles, NULL, sought, nr_sought, + &gitmodules_oids)) die(_("git fetch-pack: fetch failed.")); + fsck_gitmodules_oids(&gitmodules_oids); all_done: if (negotiator) @@ -1544,6 +1593,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, struct string_list packfile_uris = STRING_LIST_INIT_DUP; int i; struct strvec index_pack_args = STRVEC_INIT; + struct oidset gitmodules_oids = OIDSET_INIT; negotiator = &negotiator_alloc; fetch_negotiator_init(r, negotiator); @@ -1634,7 +1684,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, process_section_header(&reader, "packfile", 0); if (get_pack(args, fd, pack_lockfiles, packfile_uris.nr ? &index_pack_args : NULL, - sought, nr_sought)) + sought, nr_sought, &gitmodules_oids)) die(_("git fetch-pack: fetch failed.")); do_check_stateless_delimiter(args, &reader); @@ -1677,6 +1727,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, packname[the_hash_algo->hexsz] = '\0'; + parse_gitmodules_oids(cmd.out, &gitmodules_oids); + close(cmd.out); if (finish_command(&cmd)) @@ -1696,6 +1748,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, string_list_clear(&packfile_uris, 0); strvec_clear(&index_pack_args); + fsck_gitmodules_oids(&gitmodules_oids); + if (negotiator) negotiator->release(negotiator); diff --git a/fsck.c b/fsck.c index f82e2fe9e3..49ef6569e8 100644 --- a/fsck.c +++ b/fsck.c @@ -1243,6 +1243,11 @@ int fsck_error_function(struct fsck_options *o, return 1; } +void register_found_gitmodules(const struct object_id *oid) +{ + oidset_insert(&gitmodules_found, oid); +} + int fsck_finish(struct fsck_options *options) { int ret = 0; diff --git a/fsck.h b/fsck.h index 69cf715e79..d75b723bd5 100644 --- a/fsck.h +++ b/fsck.h @@ -62,6 +62,8 @@ int fsck_walk(struct object *obj, void *data, struct fsck_options *options); int fsck_object(struct object *obj, void *data, unsigned long size, struct fsck_options *options); +void register_found_gitmodules(const struct object_id *oid); + /* * Some fsck checks are context-dependent, and may end up queued; run this * after completing all fsck_object() calls in order to resolve any remaining diff --git a/pack-write.c b/pack-write.c index 3513665e1e..f66ea8e5a1 100644 --- a/pack-write.c +++ b/pack-write.c @@ -272,7 +272,7 @@ void fixup_pack_header_footer(int pack_fd, fsync_or_die(pack_fd, pack_name); } -char *index_pack_lockfile(int ip_out) +char *index_pack_lockfile(int ip_out, int *is_well_formed) { char packname[GIT_MAX_HEXSZ + 6]; const int len = the_hash_algo->hexsz + 6; @@ -286,11 +286,17 @@ char *index_pack_lockfile(int ip_out) */ if (read_in_full(ip_out, packname, len) == len && packname[len-1] == '\n') { const char *name; + + if (is_well_formed) + *is_well_formed = 1; packname[len-1] = 0; if (skip_prefix(packname, "keep\t", &name)) return xstrfmt("%s/pack/pack-%s.keep", get_object_directory(), name); + return NULL; } + if (is_well_formed) + *is_well_formed = 0; return NULL; } diff --git a/pack.h b/pack.h index 9fc0945ac9..09cffec395 100644 --- a/pack.h +++ b/pack.h @@ -85,7 +85,7 @@ int verify_pack_index(struct packed_git *); int verify_pack(struct repository *, struct packed_git *, verify_fn fn, struct progress *, uint32_t); off_t write_pack_header(struct hashfile *f, uint32_t); void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t, unsigned char *, off_t); -char *index_pack_lockfile(int fd); +char *index_pack_lockfile(int fd, int *is_well_formed); /* * The "hdr" output buffer should be at least this big, which will handle sizes diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index 7d5b17909b..b1bc73a9a9 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -847,8 +847,9 @@ test_expect_success 'part of packfile response provided as URI' ' test -f hfound && test -f h2found && - # Ensure that there are exactly 6 files (3 .pack and 3 .idx). - ls http_child/.git/objects/pack/* >filelist && + # Ensure that there are exactly 3 packfiles with associated .idx + ls http_child/.git/objects/pack/*.pack \ + http_child/.git/objects/pack/*.idx >filelist && test_line_count = 6 filelist ' @@ -901,8 +902,9 @@ test_expect_success 'packfile-uri with transfer.fsckobjects' ' -c fetch.uriprotocols=http,https \ clone "$HTTPD_URL/smart/http_parent" http_child && - # Ensure that there are exactly 4 files (2 .pack and 2 .idx). - ls http_child/.git/objects/pack/* >filelist && + # Ensure that there are exactly 2 packfiles with associated .idx + ls http_child/.git/objects/pack/*.pack \ + http_child/.git/objects/pack/*.idx >filelist && test_line_count = 4 filelist ' @@ -936,6 +938,54 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' test_i18ngrep "invalid author/committer line - missing email" error ' +test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmodules is separate from tree' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + rm -rf "$P" http_child && + + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + + echo "[submodule libfoo]" >"$P/.gitmodules" && + echo "path = include/foo" >>"$P/.gitmodules" && + echo "url = git://example.com/git/lib.git" >>"$P/.gitmodules" && + git -C "$P" add .gitmodules && + git -C "$P" commit -m x && + + configure_exclusion "$P" .gitmodules >h && + + sane_unset GIT_TEST_SIDEBAND_ALL && + git -c protocol.version=2 -c transfer.fsckobjects=1 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child && + + # Ensure that there are exactly 2 packfiles with associated .idx + ls http_child/.git/objects/pack/*.pack \ + http_child/.git/objects/pack/*.idx >filelist && + test_line_count = 4 filelist +' + +test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodules separate from tree is invalid' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + rm -rf "$P" http_child err && + + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + + echo "[submodule \"..\"]" >"$P/.gitmodules" && + echo "path = include/foo" >>"$P/.gitmodules" && + echo "url = git://example.com/git/lib.git" >>"$P/.gitmodules" && + git -C "$P" add .gitmodules && + git -C "$P" commit -m x && + + configure_exclusion "$P" .gitmodules >h && + + sane_unset GIT_TEST_SIDEBAND_ALL && + test_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child 2>err && + test_i18ngrep "disallowed submodule name" err +' + # DO NOT add non-httpd-specific tests here, because the last part of this # test script is only executed when httpd is available and enabled.