From patchwork Tue Jun 8 00:25:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 12304911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D133C47082 for ; Tue, 8 Jun 2021 00:26:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 13317611BD for ; Tue, 8 Jun 2021 00:26:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230487AbhFHA2Q (ORCPT ); Mon, 7 Jun 2021 20:28:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43276 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230209AbhFHA2O (ORCPT ); Mon, 7 Jun 2021 20:28:14 -0400 Received: from mail-qv1-xf49.google.com (mail-qv1-xf49.google.com [IPv6:2607:f8b0:4864:20::f49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F2F76C061787 for ; Mon, 7 Jun 2021 17:26:07 -0700 (PDT) Received: by mail-qv1-xf49.google.com with SMTP id n4-20020ad44a240000b029021cbf9668daso14274963qvz.23 for ; Mon, 07 Jun 2021 17:26:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=a4w/4CB56ohd8seT4yKr0IRci2/IH9bNaRkYotemsKU=; b=Rj4MF1ZK997VSy8nHkby09di/LRg1Lwjx8EJt2ECv08GKnZEih+rZl99Fpj2lv9ov0 Z3+mSzSoX/FdWVWCs3LOa4xGqHRe21emEeCZ6amptrEljg9gnadT4fbd58oLEIbJ1yZw ZywJ4kQilvBAo2pgB/wmEVp+XNH5hHxQXtrLVe1OrZRGfePlgTZYj5XdQvvIwF2G3SKY FFd+pb7U7RsU1CSvnd7UYuO3s8j0PvNAbhcsrWn6CymPmpGUYwj3pBJQcs+C4S/tuRXK vaelEt3BRxj+VH1U6qhOLltBNVjquWqHCyn8vsnTsmNbqXhD+WED8OvsxZA2ZRWBvpsc ZBPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=a4w/4CB56ohd8seT4yKr0IRci2/IH9bNaRkYotemsKU=; b=BkBwclytmF+0gxE3P57ipH53KwbpU5rOpP+57aNDOjqkbegmkoaCrNAtfFkTlz1T5N LOL07quaqgba4fy++9ABuWeKpNCndVgCLnw/xRfmFoA3xPZqMcopratAnrEJUc3QSTPG ZMefSnWVM5o5vRWga1k4HsDmIksyssCxK+EBWu0azXnVGz+AhN8JvkFTd1L27ZLotnH3 LDMWqYDUov6kXr0LLtGfwWHZVK4Pt1iLyBEn5fhgqERN4QBN4xTPp4ykEQEaN64QxwZD WVqMXcfBlQVW2XyccQWu+QDgQxrq6zGETKGb5Jakcmqh3zZP8EAoXSBUSowshvAqBvhW kaeQ== X-Gm-Message-State: AOAM5318MbxK5fdc31bkvT9h3ZcnGBVPs1yge5HrP1qc5DWc/o3XAwpu m7OvIZORINYH4tjFa6bgttUJ3W7sBN0BCEFJMVGrxUKjo2UfmEeziUFSiGeZDRLkaJW+YEYncfz JOSyJn1PVW+zZiXk1rT2cyIO+FWwqusFDFIiDmH4SIs9MzrOVSvgifJnwUco74WeAlWg8+lKGaV Br X-Google-Smtp-Source: ABdhPJwVJ1vRXD+nWeTZPo3yJBI4h66lkBz1nSC8fQP6n8/hiz1Ssky7MHsbj5ttT0FoJ731MuoPBJpm6efOGxNAi8G/ X-Received: from twelve4.c.googlers.com ([fda3:e722:ac3:10:24:72f4:c0a8:437a]) (user=jonathantanmy job=sendgmr) by 2002:a0c:80c1:: with SMTP id 59mr20971906qvb.31.1623111964626; Mon, 07 Jun 2021 17:26:04 -0700 (PDT) Date: Mon, 7 Jun 2021 17:25:56 -0700 In-Reply-To: Message-Id: <07290cba86fda73ee329a47db8e524b32dba25af.1623111879.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.32.0.rc1.229.g3e70b5a671-goog Subject: [PATCH v2 1/4] promisor-remote: read partialClone config here From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , me@ttaylorr.com, newren@gmail.com, emilyshaffer@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Currently, the reading of config related to promisor remotes is done in two places: once in setup.c (which sets the global variable repository_format_partial_clone, to be read by the code in promisor-remote.c), and once in promisor-remote.c. This means that care must be taken to ensure that repository_format_partial_clone is set before any code in promisor-remote.c accesses it. To simplify the code, move all such config reading to promisor-remote.c. By doing this, it will be easier to see when repository_format_partial_clone is written and, thus, to reason about the code. This will be especially helpful in a subsequent commit, which modifies this code. Signed-off-by: Jonathan Tan --- cache.h | 1 - promisor-remote.c | 14 +++++++++----- promisor-remote.h | 6 ------ setup.c | 10 +++++++--- 4 files changed, 16 insertions(+), 15 deletions(-) diff --git a/cache.h b/cache.h index ba04ff8bd3..dbdcec8601 100644 --- a/cache.h +++ b/cache.h @@ -1061,7 +1061,6 @@ extern int repository_format_worktree_config; struct repository_format { int version; int precious_objects; - char *partial_clone; /* value of extensions.partialclone */ int worktree_config; int is_bare; int hash_algo; diff --git a/promisor-remote.c b/promisor-remote.c index da3f2ca261..c0e5061dfe 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -7,11 +7,6 @@ static char *repository_format_partial_clone; -void set_repository_format_partial_clone(char *partial_clone) -{ - repository_format_partial_clone = xstrdup_or_null(partial_clone); -} - static int fetch_objects(const char *remote_name, const struct object_id *oids, int oid_nr) @@ -99,6 +94,15 @@ static int promisor_remote_config(const char *var, const char *value, void *data size_t namelen; const char *subkey; + if (!strcmp(var, "extensions.partialclone")) { + /* + * NULL value is handled in handle_extension_v0 in setup.c. + */ + if (value) + repository_format_partial_clone = xstrdup(value); + return 0; + } + if (parse_config_key(var, "remote", &name, &namelen, &subkey) < 0) return 0; diff --git a/promisor-remote.h b/promisor-remote.h index c7a14063c5..687210ab87 100644 --- a/promisor-remote.h +++ b/promisor-remote.h @@ -32,10 +32,4 @@ int promisor_remote_get_direct(struct repository *repo, const struct object_id *oids, int oid_nr); -/* - * This should be used only once from setup.c to set the value we got - * from the extensions.partialclone config option. - */ -void set_repository_format_partial_clone(char *partial_clone); - #endif /* PROMISOR_REMOTE_H */ diff --git a/setup.c b/setup.c index 59e2facd9d..d60b6bc554 100644 --- a/setup.c +++ b/setup.c @@ -470,7 +470,13 @@ static enum extension_result handle_extension_v0(const char *var, } else if (!strcmp(ext, "partialclone")) { if (!value) return config_error_nonbool(var); - data->partial_clone = xstrdup(value); + /* + * This config variable will be read together with the + * other relevant config variables in + * promisor_remote_config() in promisor_remote.c, so we + * do not need to read it here. Just report that this + * extension is known. + */ return EXTENSION_OK; } else if (!strcmp(ext, "worktreeconfig")) { data->worktree_config = git_config_bool(var, value); @@ -566,7 +572,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_ } repository_format_precious_objects = candidate->precious_objects; - set_repository_format_partial_clone(candidate->partial_clone); repository_format_worktree_config = candidate->worktree_config; string_list_clear(&candidate->unknown_extensions, 0); string_list_clear(&candidate->v1_only_extensions, 0); @@ -650,7 +655,6 @@ void clear_repository_format(struct repository_format *format) string_list_clear(&format->unknown_extensions, 0); string_list_clear(&format->v1_only_extensions, 0); free(format->work_tree); - free(format->partial_clone); init_repository_format(format); } From patchwork Tue Jun 8 00:25:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 12304909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9CDEC47095 for ; Tue, 8 Jun 2021 00:26:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AEA6661108 for ; Tue, 8 Jun 2021 00:26:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230463AbhFHA2O (ORCPT ); Mon, 7 Jun 2021 20:28:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230440AbhFHA2O (ORCPT ); Mon, 7 Jun 2021 20:28:14 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA89BC061574 for ; Mon, 7 Jun 2021 17:26:07 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id e12-20020a17090a630cb029016de1736f41so6222832pjj.3 for ; Mon, 07 Jun 2021 17:26:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=kqbsLcja9Cd7oIiWhK4mgo/i942bpLQQ3Wl2oXojnL4=; b=kzTCD2wGQss9iapAxaE2ZIu9aZL+F/L/WctYB50YznUyP41jb+N217856SxEP/pmzN j9/8TJAYQELhWW9p/RhpkIesX190QkaUPavRhLeOTAJOOb/VT47VkVIVhnE7GVxUeNbF N7V/kK4Xm0AGE3IK0lu38kI69pP0ZVex9iHg6jX/X8+jap2nfW3Z8pPm8Te1rx6m19OI le0PqN28kmTE0DsZSXkSrQ4kr8fX/YLePH7BCcARJNRFdQxYQHwAvdPSw5J6m2/V5IjZ +CRN4+272WHovtIbSYHm9tOjZlIxUkDqmg0v5V6X8Gl+ygwjqNrlkTEA+4I6KmzPX0Pi 1s8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=kqbsLcja9Cd7oIiWhK4mgo/i942bpLQQ3Wl2oXojnL4=; b=hZlcXSwSU/Ybp0kmeIPyB5iN3naSORqKRio+WYpzY1MHzVgkqjBDhNntgnidmd6x+6 oFGsVfDyhWKqGJqW4I777IUVDsNkebFw7LAyhUIT182yvbSCfc03ThSvM84CYkVjyXZH W6aNpW+JgOIc4afKPWlJOPxsnZ5wqdDCafgI5T4xnFgvRNG9q3gDY7ZBpI6bhevZuWK3 AmWDdWdtTnGM58cmkvUVExsGqp76vrkLk5+WCaXgBvTlKwFtr6+0dPPKWn9kPM3+iZg5 6evH+DS3RavQ64PCqwLezaV97ds7R2QCAOahdaOBZxea66uPdQTBwSQV5RhfC8dm0YoC phvA== X-Gm-Message-State: AOAM530KhNoiQqkcL/PYGPMTY1ByzDqzQB76x7ilBvQbigx+0NZLPY5Z PIBUK0u8VIUhTqAw/qaetZq3zqj02pqFbtvpT1cY/+wTKEHkWKfK1oWpFod6fPVBh7JU+tEwD1Y gVZf/dqOfAbpG4N9Hj6u7bw/VCGK90svdUJeswDhB4bod4kBVOSTIF5dZ+pdd4P73s94RmamoVC pF X-Google-Smtp-Source: ABdhPJzZ56B08wsdFLmjsoUnEwUmjEPKDfFbBrZj7XezQwqcKW1fbB+wBYCv/wq3WFdkloNvr+W3F6KOUk1+61NGojnC X-Received: from twelve4.c.googlers.com ([fda3:e722:ac3:10:24:72f4:c0a8:437a]) (user=jonathantanmy job=sendgmr) by 2002:aa7:8b56:0:b029:2b9:77be:d305 with SMTP id i22-20020aa78b560000b02902b977bed305mr19387663pfd.61.1623111966337; Mon, 07 Jun 2021 17:26:06 -0700 (PDT) Date: Mon, 7 Jun 2021 17:25:57 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.32.0.rc1.229.g3e70b5a671-goog Subject: [PATCH v2 2/4] promisor-remote: support per-repository config From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , me@ttaylorr.com, newren@gmail.com, emilyshaffer@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Instead of using global variables to store promisor remote information, store this config in struct repository instead, and add repository-agnostic non-static functions corresponding to the existing non-static functions that only work on the_repository. The actual lazy-fetching of missing objects currently does not work on repositories other than the_repository, and will still not work after this commit, so add a BUG message explaining this. A subsequent commit will remove this limitation. Signed-off-by: Jonathan Tan --- promisor-remote.c | 103 ++++++++++++++++++++++++++-------------------- promisor-remote.h | 22 ++++++++-- repository.c | 6 +++ repository.h | 4 ++ 4 files changed, 87 insertions(+), 48 deletions(-) diff --git a/promisor-remote.c b/promisor-remote.c index c0e5061dfe..e1e1f7e93a 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -5,7 +5,11 @@ #include "transport.h" #include "strvec.h" -static char *repository_format_partial_clone; +struct promisor_remote_config { + char *repository_format_partial_clone; + struct promisor_remote *promisors; + struct promisor_remote **promisors_tail; +}; static int fetch_objects(const char *remote_name, const struct object_id *oids, @@ -37,10 +41,8 @@ static int fetch_objects(const char *remote_name, return finish_command(&child) ? -1 : 0; } -static struct promisor_remote *promisors; -static struct promisor_remote **promisors_tail = &promisors; - -static struct promisor_remote *promisor_remote_new(const char *remote_name) +static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config, + const char *remote_name) { struct promisor_remote *r; @@ -52,18 +54,19 @@ static struct promisor_remote *promisor_remote_new(const char *remote_name) FLEX_ALLOC_STR(r, name, remote_name); - *promisors_tail = r; - promisors_tail = &r->next; + *config->promisors_tail = r; + config->promisors_tail = &r->next; return r; } -static struct promisor_remote *promisor_remote_lookup(const char *remote_name, +static struct promisor_remote *promisor_remote_lookup(struct promisor_remote_config *config, + const char *remote_name, struct promisor_remote **previous) { struct promisor_remote *r, *p; - for (p = NULL, r = promisors; r; p = r, r = r->next) + for (p = NULL, r = config->promisors; r; p = r, r = r->next) if (!strcmp(r->name, remote_name)) { if (previous) *previous = p; @@ -73,7 +76,8 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name, return NULL; } -static void promisor_remote_move_to_tail(struct promisor_remote *r, +static void promisor_remote_move_to_tail(struct promisor_remote_config *config, + struct promisor_remote *r, struct promisor_remote *previous) { if (r->next == NULL) @@ -82,14 +86,15 @@ static void promisor_remote_move_to_tail(struct promisor_remote *r, if (previous) previous->next = r->next; else - promisors = r->next ? r->next : r; + config->promisors = r->next ? r->next : r; r->next = NULL; - *promisors_tail = r; - promisors_tail = &r->next; + *config->promisors_tail = r; + config->promisors_tail = &r->next; } static int promisor_remote_config(const char *var, const char *value, void *data) { + struct promisor_remote_config *config = data; const char *name; size_t namelen; const char *subkey; @@ -99,7 +104,7 @@ static int promisor_remote_config(const char *var, const char *value, void *data * NULL value is handled in handle_extension_v0 in setup.c. */ if (value) - repository_format_partial_clone = xstrdup(value); + config->repository_format_partial_clone = xstrdup(value); return 0; } @@ -114,8 +119,8 @@ static int promisor_remote_config(const char *var, const char *value, void *data remote_name = xmemdupz(name, namelen); - if (!promisor_remote_lookup(remote_name, NULL)) - promisor_remote_new(remote_name); + if (!promisor_remote_lookup(config, remote_name, NULL)) + promisor_remote_new(config, remote_name); free(remote_name); return 0; @@ -124,9 +129,9 @@ static int promisor_remote_config(const char *var, const char *value, void *data struct promisor_remote *r; char *remote_name = xmemdupz(name, namelen); - r = promisor_remote_lookup(remote_name, NULL); + r = promisor_remote_lookup(config, remote_name, NULL); if (!r) - r = promisor_remote_new(remote_name); + r = promisor_remote_new(config, remote_name); free(remote_name); @@ -139,59 +144,65 @@ static int promisor_remote_config(const char *var, const char *value, void *data return 0; } -static int initialized; - -static void promisor_remote_init(void) +static void promisor_remote_init(struct repository *r) { - if (initialized) + struct promisor_remote_config *config; + + if (r->promisor_remote_config) return; - initialized = 1; + config = r->promisor_remote_config = + xcalloc(sizeof(*r->promisor_remote_config), 1); + config->promisors_tail = &config->promisors; - git_config(promisor_remote_config, NULL); + git_config(promisor_remote_config, config); - if (repository_format_partial_clone) { + if (config->repository_format_partial_clone) { struct promisor_remote *o, *previous; - o = promisor_remote_lookup(repository_format_partial_clone, + o = promisor_remote_lookup(config, + config->repository_format_partial_clone, &previous); if (o) - promisor_remote_move_to_tail(o, previous); + promisor_remote_move_to_tail(config, o, previous); else - promisor_remote_new(repository_format_partial_clone); + promisor_remote_new(config, config->repository_format_partial_clone); } } -static void promisor_remote_clear(void) +void promisor_remote_clear(struct promisor_remote_config *config) { - while (promisors) { - struct promisor_remote *r = promisors; - promisors = promisors->next; + FREE_AND_NULL(config->repository_format_partial_clone); + + while (config->promisors) { + struct promisor_remote *r = config->promisors; + config->promisors = config->promisors->next; free(r); } - promisors_tail = &promisors; + config->promisors_tail = &config->promisors; } -void promisor_remote_reinit(void) +void repo_promisor_remote_reinit(struct repository *r) { - initialized = 0; - promisor_remote_clear(); - promisor_remote_init(); + promisor_remote_clear(r->promisor_remote_config); + FREE_AND_NULL(r->promisor_remote_config); + promisor_remote_init(r); } -struct promisor_remote *promisor_remote_find(const char *remote_name) +struct promisor_remote *repo_promisor_remote_find(struct repository *r, + const char *remote_name) { - promisor_remote_init(); + promisor_remote_init(r); if (!remote_name) - return promisors; + return r->promisor_remote_config->promisors; - return promisor_remote_lookup(remote_name, NULL); + return promisor_remote_lookup(r->promisor_remote_config, remote_name, NULL); } -int has_promisor_remote(void) +int repo_has_promisor_remote(struct repository *r) { - return !!promisor_remote_find(NULL); + return !!repo_promisor_remote_find(r, NULL); } static int remove_fetched_oids(struct repository *repo, @@ -239,9 +250,11 @@ int promisor_remote_get_direct(struct repository *repo, if (oid_nr == 0) return 0; - promisor_remote_init(); + promisor_remote_init(repo); - for (r = promisors; r; r = r->next) { + if (repo != the_repository) + BUG("only the_repository is supported for now"); + for (r = repo->promisor_remote_config->promisors; r; r = r->next) { if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) { if (remaining_nr == 1) continue; diff --git a/promisor-remote.h b/promisor-remote.h index 687210ab87..edc45ab0f5 100644 --- a/promisor-remote.h +++ b/promisor-remote.h @@ -17,9 +17,25 @@ struct promisor_remote { const char name[FLEX_ARRAY]; }; -void promisor_remote_reinit(void); -struct promisor_remote *promisor_remote_find(const char *remote_name); -int has_promisor_remote(void); +void repo_promisor_remote_reinit(struct repository *r); +static inline void promisor_remote_reinit(void) +{ + repo_promisor_remote_reinit(the_repository); +} + +void promisor_remote_clear(struct promisor_remote_config *config); + +struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name); +static inline struct promisor_remote *promisor_remote_find(const char *remote_name) +{ + return repo_promisor_remote_find(the_repository, remote_name); +} + +int repo_has_promisor_remote(struct repository *r); +static inline int has_promisor_remote(void) +{ + return repo_has_promisor_remote(the_repository); +} /* * Fetches all requested objects from all promisor remotes, trying them one at diff --git a/repository.c b/repository.c index 448cd557d4..dca0a11ab6 100644 --- a/repository.c +++ b/repository.c @@ -11,6 +11,7 @@ #include "lockfile.h" #include "submodule-config.h" #include "sparse-index.h" +#include "promisor-remote.h" /* The main repository */ static struct repository the_repo; @@ -258,6 +259,11 @@ void repo_clear(struct repository *repo) if (repo->index != &the_index) FREE_AND_NULL(repo->index); } + + if (repo->promisor_remote_config) { + promisor_remote_clear(repo->promisor_remote_config); + FREE_AND_NULL(repo->promisor_remote_config); + } } int repo_read_index(struct repository *repo) diff --git a/repository.h b/repository.h index a45f7520fd..fc06c154e2 100644 --- a/repository.h +++ b/repository.h @@ -10,6 +10,7 @@ struct lock_file; struct pathspec; struct raw_object_store; struct submodule_cache; +struct promisor_remote_config; enum untracked_cache_setting { UNTRACKED_CACHE_UNSET = -1, @@ -139,6 +140,9 @@ struct repository { /* True if commit-graph has been disabled within this process. */ int commit_graph_disabled; + /* Configurations related to promisor remotes. */ + struct promisor_remote_config *promisor_remote_config; + /* Configurations */ /* Indicate if a repository has a different 'commondir' from 'gitdir' */ From patchwork Tue Jun 8 00:25:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 12304907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9CBBC4743F for ; Tue, 8 Jun 2021 00:26:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BBF76611BD for ; Tue, 8 Jun 2021 00:26:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230478AbhFHA2P (ORCPT ); Mon, 7 Jun 2021 20:28:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43278 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230443AbhFHA2O (ORCPT ); Mon, 7 Jun 2021 20:28:14 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B5A1C061789 for ; Mon, 7 Jun 2021 17:26:08 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id e12-20020a17090a630cb029016de1736f41so6222876pjj.3 for ; Mon, 07 Jun 2021 17:26:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Fs3dS7gGKmtreGGRveiB5U5kmMhmWktwIZKtGKPRLCA=; b=SYRPKIFrCZ4UvqgDO8b+w9wz69GwtuPxVQv7rFTrIL8mflIKdMzltes8jrV8P8a6/4 As0DCmqrV2i0LWwd8pmPmGAn9RXjNDE1D6iOGLTEKAVrZ2fhUATWlbM+jvxp7wkL03Ws 7tux7NHil9IeAkBtFDCB9XRqJT91lArc8kFGI+WM1tTy2vqLxI/O78JJ4Yw0e/8B6bvY Bd/zyzPqKFn6lnW0k9G0PC963az5MUh9OJiJdVGT2sYJHswaVqg3nDWmmPoxG9rOoBIp D+bm4niaol2/EdZ2IiyN7IwXa5rnypu7d4MyIpWiXPeEM9E10MWGDTccLvlZUKTJ1Byq hWpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Fs3dS7gGKmtreGGRveiB5U5kmMhmWktwIZKtGKPRLCA=; b=i9a/SubFK7Nlzjzmb+23b82EtkVZ22nV032pAS+IBDJCs03GTSo5Kx5nKmoUy1AXrc tck7NP2lunc4vQhmikZ6+Y0fb29gjkmip71EGZVsg3692Ka/+XCIOECMAvhLzMJyToo6 oIlIBLZIkwD3BKSCXuPQTt4pbyqLDxnJTiOCJN15lT77F0TLWMuD5LOcEBjhU4vbPhQ/ h84nbWXk/gC1mFztV+1y1oMp04U94Vk/BGTUV11WflaEfgKZ6U2E1/K7BNyUIds328T6 46zYYXLgQuuET5JXlILlfNhXDz2icwTQY8N5sDIEmk6f2m7kcEKiag6sLtqjKFWfSc7b pLxg== X-Gm-Message-State: AOAM533Mj84QmZUzATeOTk6YeDbNIabSy+/Xcxj9dN9+qNHPw8hS0DgQ WTZhtK04RQkwS2Py9rqj/ohW2QsTmhEqAT0suMBRMmlXCFW3CNtDTkR8upehAyJwQOIgceF0C2l qrJyThdcb+o1OT8HYkD0zPpWo+u5ANjvnbRJRNYoCDmrfu4+E+AhXQPavZIpIp5GP60Tz1uydGi AI X-Google-Smtp-Source: ABdhPJyrOHwLeTzEquAlmjgGEZZGPIeU9Gm1UMV3GC/7pi00w+v7uyHeCFIZC1pPFKyiRLANXfdlyQ5Qkd7weja2z4Tp X-Received: from twelve4.c.googlers.com ([fda3:e722:ac3:10:24:72f4:c0a8:437a]) (user=jonathantanmy job=sendgmr) by 2002:a17:902:904a:b029:101:af84:4f55 with SMTP id w10-20020a170902904ab0290101af844f55mr20539341plz.80.1623111967686; Mon, 07 Jun 2021 17:26:07 -0700 (PDT) Date: Mon, 7 Jun 2021 17:25:58 -0700 In-Reply-To: Message-Id: <9cbdf60981e66e938b6c57c041c15e85245c734d.1623111879.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.32.0.rc1.229.g3e70b5a671-goog Subject: [PATCH v2 3/4] run-command: move envvar-resetting function From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , me@ttaylorr.com, newren@gmail.com, emilyshaffer@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org There is a function that resets environment variables, used when invoking a sub-process in a submodule. The lazy-fetching code (used in partial clones) will need this function in a subsequent commit, so move it to a more central location. Signed-off-by: Jonathan Tan --- run-command.c | 10 ++++++++++ run-command.h | 9 +++++++++ submodule.c | 14 ++------------ 3 files changed, 21 insertions(+), 12 deletions(-) diff --git a/run-command.c b/run-command.c index be6bc128cd..a6c458119c 100644 --- a/run-command.c +++ b/run-command.c @@ -1892,3 +1892,13 @@ int run_auto_maintenance(int quiet) return run_command(&maint); } + +void prepare_other_repo_env(struct strvec *env_array) +{ + const char * const *var; + + for (var = local_repo_env; *var; var++) { + if (strcmp(*var, CONFIG_DATA_ENVIRONMENT)) + strvec_push(env_array, *var); + } +} diff --git a/run-command.h b/run-command.h index d08414a92e..a1d9107f5b 100644 --- a/run-command.h +++ b/run-command.h @@ -483,4 +483,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn, task_finished_fn, void *pp_cb, const char *tr2_category, const char *tr2_label); +/** + * Convenience function which adds all GIT_* environment variables to env_array + * with the exception of GIT_CONFIG_PARAMETERS. When used as the env_array of a + * subprocess, these entries cause the corresponding environment variables to + * be unset in the subprocess. See local_repo_env in cache.h for more + * information. + */ +void prepare_other_repo_env(struct strvec *env_array); + #endif diff --git a/submodule.c b/submodule.c index 0b1d9c1dde..a30216db52 100644 --- a/submodule.c +++ b/submodule.c @@ -484,26 +484,16 @@ static void print_submodule_diff_summary(struct repository *r, struct rev_info * strbuf_release(&sb); } -static void prepare_submodule_repo_env_no_git_dir(struct strvec *out) -{ - const char * const *var; - - for (var = local_repo_env; *var; var++) { - if (strcmp(*var, CONFIG_DATA_ENVIRONMENT)) - strvec_push(out, *var); - } -} - void prepare_submodule_repo_env(struct strvec *out) { - prepare_submodule_repo_env_no_git_dir(out); + prepare_other_repo_env(out); strvec_pushf(out, "%s=%s", GIT_DIR_ENVIRONMENT, DEFAULT_GIT_DIR_ENVIRONMENT); } static void prepare_submodule_repo_env_in_gitdir(struct strvec *out) { - prepare_submodule_repo_env_no_git_dir(out); + prepare_other_repo_env(out); strvec_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT); } From patchwork Tue Jun 8 00:25:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 12304913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6F4CC47095 for ; Tue, 8 Jun 2021 00:27:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C066B6101A for ; Tue, 8 Jun 2021 00:27:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230525AbhFHA3O (ORCPT ); Mon, 7 Jun 2021 20:29:14 -0400 Received: from mail-pj1-f73.google.com ([209.85.216.73]:38669 "EHLO mail-pj1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230514AbhFHA3N (ORCPT ); Mon, 7 Jun 2021 20:29:13 -0400 Received: by mail-pj1-f73.google.com with SMTP id e12-20020a17090a630cb029016de1736f41so6222904pjj.3 for ; Mon, 07 Jun 2021 17:27:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=/PXpxErsW2UOsrNxkS34n5yXSF3eO2RUITunVvXEVnQ=; b=FKIZNDejUlXBfU0IPhyCfOeJAkcd6RPBWrpJZItUoqTJslt4fI443juit1WRz0sX5n GuUuchbaMUqSNVNuL/oWdKL/bm6yR0qUT31hBy1HWpZhQm3GpYz1rCy1FqPOJOX8icBv oB2duCWwEDlWNCvYDeXGPAT2B7lLNPNZVdGtGH24sjPHNZ6cIPnyNdtQhzMpyPl5PBYV IINSWQsr0B9yhwULioUwxu9Wn5sYX2ZY4aP0Oth1T4Kk4+8UN/KYguA9089mmdAoH8Sa zRfiaDXYCglQvP0N6t6WrndMYubdxa7m64fF0JM3vVUlzVKoe3OeU9DFCqO/wRWxpjBn /s2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=/PXpxErsW2UOsrNxkS34n5yXSF3eO2RUITunVvXEVnQ=; b=njSeYe0f1AN3xuTYz1UxZh2Lbt/Wpk+YcMkr/crz4Hh/RxJs4W73P6oxz2eUiI7F1J k81g/P1828zUWhV4WJ65Z7sjhCYbut5+DtEPr+naPEExFkU/yQvSZlJpH4gI2gkvGlSF p+GpjKthyzOF4p/71lS9rFI2K2uRxVv7qa/hlxHGCo3ePCP3bdZXQkKY2MG9mBfIsB4H mXRMUqxbYhoFV1FLCwDDO7t1U7bXUoLwl/mP0LnJ0HtCPGma5UU7WhHUV8y7n3knCNWy tqSlAzhGmYCAv7Aiwz63xca8WKDC+0zT13nUTLQMhJp6SbspFt2k/b/hQJUmWSvqYSsU 8jEA== X-Gm-Message-State: AOAM5330kgQp8Ixr4cgCqT0qief93uNOi6dKp/HdHSTUizUSa1097hf2 fnMQLmoWctU/sxFKSdTWd0JA6OfTM1GbGkgIlWBeQM0UrkZMARM5HCQN87Z7YLQhmZDeWyZtFvp njWMYSbFdylJJDcAKB/H1Qqw7gsLGZwNQ3nFFMxCjGQNfF7rMIQaRL+P05hXd5xhp8wchh4nyHa M4 X-Google-Smtp-Source: ABdhPJxziIa7JK4aetqzLaQIzKRSdP4vuHMwmhxGrXyMuoaIxgT34qOpWMnas81oBnYo4p5oaj2G9Pa0OgQhL5ytuLLA X-Received: from twelve4.c.googlers.com ([fda3:e722:ac3:10:24:72f4:c0a8:437a]) (user=jonathantanmy job=sendgmr) by 2002:a63:2484:: with SMTP id k126mr19765966pgk.1.1623111969291; Mon, 07 Jun 2021 17:26:09 -0700 (PDT) Date: Mon, 7 Jun 2021 17:25:59 -0700 In-Reply-To: Message-Id: <5b41569aced7fbd95ee2d0b4e871d03ffd8dcd67.1623111879.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.32.0.rc1.229.g3e70b5a671-goog Subject: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , me@ttaylorr.com, newren@gmail.com, emilyshaffer@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This is one step towards supporting partial clone submodules. Even after this patch, we will still lack partial clone submodules support, primarily because a lot of Git code that accesses submodule objects does so by adding their object stores as alternates, meaning that any lazy fetches that would occur in the submodule would be done based on the config of the superproject, not of the submodule. This also prevents testing of the functionality in this patch by user-facing commands. So for now, test this mechanism using a test helper. Signed-off-by: Jonathan Tan --- Makefile | 1 + object-file.c | 7 ++---- promisor-remote.c | 14 ++++++++---- t/helper/test-partial-clone.c | 43 +++++++++++++++++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t0410-partial-clone.sh | 23 +++++++++++++++++++ 7 files changed, 80 insertions(+), 10 deletions(-) create mode 100644 t/helper/test-partial-clone.c diff --git a/Makefile b/Makefile index c3565fc0f8..f6653bcd5e 100644 --- a/Makefile +++ b/Makefile @@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o TEST_BUILTINS_OBJS += test-online-cpus.o TEST_BUILTINS_OBJS += test-parse-options.o TEST_BUILTINS_OBJS += test-parse-pathspec-file.o +TEST_BUILTINS_OBJS += test-partial-clone.o TEST_BUILTINS_OBJS += test-path-utils.o TEST_BUILTINS_OBJS += test-pcre2-config.o TEST_BUILTINS_OBJS += test-pkt-line.o diff --git a/object-file.c b/object-file.c index f233b440b2..ebf273e9e7 100644 --- a/object-file.c +++ b/object-file.c @@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r, } /* Check if it is a missing object */ - if (fetch_if_missing && has_promisor_remote() && - !already_retried && r == the_repository && + if (fetch_if_missing && repo_has_promisor_remote(r) && + !already_retried && !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) { /* * TODO Investigate checking promisor_remote_get_direct() * TODO return value and stopping on error here. - * TODO Pass a repository struct through - * promisor_remote_get_direct(), such that arbitrary - * repositories work. */ promisor_remote_get_direct(r, real, 1); already_retried = 1; diff --git a/promisor-remote.c b/promisor-remote.c index e1e1f7e93a..1491374d65 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -11,7 +11,8 @@ struct promisor_remote_config { struct promisor_remote **promisors_tail; }; -static int fetch_objects(const char *remote_name, +static int fetch_objects(struct repository *repo, + const char *remote_name, const struct object_id *oids, int oid_nr) { @@ -21,6 +22,11 @@ static int fetch_objects(const char *remote_name, child.git_cmd = 1; child.in = -1; + if (repo != the_repository) { + prepare_other_repo_env(&child.env_array); + strvec_pushf(&child.env_array, "%s=%s", GIT_DIR_ENVIRONMENT, + repo->gitdir); + } strvec_pushl(&child.args, "-c", "fetch.negotiationAlgorithm=noop", "fetch", remote_name, "--no-tags", "--no-write-fetch-head", "--recurse-submodules=no", @@ -154,7 +160,7 @@ static void promisor_remote_init(struct repository *r) xcalloc(sizeof(*r->promisor_remote_config), 1); config->promisors_tail = &config->promisors; - git_config(promisor_remote_config, config); + repo_config(r, promisor_remote_config, config); if (config->repository_format_partial_clone) { struct promisor_remote *o, *previous; @@ -252,10 +258,8 @@ int promisor_remote_get_direct(struct repository *repo, promisor_remote_init(repo); - if (repo != the_repository) - BUG("only the_repository is supported for now"); for (r = repo->promisor_remote_config->promisors; r; r = r->next) { - if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) { + if (fetch_objects(repo, r->name, remaining_oids, remaining_nr) < 0) { if (remaining_nr == 1) continue; remaining_nr = remove_fetched_oids(repo, &remaining_oids, diff --git a/t/helper/test-partial-clone.c b/t/helper/test-partial-clone.c new file mode 100644 index 0000000000..3f102cfddd --- /dev/null +++ b/t/helper/test-partial-clone.c @@ -0,0 +1,43 @@ +#include "cache.h" +#include "test-tool.h" +#include "repository.h" +#include "object-store.h" + +/* + * Prints the size of the object corresponding to the given hash in a specific + * gitdir. This is similar to "git -C gitdir cat-file -s", except that this + * exercises the code that accesses the object of an arbitrary repository that + * is not the_repository. ("git -C gitdir" makes it so that the_repository is + * the one in gitdir.) + */ +static void object_info(const char *gitdir, const char *oid_hex) +{ + struct repository r; + struct object_id oid; + unsigned long size; + struct object_info oi = {.sizep = &size}; + const char *p; + + if (repo_init(&r, gitdir, NULL)) + die("could not init repo"); + if (parse_oid_hex(oid_hex, &oid, &p)) + die("could not parse oid"); + if (oid_object_info_extended(&r, &oid, &oi, 0)) + die("could not obtain object info"); + printf("%d\n", (int) size); +} + +int cmd__partial_clone(int argc, const char **argv) +{ + setup_git_directory(); + + if (argc < 4) + die("too few arguments"); + + if (!strcmp(argv[1], "object-info")) + object_info(argv[2], argv[3]); + else + die("invalid argument '%s'", argv[1]); + + return 0; +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index c5bd0c6d4c..b21e8f1519 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -46,6 +46,7 @@ static struct test_cmd cmds[] = { { "online-cpus", cmd__online_cpus }, { "parse-options", cmd__parse_options }, { "parse-pathspec-file", cmd__parse_pathspec_file }, + { "partial-clone", cmd__partial_clone }, { "path-utils", cmd__path_utils }, { "pcre2-config", cmd__pcre2_config }, { "pkt-line", cmd__pkt_line }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index e8069a3b22..f845ced4b3 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv); int cmd__online_cpus(int argc, const char **argv); int cmd__parse_options(int argc, const char **argv); int cmd__parse_pathspec_file(int argc, const char** argv); +int cmd__partial_clone(int argc, const char **argv); int cmd__path_utils(int argc, const char **argv); int cmd__pcre2_config(int argc, const char **argv); int cmd__pkt_line(int argc, const char **argv); diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 584a039b85..a211a66c67 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -604,6 +604,29 @@ test_expect_success 'do not fetch when checking existence of tree we construct o git -C repo cherry-pick side1 ' +test_expect_success 'lazy-fetch when accessing object not in the_repository' ' + rm -rf full partial.git && + test_create_repo full && + test_commit -C full create-a-file file.txt && + + test_config -C full uploadpack.allowfilter 1 && + test_config -C full uploadpack.allowanysha1inwant 1 && + git clone --filter=blob:none --bare "file://$(pwd)/full" partial.git && + FILE_HASH=$(git -C full rev-parse HEAD:file.txt) && + + # Sanity check that the file is missing + git -C partial.git rev-list --objects --missing=print HEAD >out && + grep "[?]$FILE_HASH" out && + + git -C full cat-file -s "$FILE_HASH" >expect && + test-tool partial-clone object-info partial.git "$FILE_HASH" >actual && + test_cmp expect actual && + + # Sanity check that the file is now present + git -C partial.git rev-list --objects --missing=print HEAD >out && + ! grep "[?]$FILE_HASH" out +' + . "$TEST_DIRECTORY"/lib-httpd.sh start_httpd