From patchwork Thu Apr 22 15:17:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12218657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6323C433B4 for ; Thu, 22 Apr 2021 15:18:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 68FF66144A for ; Thu, 22 Apr 2021 15:18:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237734AbhDVPSm (ORCPT ); Thu, 22 Apr 2021 11:18:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236545AbhDVPSj (ORCPT ); Thu, 22 Apr 2021 11:18:39 -0400 Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9F6AC06174A for ; Thu, 22 Apr 2021 08:18:04 -0700 (PDT) Received: by mail-qt1-x831.google.com with SMTP id 1so34093078qtb.0 for ; Thu, 22 Apr 2021 08:18:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=C/38sFQa6zRAfOZz7FManGZbEWRjFsxK9Oe9+m5j86c=; b=Su4NFkbNk9wBRvUQK5aOPnH6vd3zVwDcKprz/eskLtKI9ZVdeFA4DRgnAZftO4fWt4 4DcFMxa5QE5obK5qmEJBnUle2Iz3Uro8oqatdU3tDt82nHVxvzbe9OQnGOSytA+7R5H5 RklRqtRMX7LR5Fgo56YvhSlDOsLAb8fSNXX3JhjIUE5pnAQieLPHO6QFp5AwegcUjE7x /SqI4sVcyGK3Sp9vS5urv+iyUF0Vh4x3bs3utkRY2D7mPzeHlEINxga/fdVUXYJc+2i3 aKmqaem8SYChzeTjCtl+FAiJiW3vE7F4Of3GAKlPiQBmdiOgHWkq0N9PjKmYdKTr+ZeE ZCOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=C/38sFQa6zRAfOZz7FManGZbEWRjFsxK9Oe9+m5j86c=; b=iJKiyuHcqnTniyv5NVXxrcyNx6ukCjvYLayuJPRr90CpZqGJgY/9EWMQaHoQAWO6zg pgufa2BlWxO64kf7b7KqblLGXvgdABftXOG0noQd8N2SQZ9mG3zSKYUB2qnFEZGFdZ7j Ll8eF5WA2v1B/4aRvT/SYHvhseCiV1b5byQAcTPkDJQJ8vCUGaT8Fq3RGTGq0Xlul0Jt XyHP7BlZUPs0M/bSIW7ybDE84VqsMVw+WALobLLoD78299traWg5SgGlaT2utWsNZyv8 72zxbKAPO1yxKoD/GvYIaG4AzQzi27LkbDoVQTiRDxfph/QaTHpz33NDvhG/4+Y4j9f4 awaA== X-Gm-Message-State: AOAM533CFPCD3hv2AAaQf2hU0iYRaPeEUp2rDVr9DJHHnkx+M5HDbAlL XXbI1JJ+V04mTsq/OMw8o5WBcR9tMR1Mow== X-Google-Smtp-Source: ABdhPJyx3jINAsYMWH5mBQUBO/bIqDUpltxamNBVGepsAAsY+HNSCj9QXtVghVdtyjH4IAv6tHc8eQ== X-Received: by 2002:ac8:470e:: with SMTP id f14mr3721214qtp.54.1619104683641; Thu, 22 Apr 2021 08:18:03 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id l16sm2348909qkg.91.2021.04.22.08.18.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Apr 2021 08:18:03 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com Subject: [PATCH 1/7] make_transient_cache_entry(): optionally alloc from mem_pool Date: Thu, 22 Apr 2021 12:17:47 -0300 Message-Id: X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Allow make_transient_cache_entry() to optionally receive a mem_pool struct in which it should allocate the entry. This will be used in the following patch, to store some transient entries which should persist until parallel checkout finishes. Signed-off-by: Matheus Tavares --- builtin/checkout--worker.c | 2 +- builtin/checkout.c | 2 +- builtin/difftool.c | 2 +- cache.h | 11 ++++++----- read-cache.c | 12 ++++++++---- unpack-trees.c | 2 +- 6 files changed, 18 insertions(+), 13 deletions(-) diff --git a/builtin/checkout--worker.c b/builtin/checkout--worker.c index 31e0de2f7e..289a9b8f89 100644 --- a/builtin/checkout--worker.c +++ b/builtin/checkout--worker.c @@ -39,7 +39,7 @@ static void packet_to_pc_item(const char *buffer, int len, } memset(pc_item, 0, sizeof(*pc_item)); - pc_item->ce = make_empty_transient_cache_entry(fixed_portion->name_len); + pc_item->ce = make_empty_transient_cache_entry(fixed_portion->name_len, NULL); pc_item->ce->ce_namelen = fixed_portion->name_len; pc_item->ce->ce_mode = fixed_portion->ce_mode; memcpy(pc_item->ce->name, variant, pc_item->ce->ce_namelen); diff --git a/builtin/checkout.c b/builtin/checkout.c index 4c696ef480..db667d0267 100644 --- a/builtin/checkout.c +++ b/builtin/checkout.c @@ -291,7 +291,7 @@ static int checkout_merged(int pos, const struct checkout *state, int *nr_checko if (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid)) die(_("Unable to add merge result for '%s'"), path); free(result_buf.ptr); - ce = make_transient_cache_entry(mode, &oid, path, 2); + ce = make_transient_cache_entry(mode, &oid, path, 2, NULL); if (!ce) die(_("make_cache_entry failed for path '%s'"), path); status = checkout_entry(ce, state, NULL, nr_checkouts); diff --git a/builtin/difftool.c b/builtin/difftool.c index ef25729d49..afacbcd581 100644 --- a/builtin/difftool.c +++ b/builtin/difftool.c @@ -323,7 +323,7 @@ static int checkout_path(unsigned mode, struct object_id *oid, struct cache_entry *ce; int ret; - ce = make_transient_cache_entry(mode, oid, path, 0); + ce = make_transient_cache_entry(mode, oid, path, 0, NULL); ret = checkout_entry(ce, state, NULL, NULL); discard_cache_entry(ce); diff --git a/cache.h b/cache.h index 148d9ab5f1..b6b42cc3f3 100644 --- a/cache.h +++ b/cache.h @@ -356,16 +356,17 @@ struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t name_len); /* - * Create a cache_entry that is not intended to be added to an index. - * Caller is responsible for discarding the cache_entry - * with `discard_cache_entry`. + * Create a cache_entry that is not intended to be added to an index. If `mp` + * is not NULL, the entry is allocated within the given memory pool. Caller is + * responsible for discarding "loose" entries with `discard_cache_entry()` and + * the mem_pool with `mem_pool_discard(mp, should_validate_cache_entries())`. */ struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct object_id *oid, const char *path, - int stage); + int stage, struct mem_pool *mp); -struct cache_entry *make_empty_transient_cache_entry(size_t name_len); +struct cache_entry *make_empty_transient_cache_entry(size_t len, struct mem_pool *mp); /* * Discard cache entry. diff --git a/read-cache.c b/read-cache.c index 5a907af2fb..eb389ccb8f 100644 --- a/read-cache.c +++ b/read-cache.c @@ -813,8 +813,10 @@ struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t le return mem_pool__ce_calloc(find_mem_pool(istate), len); } -struct cache_entry *make_empty_transient_cache_entry(size_t len) +struct cache_entry *make_empty_transient_cache_entry(size_t len, struct mem_pool *mp) { + if (mp) + return mem_pool__ce_calloc(mp, len); return xcalloc(1, cache_entry_size(len)); } @@ -848,8 +850,10 @@ struct cache_entry *make_cache_entry(struct index_state *istate, return ret; } -struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct object_id *oid, - const char *path, int stage) +struct cache_entry *make_transient_cache_entry(unsigned int mode, + const struct object_id *oid, + const char *path, int stage, + struct mem_pool *mp) { struct cache_entry *ce; int len; @@ -860,7 +864,7 @@ struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct o } len = strlen(path); - ce = make_empty_transient_cache_entry(len); + ce = make_empty_transient_cache_entry(len, mp); oidcpy(&ce->oid, oid); memcpy(ce->name, path, len); diff --git a/unpack-trees.c b/unpack-trees.c index 4b77e52c6b..fa5b7ab7ee 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1034,7 +1034,7 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, size_t len = traverse_path_len(info, tree_entry_len(n)); struct cache_entry *ce = is_transient ? - make_empty_transient_cache_entry(len) : + make_empty_transient_cache_entry(len, NULL) : make_empty_cache_entry(istate, len); ce->ce_mode = create_ce_mode(n->mode); From patchwork Thu Apr 22 15:17:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12218659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96906C43460 for ; Thu, 22 Apr 2021 15:18:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6925E6144A for ; Thu, 22 Apr 2021 15:18:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237767AbhDVPSn (ORCPT ); Thu, 22 Apr 2021 11:18:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237716AbhDVPSm (ORCPT ); Thu, 22 Apr 2021 11:18:42 -0400 Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BD36C06174A for ; Thu, 22 Apr 2021 08:18:06 -0700 (PDT) Received: by mail-qk1-x729.google.com with SMTP id v7so18560244qkj.13 for ; Thu, 22 Apr 2021 08:18:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rs9dAKAw99xqaRokvRc7Ibbzn0q24C9LzMNXl2kVJQI=; b=xnhGSyFry58qNpo4ilT+Rp+py1adJw2EvjX3Fdv5oaUVSjaz1+N1cOVKB8CwR34WzB +ZFlRBolIXEFa4voF9bGTtWV9zf5XtW10wa7QJwZ7FzCc9iG3kJ4Q4Pqf3iT8C4rHYXL AWZb+wQ0fQAl1CzNIceEHoNdAhcPC1QyT4NA9xkOOrFBIHk1VOkbV1NCUSQEb5JfGORa UZwTIyDGn/1IsHdhVdOEIcD4+6oO+ryGqYUnUo28zg/CRil8ceNz+66I6vz6zs8Hr1Xo 5ixVu7DfPkXVzjhYY+d9fOWr1H51IF3ZdnTItzayFJ+mzm2xGl5atUKtJhL2F6wFWukI /XgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rs9dAKAw99xqaRokvRc7Ibbzn0q24C9LzMNXl2kVJQI=; b=pYq2CPLdZeLO+zmRL3y7fVXXjFvxzhQC0V/W0WrICM3p57Hv83U5RH1A0aUtkPq51m uvv/lJpZ2k6/+8K0ZA8ANB8LyhdgxBKRCSmG5Y3bkBBEKGs3SDBjkHwmV3sc5iBuTCDt OW8ReQ7pCGr+Y/wJoQ2G3zjiW9qvMI78Uxmxne3AF1BTigsAvA/QMlDSffeBhxX/5xrH tgYTX/VDyNOnyAB+wUs8PRaRL+RLgIK5jWI6sXF86dvQggUZxs9fO4kHBdag0ViMGtBa ksrCG1Rnc2i3oPwia1SKzMWcN1W1aUzvA0lbtCJFMdloYfa0+rmAp4fsTVnVXoM50wgV ddgQ== X-Gm-Message-State: AOAM533BV6Ofr/j83i9DxG6k+xhVM1qU2/BjC4WSAAtdE64+A4Pxzd65 ROTyVVAyytoTCS7s/u084rHIkoJttRbh3Q== X-Google-Smtp-Source: ABdhPJzc+3iQwnuM9UljmN1x+NKCVO/rrIx+Z5wP+3eWwj+5wY8MRk3mK3feipxk+Fk1ylK/gPryCQ== X-Received: by 2002:a37:6801:: with SMTP id d1mr4141822qkc.363.1619104685398; Thu, 22 Apr 2021 08:18:05 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id l16sm2348909qkg.91.2021.04.22.08.18.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Apr 2021 08:18:05 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com Subject: [PATCH 2/7] builtin/checkout.c: complete parallel checkout support Date: Thu, 22 Apr 2021 12:17:48 -0300 Message-Id: <5e0dee7beba083159f4277ddcd9e931859239bde.1619104091.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org There is one code path in builtin/checkout.c which still doesn't benefit from parallel checkout because it calls checkout_entry() directly, instead of unpack_trees(). Let's add parallel checkout support for this missing spot as well. Note: the transient cache entries allocated in checkout_merged() are now allocated in a mem_pool which is only discarded after parallel checkout finishes. This is done because the entries need to be valid when run_parallel_checkout() is called. Signed-off-by: Matheus Tavares --- builtin/checkout.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/builtin/checkout.c b/builtin/checkout.c index db667d0267..b71dc08430 100644 --- a/builtin/checkout.c +++ b/builtin/checkout.c @@ -27,6 +27,7 @@ #include "wt-status.h" #include "xdiff-interface.h" #include "entry.h" +#include "parallel-checkout.h" static const char * const checkout_usage[] = { N_("git checkout [] "), @@ -230,7 +231,8 @@ static int checkout_stage(int stage, const struct cache_entry *ce, int pos, return error(_("path '%s' does not have their version"), ce->name); } -static int checkout_merged(int pos, const struct checkout *state, int *nr_checkouts) +static int checkout_merged(int pos, const struct checkout *state, + int *nr_checkouts, struct mem_pool *ce_mem_pool) { struct cache_entry *ce = active_cache[pos]; const char *path = ce->name; @@ -291,11 +293,10 @@ static int checkout_merged(int pos, const struct checkout *state, int *nr_checko if (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid)) die(_("Unable to add merge result for '%s'"), path); free(result_buf.ptr); - ce = make_transient_cache_entry(mode, &oid, path, 2, NULL); + ce = make_transient_cache_entry(mode, &oid, path, 2, ce_mem_pool); if (!ce) die(_("make_cache_entry failed for path '%s'"), path); status = checkout_entry(ce, state, NULL, nr_checkouts); - discard_cache_entry(ce); return status; } @@ -359,16 +360,22 @@ static int checkout_worktree(const struct checkout_opts *opts, int nr_checkouts = 0, nr_unmerged = 0; int errs = 0; int pos; + int pc_workers, pc_threshold; + struct mem_pool ce_mem_pool; state.force = 1; state.refresh_cache = 1; state.istate = &the_index; + mem_pool_init(&ce_mem_pool, 0); + get_parallel_checkout_configs(&pc_workers, &pc_threshold); init_checkout_metadata(&state.meta, info->refname, info->commit ? &info->commit->object.oid : &info->oid, NULL); enable_delayed_checkout(&state); + if (pc_workers > 1) + init_parallel_checkout(); for (pos = 0; pos < active_nr; pos++) { struct cache_entry *ce = active_cache[pos]; if (ce->ce_flags & CE_MATCHED) { @@ -384,10 +391,15 @@ static int checkout_worktree(const struct checkout_opts *opts, &nr_checkouts, opts->overlay_mode); else if (opts->merge) errs |= checkout_merged(pos, &state, - &nr_unmerged); + &nr_unmerged, + &ce_mem_pool); pos = skip_same_name(ce, pos) - 1; } } + if (pc_workers > 1) + errs |= run_parallel_checkout(&state, pc_workers, pc_threshold, + NULL, NULL); + mem_pool_discard(&ce_mem_pool, should_validate_cache_entries()); remove_marked_cache_entries(&the_index, 1); remove_scheduled_dirs(); errs |= finish_delayed_checkout(&state, &nr_checkouts); From patchwork Thu Apr 22 15:17:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12218661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B59F3C43461 for ; Thu, 22 Apr 2021 15:18:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7F1DE6144A for ; Thu, 22 Apr 2021 15:18:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237794AbhDVPSo (ORCPT ); Thu, 22 Apr 2021 11:18:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237751AbhDVPSm (ORCPT ); Thu, 22 Apr 2021 11:18:42 -0400 Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3418AC06174A for ; Thu, 22 Apr 2021 08:18:08 -0700 (PDT) Received: by mail-qk1-x736.google.com with SMTP id s5so37857274qkj.5 for ; Thu, 22 Apr 2021 08:18:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TaxKTevRj9KNItutE8NJIJeb/G3/fDBUHsrFXaVnfbw=; b=BBdjEBjSTdrCf0uYxVu1FqQKsaAdTis6HUSxE+lJHt3AkabcyGaTTjAFzBk6KuJ9Yz omHAMQPjxUprpgNOQaH8F1TVKTQsdiW8yD1pvP7TN06+r7NNofWDH+4cCB+DvStr3wO3 oOkUO069FzmS6DzGHX1i27dDN7/tHiJ/ECKECaIeuQenSBBJ7GNjZ8VT86zVmpbKQ8/X YRaG/HRVAYx3JwpRYXgIOOfFxa3wA59V6G6qe2nrNXQ5eiQ0+/2QSYwfN1HSsQu0Mn+D ymlpXxaHffseiG0NG4I2UOMqbCpUw916iQnJfc+40bPoF4iD1nJOPTIbRIEXkN1kIpL0 21mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TaxKTevRj9KNItutE8NJIJeb/G3/fDBUHsrFXaVnfbw=; b=qDDrS1vTX/jBe4FRLk6r4kOKfZgiqtp3dZWaxxikQHRpHH8oP4TObqgo/pCm3KQ/Pn 0w39Ubebykz0gdcV6Fxp8g2qc0mOHhSXX2y4igS8ZXpgqB2VUEUobCmNtCHcZaNHfXGZ B5jognf/zMigkpCRyKPHB9by6JN9MxAqPAMKhRTJIxY7Z3CPfKkpzUK0R6tVU5BFkLKa h7eFihwPsmmqCyXaNTLu0DDcjnnAGJ7MUTA4pndFRFgDM0fDzEii5hZsDSlJYG0FYRvH xr9kmTLJox9fFoOEdq0LSR1IxVqOwW9eXhm/galbSiOdIKj328hN338I6S1tTOlSpSgN oFSg== X-Gm-Message-State: AOAM533NpxBbnjQDQD/g/AlnwJ2kqi6S6+lipiRlFUODJPqCf0xyl3i6 gAPBGFhXbuTmsp3gowXwVMKhsx2/+g4pgw== X-Google-Smtp-Source: ABdhPJxMaciAYSFVC0glQzBQ6YG5saYIvHeUofppLRV0gxtM4tbLsUyk7NR8GB1M8zj+h5Ye8KrbcQ== X-Received: by 2002:a37:b103:: with SMTP id a3mr4287855qkf.261.1619104687203; Thu, 22 Apr 2021 08:18:07 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id l16sm2348909qkg.91.2021.04.22.08.18.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Apr 2021 08:18:06 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com Subject: [PATCH 3/7] checkout-index: add parallel checkout support Date: Thu, 22 Apr 2021 12:17:49 -0300 Message-Id: <6bf1db2fd8e807f9f259787ef75651d43b61e03a.1619104091.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Note: previously, `checkout_all()` would not return on errors, but instead call `exit()` with a non-zero code. However, it only did that after calling `checkout_entry()` for all index entries, thus not stopping on the first error, but attempting to write the maximum number of entries possible. In order to maintain this behavior we now propagate `checkout_all()`s error status to `cmd_checkout_index()`, so that it can call `run_parallel_checkout()` and attempt to write the queued entries before exiting with the error code. Signed-off-by: Matheus Tavares --- builtin/checkout-index.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/builtin/checkout-index.c b/builtin/checkout-index.c index c0bf4ac1b2..e8a82ea9ed 100644 --- a/builtin/checkout-index.c +++ b/builtin/checkout-index.c @@ -12,6 +12,7 @@ #include "cache-tree.h" #include "parse-options.h" #include "entry.h" +#include "parallel-checkout.h" #define CHECKOUT_ALL 4 static int nul_term_line; @@ -115,7 +116,7 @@ static int checkout_file(const char *name, const char *prefix) return -1; } -static void checkout_all(const char *prefix, int prefix_length) +static int checkout_all(const char *prefix, int prefix_length) { int i, errs = 0; struct cache_entry *last_ce = NULL; @@ -142,11 +143,7 @@ static void checkout_all(const char *prefix, int prefix_length) } if (last_ce && to_tempfile) write_tempfile_record(last_ce->name, prefix); - if (errs) - /* we have already done our error reporting. - * exit with the same code as die(). - */ - exit(128); + return !!errs; } static const char * const builtin_checkout_index_usage[] = { @@ -182,6 +179,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix) int force = 0, quiet = 0, not_new = 0; int index_opt = 0; int err = 0; + int pc_workers, pc_threshold; struct option builtin_checkout_index_options[] = { OPT_BOOL('a', "all", &all, N_("check out all files in the index")), @@ -236,6 +234,10 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix) hold_locked_index(&lock_file, LOCK_DIE_ON_ERROR); } + get_parallel_checkout_configs(&pc_workers, &pc_threshold); + if (pc_workers > 1) + init_parallel_checkout(); + /* Check out named files first */ for (i = 0; i < argc; i++) { const char *arg = argv[i]; @@ -275,12 +277,16 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix) strbuf_release(&buf); } + if (all) + err |= checkout_all(prefix, prefix_length); + + if (pc_workers > 1) + err |= run_parallel_checkout(&state, pc_workers, pc_threshold, + NULL, NULL); + if (err) return 1; - if (all) - checkout_all(prefix, prefix_length); - if (is_lock_file_locked(&lock_file) && write_locked_index(&the_index, &lock_file, COMMIT_LOCK)) die("Unable to write new index file"); From patchwork Thu Apr 22 15:17:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12218663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9346CC433ED for ; Thu, 22 Apr 2021 15:18:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 53AB4613C8 for ; Thu, 22 Apr 2021 15:18:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237881AbhDVPTA (ORCPT ); Thu, 22 Apr 2021 11:19:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47382 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237904AbhDVPSz (ORCPT ); Thu, 22 Apr 2021 11:18:55 -0400 Received: from mail-qk1-x72c.google.com (mail-qk1-x72c.google.com [IPv6:2607:f8b0:4864:20::72c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F6A9C06138E for ; Thu, 22 Apr 2021 08:18:17 -0700 (PDT) Received: by mail-qk1-x72c.google.com with SMTP id 66so7069084qkf.2 for ; Thu, 22 Apr 2021 08:18:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pQ0ZQZDm7kU0tedVttmtXpkGv1JCa4r2oeCDSjzZKX4=; b=lVGRU6p8m4yhFxyhUkiHRMWK2PsoQZSnCQ6UIvySA9yoy9l7ig50vlsBoa56iKBn4C GE9/ZkBOnTk/HSEisjCDA/VQlO1klnrM4+/Oie/ANTjCMc430ddAIE0H/+WaAfc0ZVp/ 71ICggu4c2D/w3GrOPhGlp5sfW2ChAdQbqt5OtN8fp2tE8+GOYZRb3L+urKNgX2g/IX4 u3cbYlRb/BbuTUXeCLjr83GEKWl4e79M9ljPX1uP+vIPZECUWeLd+UI6ECaz4wROMwBG cxl+Q7DYsf9zq/kc+1++Mi+Il6KTmv8rIfhpQgBvvIFRoJ8eYIKYahs9oDjIu/b+9N7k 3MKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pQ0ZQZDm7kU0tedVttmtXpkGv1JCa4r2oeCDSjzZKX4=; b=jo5PXL1IJgAjgGvCIkN0IDTDHLMSns5Ii8WMY+VmMP7knqZ4On2gOmWP7qNkdeGwOD 3ahlcfRHBdzcguh6mzJBo/edgBGyrskPSooc8eUODbXVGR/AibGM1rBkLH2kEXRPpcl9 WV29dj9Dbdk5oHbOlWy0d2qtvu6iiQtBEiozexwAxs/64iHjuZabJxqbGrBVrQ1nkFlL K4c6CZR+oJ171vmBc3d6SAO3xjlBUbsl0TFIlUzOhqM5I+TmWnXRDmfa0yrF6JKy2ESU Y1+xaYD194Uz/A+H9ut3YGFERZNo0oFcr31bjiiPhD1aXRB39t9wDa9vA0JHs4CWQQWP z1cg== X-Gm-Message-State: AOAM531HsJTDBUovpqVx1Vot/TztsZ8LTIoWmejNhQdyZ8mn651FRU3X Y7VTWnJQNEVYMLq0roiuwFhFlbnpaKswSQ== X-Google-Smtp-Source: ABdhPJyrnKBYDWOFvsp0b3fyBlXkOIVIh0le8hDh2hrib5ltKn1Zch549k1iYmqQ3vkEcUHlQbKg1Q== X-Received: by 2002:a37:bc43:: with SMTP id m64mr4199965qkf.186.1619104696604; Thu, 22 Apr 2021 08:18:16 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id l16sm2348909qkg.91.2021.04.22.08.18.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Apr 2021 08:18:15 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, Jonathan Nieder Subject: [PATCH 4/7] parallel-checkout: add tests for basic operations Date: Thu, 22 Apr 2021 12:17:50 -0300 Message-Id: <6379b8df6a59361dd44733e379880a11c6cd977c.1619104091.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add tests to populate the working tree during clone and checkout using sequential and parallel mode, to confirm that they produce identical results. Also test basic checkout mechanics, such as checking for symlinks in the leading directories and the abidance to --force. Note: some helper functions are added to a common lib file which is only included by t2080 for now. But they will also be used by other parallel-checkout tests in the following patches. Original-patch-by: Jeff Hostetler Signed-off-by: Jeff Hostetler Signed-off-by: Matheus Tavares --- t/lib-parallel-checkout.sh | 37 +++++++ t/t2080-parallel-checkout-basics.sh | 150 ++++++++++++++++++++++++++++ 2 files changed, 187 insertions(+) create mode 100644 t/lib-parallel-checkout.sh create mode 100755 t/t2080-parallel-checkout-basics.sh diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh new file mode 100644 index 0000000000..39fd36fdf6 --- /dev/null +++ b/t/lib-parallel-checkout.sh @@ -0,0 +1,37 @@ +# Helpers for t208* tests + +set_checkout_config () { + if test $# -ne 2 + then + BUG "set_checkout_config() requires two arguments" + fi && + + test_config_global checkout.workers $1 && + test_config_global checkout.thresholdForParallelism $2 +} + +# Run "${@:2}" and check that $1 checkout workers were used +test_checkout_workers () { + if test $# -lt 2 + then + BUG "too few arguments to test_checkout_workers()" + fi && + + expected_workers=$1 && + shift && + + rm -f trace && + GIT_TRACE2="$(pwd)/trace" "$@" && + + workers=$(grep "child_start\[..*\] git checkout--worker" trace | wc -l) && + test $workers -eq $expected_workers && + rm -f trace +} + +# Verify that both the working tree and the index were created correctly +verify_checkout () { + git -C "$1" diff-index --quiet HEAD -- && + git -C "$1" diff-index --quiet --cached HEAD -- && + git -C "$1" status --porcelain >"$1".status && + test_must_be_empty "$1".status +} diff --git a/t/t2080-parallel-checkout-basics.sh b/t/t2080-parallel-checkout-basics.sh new file mode 100755 index 0000000000..0cb1493cdc --- /dev/null +++ b/t/t2080-parallel-checkout-basics.sh @@ -0,0 +1,150 @@ +#!/bin/sh + +test_description='parallel-checkout basics + +Ensure that parallel-checkout basically works on clone and checkout, spawning +the required number of workers and correctly populating both the index and the +working tree. +' + +TEST_NO_CREATE_REPO=1 +. ./test-lib.sh +. "$TEST_DIRECTORY/lib-parallel-checkout.sh" + +# Test parallel-checkout with a branch switch containing file creations, +# deletions, and modification; with different entry types. Switching from B1 to +# B2 will have the following changes: +# +# - a (file): modified +# - e/x (file): deleted +# - b (symlink): deleted +# - b/f (file): created +# - e (symlink): created +# - d (submodule): created +# +test_expect_success SYMLINKS 'setup repo for checkout with various types of changes' ' + git init various && + ( + cd various && + git checkout -b B1 && + echo a >a && + mkdir e && + echo e/x >e/x && + ln -s e b && + git add -A && + git commit -m B1 && + + git checkout -b B2 && + echo modified >a && + rm -rf e && + rm b && + mkdir b && + echo b/f >b/f && + ln -s b e && + git init d && + test_commit -C d f && + git submodule add ./d && + git add -A && + git commit -m B2 && + + git checkout --recurse-submodules B1 + ) +' + +test_expect_success SYMLINKS 'sequential checkout' ' + cp -R various various_sequential && + set_checkout_config 1 0 && + test_checkout_workers 0 \ + git -C various_sequential checkout --recurse-submodules B2 && + verify_checkout various_sequential +' + +test_expect_success SYMLINKS 'parallel checkout' ' + cp -R various various_parallel && + set_checkout_config 2 0 && + test_checkout_workers 2 \ + git -C various_parallel checkout --recurse-submodules B2 && + verify_checkout various_parallel +' + +test_expect_success SYMLINKS 'fallback to sequential checkout (threshold)' ' + cp -R various various_sequential_fallback && + set_checkout_config 2 100 && + test_checkout_workers 0 \ + git -C various_sequential_fallback checkout --recurse-submodules B2 && + verify_checkout various_sequential_fallback +' + +test_expect_success SYMLINKS 'parallel checkout on clone' ' + git -C various checkout --recurse-submodules B2 && + set_checkout_config 2 0 && + test_checkout_workers 2 \ + git clone --recurse-submodules various various_parallel_clone && + verify_checkout various_parallel_clone +' + +test_expect_success SYMLINKS 'fallback to sequential checkout on clone (threshold)' ' + git -C various checkout --recurse-submodules B2 && + set_checkout_config 2 100 && + test_checkout_workers 0 \ + git clone --recurse-submodules various various_sequential_fallback_clone && + verify_checkout various_sequential_fallback_clone +' + +# Just to be paranoid, actually compare the working trees' contents directly. +test_expect_success SYMLINKS 'compare the working trees' ' + rm -rf various_*/.git && + rm -rf various_*/d/.git && + + diff -r various_sequential various_parallel && + diff -r various_sequential various_sequential_fallback && + diff -r various_sequential various_parallel_clone && + diff -r various_sequential various_sequential_fallback_clone +' + +test_expect_success 'parallel checkout respects --[no]-force' ' + set_checkout_config 2 0 && + git init dirty && + ( + cd dirty && + mkdir D && + test_commit D/F && + test_commit F && + + rm -rf D && + echo changed >D && + echo changed >F.t && + + # We expect 0 workers because there is nothing to be done + test_checkout_workers 0 git checkout HEAD && + test_path_is_file D && + grep changed D && + grep changed F.t && + + test_checkout_workers 2 git checkout --force HEAD && + test_path_is_dir D && + grep D/F D/F.t && + grep F F.t + ) +' + +test_expect_success SYMLINKS 'parallel checkout checks for symlinks in leading dirs' ' + set_checkout_config 2 0 && + git init symlinks && + ( + cd symlinks && + mkdir D untracked && + # Commit 2 files to have enough work for 2 parallel workers + test_commit D/A && + test_commit D/B && + rm -rf D && + ln -s untracked D && + + test_checkout_workers 2 git checkout --force HEAD && + ! test -h D && + grep D/A D/A.t && + grep D/B D/B.t + ) +' + +test_done From patchwork Thu Apr 22 15:17:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12218665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 800C3C433B4 for ; Thu, 22 Apr 2021 15:18:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 477EB61425 for ; Thu, 22 Apr 2021 15:18:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237904AbhDVPTC (ORCPT ); Thu, 22 Apr 2021 11:19:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237919AbhDVPSz (ORCPT ); Thu, 22 Apr 2021 11:18:55 -0400 Received: from mail-qv1-xf33.google.com (mail-qv1-xf33.google.com [IPv6:2607:f8b0:4864:20::f33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9780BC06138F for ; Thu, 22 Apr 2021 08:18:19 -0700 (PDT) Received: by mail-qv1-xf33.google.com with SMTP id l2so4281062qvb.7 for ; Thu, 22 Apr 2021 08:18:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=P4rxRcR9+uJuRupxBojB6SFa/+ZbjggBCNEtBkv/ZiM=; b=QpKABmlFVkdJs77Jo2Q9I2KDPJxi01UMzXUKh7fhRHlRuobpznXu2531ubp8ihvtUo UFBIn4AW+rQqxYvHT9UyemiEIVAcU16rHURe3HRYjNr93TnRVbdM8OQ6Ae07rJbirKEw dGYYfbyH7kaHIB2f6aIIqZIr5C7jsF4oZ1Lr4/zI1bLTPKRG183E8FwmMq44nTsRuDBx 7V/evwK7AB8Q8cE8rCS6rjXxfVziccRUAf5AkRzOSrpLv1wEcJPT8kEtv0GA0CDEJUsv 7xpNECwiP4Bjcjz7j0MZCm99UWJ0j45+YSLS0x4NG+OLxQgiBo8eckZzw9p6JDYSOv89 GP0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P4rxRcR9+uJuRupxBojB6SFa/+ZbjggBCNEtBkv/ZiM=; b=P7sJTX9BD280zAqvNxEsejW8SEdI7dm6EjXnikSytIxJIfgE2swptMdDvGiTp+j5Nl At+tU+7WlSk1Wy3L1P0a8XIYchlUj3JLrfW9TYRm5pLEoK6QDLw67W1mncGfr01ZaRMa qBZmiPwM56BDRVZ1Vjt042Os+ZO7/Mt1UdQV24beG9XiZIrAvcFN6w58y0G8bseCaJ9l +Si8yJ3mz2nkZGppunFMShLmqNwgAeX1giahs/v68Q2jlXDxdwV66+Crmnmhc6BzlJyp Ftb91aotBvmQpDKzIcsWUzGpZNHXwgF83Ej8NNShpbgmxw14mo2KOeNn9G7e1W2rHaLv Afpw== X-Gm-Message-State: AOAM533O7RwSTUcbOig2CG7xwSnxuTkQ4RbSSTCZXFRzZmpVljWrrpCk vc8gK7ktyRe8jpQTJUvA1tgtluDovBpKSg== X-Google-Smtp-Source: ABdhPJwrtd4LkV6LqteEg9c7Ji5qropUc37McWwg1Gx3in4inO7WJ1S1KsfVE9iFDWSGzD/C+mPVbQ== X-Received: by 2002:a0c:b410:: with SMTP id u16mr3962686qve.8.1619104698482; Thu, 22 Apr 2021 08:18:18 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id l16sm2348909qkg.91.2021.04.22.08.18.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Apr 2021 08:18:18 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com Subject: [PATCH 5/7] parallel-checkout: add tests related to path collisions Date: Thu, 22 Apr 2021 12:17:51 -0300 Message-Id: X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add tests to confirm that path collisions are properly detected by checkout workers, both to avoid race conditions and to report colliding entries on clone. Original-patch-by: Jeff Hostetler Signed-off-by: Jeff Hostetler Signed-off-by: Matheus Tavares --- parallel-checkout.c | 4 + t/lib-parallel-checkout.sh | 4 +- t/t2081-parallel-checkout-collisions.sh | 162 ++++++++++++++++++++++++ 3 files changed, 168 insertions(+), 2 deletions(-) create mode 100755 t/t2081-parallel-checkout-collisions.sh diff --git a/parallel-checkout.c b/parallel-checkout.c index 09e8b10a35..6fb3f1e6c9 100644 --- a/parallel-checkout.c +++ b/parallel-checkout.c @@ -8,6 +8,7 @@ #include "sigchain.h" #include "streaming.h" #include "thread-utils.h" +#include "trace2.h" struct pc_worker { struct child_process cp; @@ -326,6 +327,7 @@ void write_pc_item(struct parallel_checkout_item *pc_item, if (dir_sep && !has_dirs_only_path(path.buf, dir_sep - path.buf, state->base_dir_len)) { pc_item->status = PC_ITEM_COLLIDED; + trace2_data_string("pcheckout", NULL, "collision/dirname", path.buf); goto out; } @@ -341,6 +343,8 @@ void write_pc_item(struct parallel_checkout_item *pc_item, * call should have already caught these cases. */ pc_item->status = PC_ITEM_COLLIDED; + trace2_data_string("pcheckout", NULL, + "collision/basename", path.buf); } else { error_errno("failed to open file '%s'", path.buf); pc_item->status = PC_ITEM_FAILED; diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh index 39fd36fdf6..16ee18389b 100644 --- a/t/lib-parallel-checkout.sh +++ b/t/lib-parallel-checkout.sh @@ -21,12 +21,12 @@ test_checkout_workers () { shift && rm -f trace && - GIT_TRACE2="$(pwd)/trace" "$@" && + GIT_TRACE2="$(pwd)/trace" "$@" 2>&8 && workers=$(grep "child_start\[..*\] git checkout--worker" trace | wc -l) && test $workers -eq $expected_workers && rm -f trace -} +} 8>&2 2>&4 # Verify that both the working tree and the index were created correctly verify_checkout () { diff --git a/t/t2081-parallel-checkout-collisions.sh b/t/t2081-parallel-checkout-collisions.sh new file mode 100755 index 0000000000..f6fcfc0c1e --- /dev/null +++ b/t/t2081-parallel-checkout-collisions.sh @@ -0,0 +1,162 @@ +#!/bin/sh + +test_description="path collisions during parallel checkout + +Parallel checkout must detect path collisions to: + +1) Avoid racily writing to different paths that represent the same file on disk. +2) Report the colliding entries on clone. + +The tests in this file exercise parallel checkout's collision detection code in +both these mechanics. +" + +. ./test-lib.sh +. "$TEST_DIRECTORY/lib-parallel-checkout.sh" + +TEST_ROOT="$PWD" + +test_expect_success CASE_INSENSITIVE_FS 'setup' ' + empty_oid=$(git hash-object -w --stdin objs <<-EOF && + 100644 $empty_oid FILE_X + 100644 $empty_oid FILE_x + 100644 $empty_oid file_X + 100644 $empty_oid file_x + EOF + git update-index --index-info >filter.log + EOF +' + +test_workers_in_event_trace () +{ + test $1 -eq $(grep ".event.:.child_start..*checkout--worker" $2 | wc -l) +} + +test_expect_success CASE_INSENSITIVE_FS 'worker detects basename collision' ' + GIT_TRACE2_EVENT="$(pwd)/trace" git \ + -c checkout.workers=2 -c checkout.thresholdForParallelism=0 \ + checkout . && + + test_workers_in_event_trace 2 trace && + collisions=$(grep -i "category.:.pcheckout.,.key.:.collision/basename.,.value.:.file_x.}" trace | wc -l) && + test $collisions -eq 3 +' + +test_expect_success CASE_INSENSITIVE_FS 'worker detects dirname collision' ' + test_config filter.logger.smudge "\"$TEST_ROOT/logger_script\" %f" && + empty_oid=$(git hash-object -w --stdin objs <<-EOF && + 100644 $empty_oid A/B + 100644 $empty_oid A/C + 100644 $empty_oid a + 100644 $attr_oid .gitattributes + EOF + git rm -rf . && + git update-index --index-info expected.log && + test_cmp filter.log expected.log && + + # Check that it used the right number of workers and detected the collisions + test_workers_in_event_trace 2 trace && + grep "category.:.pcheckout.,.key.:.collision/dirname.,.value.:.A/B.}" trace && + grep "category.:.pcheckout.,.key.:.collision/dirname.,.value.:.A/C.}" trace +' + +test_expect_success SYMLINKS,CASE_INSENSITIVE_FS 'do not follow symlinks colliding with leading dir' ' + empty_oid=$(git hash-object -w --stdin objs <<-EOF && + 120000 $symlink_oid D + 100644 $empty_oid d/x + 100644 $empty_oid e/y + EOF + git rm -rf . && + git update-index --index-info stderr && + + grep FILE_X stderr && + grep FILE_x stderr && + grep file_X stderr && + grep file_x stderr && + grep "the following paths have collided" stderr +' + +# This test ensures that the collision report code is correctly looking for +# colliding peers in the second half of the cache_entry array. This is done by +# defining a smudge command for the *last* array entry, which makes it +# non-eligible for parallel-checkout. Thus, it is checked out *first*, before +# spawning the workers. +# +# Note: this test doesn't work on Windows because, on this system, the +# collision report code uses strcmp() to find the colliding pairs when +# core.ignoreCase is false. And we need this setting for this test so that only +# 'file_x' matches the pattern of the filter attribute. But the test works on +# OSX, where the colliding pairs are found using inode. +# +test_expect_success CASE_INSENSITIVE_FS,!MINGW,!CYGWIN \ + 'collision report on clone (w/ colliding peer after the detected entry)' ' + + test_config_global filter.logger.smudge "\"$TEST_ROOT/logger_script\" %f" && + git reset --hard basename_collision && + echo "file_x filter=logger" >.gitattributes && + git add .gitattributes && + git commit -m "filter for file_x" && + + rm -rf clone-repo && + set_checkout_config 2 0 && + test_checkout_workers 2 \ + git -c core.ignoreCase=false clone . clone-repo 2>stderr && + + grep FILE_X stderr && + grep FILE_x stderr && + grep file_X stderr && + grep file_x stderr && + grep "the following paths have collided" stderr && + + # Check that only "file_x" was filtered + echo file_x >expected.log && + test_cmp clone-repo/filter.log expected.log +' + +test_done From patchwork Thu Apr 22 15:17:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12218667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39419C433ED for ; Thu, 22 Apr 2021 15:18:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0738B61222 for ; Thu, 22 Apr 2021 15:18:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237925AbhDVPTE (ORCPT ); Thu, 22 Apr 2021 11:19:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237921AbhDVPS4 (ORCPT ); Thu, 22 Apr 2021 11:18:56 -0400 Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 67EBFC06174A for ; Thu, 22 Apr 2021 08:18:21 -0700 (PDT) Received: by mail-qk1-x72b.google.com with SMTP id 66so7069339qkf.2 for ; Thu, 22 Apr 2021 08:18:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jJwbUc78YqmzkapRpqUxn8jTO7PGV0uZYBEfCqFIw+E=; b=LxGSdPVqnD1iP5Qtdn1lk2QIrV58c55DpX1YWD+LFMiBc0ZNtm1ACMDrydCm5eprNe jyHqYtUfSIfrK2YJXcitwfUftla3xnkg99I2ejKIsI64viQcsrd+on49JXkErCkyK5HL ML1i5R+q1jB2de1B8+veAEONEQcp1pxgozFeImU6EfLsw7L2hUg0BEhG/YXlsPBFH8qn MPnm122Wi+ZWci1Q7PWwahyrLNC/MDQsIk+ZkB8o/zm7DP8Hx7/dGAefWN8sK1UPFzon DiYAV76+CkcaJNJtvk+z7VhyYVIHRuNAPBj1m6fET0Htj4Cj7yyitT6pZPlCwX3XnWGO ZsJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jJwbUc78YqmzkapRpqUxn8jTO7PGV0uZYBEfCqFIw+E=; b=n+rxwlfa1AW/XxRVRQqU46NkHHEX9y2/eZUAEtSj08UlgCNsg7Be7+SX2r6bzrGH+w ZdZC9kizKXIpj12+8A+/jY9gTWvm9IQWb7/NOqkcDUnLbvSXNLmstj2bP0Qv9we1NSUe SmKFWnMOs2DFfMUamG9YCSzRwZifTyhTptORO+6LuYxV3tz0C+J0+VRtaHW6LhSeFrUa V9/dDdt6gY9e9nspwmT1uJClfT6EdapAW3CfEhsTvWHDXn+IMLZU0Z+h0bS6sIf80oEz M46dkP3V2+AY7VcEy66lB+2dkr6xpXxD7zuzSM2DnTP/2aPKfbCZJxr7LcARzdErrLa0 KC3g== X-Gm-Message-State: AOAM533l0Vbi+pvnpy3+a4vUj1edvhH8mSQ9tTideSYl8cXnim2Qoygr i4ljtFvrvF5Lx/b+dmIvHb/RFb+Gd/7cug== X-Google-Smtp-Source: ABdhPJw/AATdordA8uVnjLdQ0BKwec46YkV8Ech4ODgwOUI6FQ6i+VnZILaGkr+CgJog47IUWmNf/Q== X-Received: by 2002:a37:6801:: with SMTP id d1mr4143025qkc.363.1619104700234; Thu, 22 Apr 2021 08:18:20 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id l16sm2348909qkg.91.2021.04.22.08.18.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Apr 2021 08:18:19 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com Subject: [PATCH 6/7] parallel-checkout: add tests related to .gitattributes Date: Thu, 22 Apr 2021 12:17:52 -0300 Message-Id: <3a0845fdbb0481ead96b15be472df636bcf73aee.1619104091.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add tests to confirm that the `struct conv_attrs` data is correctly passed from the main process to the workers, and that they can properly convert the blobs before writing them to the working tree. Also check that parallel-ineligible entries, such as regular files that require external filters, are correctly smudge and written when parallel-checkout is enabled. Note: to avoid repeating code, some helper functions are extracted from t0028 into a common lib file. Original-patch-by: Jeff Hostetler Signed-off-by: Jeff Hostetler Signed-off-by: Matheus Tavares --- t/lib-encoding.sh | 25 +++ t/t0028-working-tree-encoding.sh | 25 +-- t/t2082-parallel-checkout-attributes.sh | 194 ++++++++++++++++++++++++ 3 files changed, 220 insertions(+), 24 deletions(-) create mode 100644 t/lib-encoding.sh create mode 100755 t/t2082-parallel-checkout-attributes.sh diff --git a/t/lib-encoding.sh b/t/lib-encoding.sh new file mode 100644 index 0000000000..c52ffbbed5 --- /dev/null +++ b/t/lib-encoding.sh @@ -0,0 +1,25 @@ +# Encoding helpers used by t0028 and t2082 + +test_lazy_prereq NO_UTF16_BOM ' + test $(printf abc | iconv -f UTF-8 -t UTF-16 | wc -c) = 6 +' + +test_lazy_prereq NO_UTF32_BOM ' + test $(printf abc | iconv -f UTF-8 -t UTF-32 | wc -c) = 12 +' + +write_utf16 () { + if test_have_prereq NO_UTF16_BOM + then + printf '\376\377' + fi && + iconv -f UTF-8 -t UTF-16 +} + +write_utf32 () { + if test_have_prereq NO_UTF32_BOM + then + printf '\0\0\376\377' + fi && + iconv -f UTF-8 -t UTF-32 +} diff --git a/t/t0028-working-tree-encoding.sh b/t/t0028-working-tree-encoding.sh index f970a9806b..82905a2156 100755 --- a/t/t0028-working-tree-encoding.sh +++ b/t/t0028-working-tree-encoding.sh @@ -6,33 +6,10 @@ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME . ./test-lib.sh +. "$TEST_DIRECTORY/lib-encoding.sh" GIT_TRACE_WORKING_TREE_ENCODING=1 && export GIT_TRACE_WORKING_TREE_ENCODING -test_lazy_prereq NO_UTF16_BOM ' - test $(printf abc | iconv -f UTF-8 -t UTF-16 | wc -c) = 6 -' - -test_lazy_prereq NO_UTF32_BOM ' - test $(printf abc | iconv -f UTF-8 -t UTF-32 | wc -c) = 12 -' - -write_utf16 () { - if test_have_prereq NO_UTF16_BOM - then - printf '\376\377' - fi && - iconv -f UTF-8 -t UTF-16 -} - -write_utf32 () { - if test_have_prereq NO_UTF32_BOM - then - printf '\0\0\376\377' - fi && - iconv -f UTF-8 -t UTF-32 -} - test_expect_success 'setup test files' ' git config core.eol lf && diff --git a/t/t2082-parallel-checkout-attributes.sh b/t/t2082-parallel-checkout-attributes.sh new file mode 100755 index 0000000000..2525457961 --- /dev/null +++ b/t/t2082-parallel-checkout-attributes.sh @@ -0,0 +1,194 @@ +#!/bin/sh + +test_description='parallel-checkout: attributes + +Verify that parallel-checkout correctly creates files that require +conversions, as specified in .gitattributes. The main point here is +to check that the conv_attr data is correctly sent to the workers +and that it contains sufficient information to smudge files +properly (without access to the index or attribute stack). +' + +TEST_NO_CREATE_REPO=1 +. ./test-lib.sh +. "$TEST_DIRECTORY/lib-parallel-checkout.sh" +. "$TEST_DIRECTORY/lib-encoding.sh" + +test_expect_success 'parallel-checkout with ident' ' + set_checkout_config 2 0 && + git init ident && + ( + cd ident && + echo "A ident" >.gitattributes && + echo "\$Id\$" >A && + echo "\$Id\$" >B && + git add -A && + git commit -m id && + + rm A B && + test_checkout_workers 2 git reset --hard && + hexsz=$(test_oid hexsz) && + grep -E "\\\$Id: [0-9a-f]{$hexsz} \\\$" A && + grep "\\\$Id\\\$" B + ) +' + +test_expect_success 'parallel-checkout with re-encoding' ' + set_checkout_config 2 0 && + git init encoding && + ( + cd encoding && + echo text >utf8-text && + write_utf16 utf16-text && + + echo "A working-tree-encoding=UTF-16" >.gitattributes && + cp utf16-text A && + cp utf8-text B && + git add A B .gitattributes && + git commit -m encoding && + + # Check that A is stored in UTF-8 + git cat-file -p :A >A.internal && + test_cmp_bin utf8-text A.internal && + + rm A B && + test_checkout_workers 2 git checkout A B && + + # Check that A (and only A) is re-encoded during checkout + test_cmp_bin utf16-text A && + test_cmp_bin utf8-text B + ) +' + +test_expect_success 'parallel-checkout with eol conversions' ' + set_checkout_config 2 0 && + git init eol && + ( + cd eol && + printf "multi\r\nline\r\ntext" >crlf-text && + printf "multi\nline\ntext" >lf-text && + + git config core.autocrlf false && + echo "A eol=crlf" >.gitattributes && + cp crlf-text A && + cp lf-text B && + git add A B .gitattributes && + git commit -m eol && + + # Check that A is stored with LF format + git cat-file -p :A >A.internal && + test_cmp_bin lf-text A.internal && + + rm A B && + test_checkout_workers 2 git checkout A B && + + # Check that A (and only A) is converted to CRLF during checkout + test_cmp_bin crlf-text A && + test_cmp_bin lf-text B + ) +' + +# Entries that require an external filter are not eligible for parallel +# checkout. Check that both the parallel-eligible and non-eligible entries are +# properly writen in a single checkout operation. +# +test_expect_success 'parallel-checkout and external filter' ' + set_checkout_config 2 0 && + git init filter && + ( + cd filter && + write_script <<-\EOF rot13.sh && + tr \ + "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ" \ + "nopqrstuvwxyzabcdefghijklmNOPQRSTUVWXYZABCDEFGHIJKLM" + EOF + + git config filter.rot13.clean "\"$(pwd)/rot13.sh\"" && + git config filter.rot13.smudge "\"$(pwd)/rot13.sh\"" && + git config filter.rot13.required true && + + echo abcd >original && + echo nopq >rot13 && + + echo "A filter=rot13" >.gitattributes && + cp original A && + cp original B && + cp original C && + git add A B C .gitattributes && + git commit -m filter && + + # Check that A (and only A) was cleaned + git cat-file -p :A >A.internal && + test_cmp rot13 A.internal && + git cat-file -p :B >B.internal && + test_cmp original B.internal && + git cat-file -p :C >C.internal && + test_cmp original C.internal && + + rm A B C *.internal && + test_checkout_workers 2 git checkout A B C && + + # Check that A (and only A) was smudged during checkout + test_cmp original A && + test_cmp original B && + test_cmp original C + ) +' + +# The delayed queue is independent from the parallel queue, and they should be +# able to work together in the same checkout process. +# +test_expect_success PERL 'parallel-checkout and delayed checkout' ' + write_script rot13-filter.pl "$PERL_PATH" \ + <"$TEST_DIRECTORY"/t0021/rot13-filter.pl && + + test_config_global filter.delay.process \ + "\"$(pwd)/rot13-filter.pl\" --always-delay \"$(pwd)/delayed.log\" clean smudge delay" && + test_config_global filter.delay.required true && + + echo "abcd" >original && + echo "nopq" >rot13 && + + git init delayed && + ( + cd delayed && + echo "*.d filter=delay" >.gitattributes && + cp ../original W.d && + cp ../original X.d && + cp ../original Y && + cp ../original Z && + git add -A && + git commit -m delayed && + + # Check that *.d files were cleaned + git cat-file -p :W.d >W.d.internal && + test_cmp W.d.internal ../rot13 && + git cat-file -p :X.d >X.d.internal && + test_cmp X.d.internal ../rot13 && + git cat-file -p :Y >Y.internal && + test_cmp Y.internal ../original && + git cat-file -p :Z >Z.internal && + test_cmp Z.internal ../original && + + rm * + ) && + + set_checkout_config 2 0 && + test_checkout_workers 2 git -C delayed checkout -f && + verify_checkout delayed && + + # Check that the *.d files got to the delay queue and were filtered + grep "smudge W.d .* \[DELAYED\]" delayed.log && + grep "smudge X.d .* \[DELAYED\]" delayed.log && + test_cmp delayed/W.d original && + test_cmp delayed/X.d original && + + # Check that the parallel-eligible entries went to the right queue and + # were not filtered + ! grep "smudge Y .* \[DELAYED\]" delayed.log && + ! grep "smudge Z .* \[DELAYED\]" delayed.log && + test_cmp delayed/Y original && + test_cmp delayed/Z original +' + +test_done From patchwork Thu Apr 22 15:17:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12218669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4518C433ED for ; Thu, 22 Apr 2021 15:18:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 80BCD61222 for ; Thu, 22 Apr 2021 15:18:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237931AbhDVPTX (ORCPT ); Thu, 22 Apr 2021 11:19:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237824AbhDVPS6 (ORCPT ); Thu, 22 Apr 2021 11:18:58 -0400 Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12E0FC06138B for ; Thu, 22 Apr 2021 08:18:23 -0700 (PDT) Received: by mail-qv1-xf30.google.com with SMTP id bs7so21671498qvb.12 for ; Thu, 22 Apr 2021 08:18:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/TaqGOy5FFgW09kpy4dpdt/P4d0VaTPgGmFRT+BnrbM=; b=pKKUpVI28X9EiB2Nhji3tj+j4ttP86rsnsVO6gUz+ov7Ts5ag/yVzVzXmMBxW7pjFo HV7xbuJUz+RgMvl9UHFA3LpwSHMY6+TE7XKiIe9x4LlSuMk/LzXNpTX9EXhI7/BPvC7X FRl9SYx8fmRY1ajfaWnddwdX69BILn5+IBZMRfFfd0eDo+s5E2Y3AP5UZPd4YG3sxSRR DbTUi/9vTF9fb8UI+FgFZ56/j6uBDCkrV/ZErNBt9DQjGGnb3PhsQEvtDVsmyQc5ctz+ BNmO14arbgQdvgfZ8xxqCv+xBO4wJkuxpvjL6s7Emz9oY/BsIj1NI37GvqiklDX7VVLN spsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/TaqGOy5FFgW09kpy4dpdt/P4d0VaTPgGmFRT+BnrbM=; b=torPLrWUnmca1lWpTczzwgSBV8FatEtEvqi6It5TGtGxCVr3nGeAOpPSVP6LKBpWxP 8jI1utliv0zGJKj1dJuyP1rYGTRP5lxJ52nXhYvwf1H64QJ1UcqfsHzdgaFB0tVscbaD +eFxFdsp45f8nFUhOpK3ub9soJbLoEWF351FWIvOVcCfQsY2EbQPCcAuwD8uQJEK1XvR V7tLXTYGkEwjXuW/rk4Hj2GLogwe/G86AZuAA/AxHGE3faOGRlBNmFDwI3lPbhUILPzD SC/3tCUS2ui3A8KehSUJ0ZEYYoU6RfFvO4+HP8d4bGh0Th7lPOk4Qyxj5LKMTDhohSuT 0adw== X-Gm-Message-State: AOAM533wYfquqRhf/gaZn1KEzAi9RkAg2SysU/MTFSnGRuzxm0931Bul 7Xc7HmhAC7XTpPyFVGrPnrN08jIlWRgFAg== X-Google-Smtp-Source: ABdhPJzcx+K7y7nTF2J7t7ctk9qHmMThVAkvjyO3GIgGQgfcG1DvOwWXZZJCKwI4d/tX1gzMxUy9ew== X-Received: by 2002:a0c:f744:: with SMTP id e4mr4171371qvo.5.1619104701942; Thu, 22 Apr 2021 08:18:21 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id l16sm2348909qkg.91.2021.04.22.08.18.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Apr 2021 08:18:21 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com Subject: [PATCH 7/7] ci: run test round with parallel-checkout enabled Date: Thu, 22 Apr 2021 12:17:53 -0300 Message-Id: X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org We already have tests for the basic parallel-checkout operations. But this code can also run be executed by other commands, such as git-read-tree and git-sparse-checkout, which are currently not tested with multiple workers. To promote a wider test coverage without duplicating tests: 1. Add the GIT_TEST_CHECKOUT_WORKERS environment variable, to optionally force parallel-checkout execution during the whole test suite. 2. Set this variable (with a value of 2) in the second test round of our linux-gcc CI job. This round runs `make test` again with some optional GIT_TEST_* variables enabled, so there is no additional overhead in exercising the parallel-checkout code here. Note that tests checking out less than two parallel-eligible entries will fall back to the sequential mode. Nevertheless, it's still a good exercise for the parallel-checkout framework as the fallback codepath also writes the queued entries using the parallel-checkout functions (only without spawning any worker). Signed-off-by: Matheus Tavares --- ci/run-build-and-tests.sh | 1 + parallel-checkout.c | 14 ++++++++++++++ t/README | 4 ++++ t/lib-parallel-checkout.sh | 3 +++ 4 files changed, 22 insertions(+) diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index a66b5e8c75..23b28e7391 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -25,6 +25,7 @@ linux-gcc) export GIT_TEST_ADD_I_USE_BUILTIN=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master export GIT_TEST_WRITE_REV_INDEX=1 + export GIT_TEST_CHECKOUT_WORKERS=2 make test ;; linux-clang) diff --git a/parallel-checkout.c b/parallel-checkout.c index 6fb3f1e6c9..6b1af32bb3 100644 --- a/parallel-checkout.c +++ b/parallel-checkout.c @@ -35,6 +35,20 @@ static const int DEFAULT_NUM_WORKERS = 1; void get_parallel_checkout_configs(int *num_workers, int *threshold) { + char *env_workers = getenv("GIT_TEST_CHECKOUT_WORKERS"); + + if (env_workers && *env_workers) { + if (strtol_i(env_workers, 10, num_workers)) { + die("invalid value for GIT_TEST_CHECKOUT_WORKERS: '%s'", + env_workers); + } + if (*num_workers < 1) + *num_workers = online_cpus(); + + *threshold = 0; + return; + } + if (git_config_get_int("checkout.workers", num_workers)) *num_workers = DEFAULT_NUM_WORKERS; else if (*num_workers < 1) diff --git a/t/README b/t/README index fd9375b146..a194488f27 100644 --- a/t/README +++ b/t/README @@ -436,6 +436,10 @@ and "sha256". GIT_TEST_WRITE_REV_INDEX=, when true enables the 'pack.writeReverseIndex' setting. +GIT_TEST_CHECKOUT_WORKERS= overrides the 'checkout.workers' setting +to and 'checkout.thresholdForParallelism' to 0, forcing the +execution of the parallel-checkout code. + Naming Tests ------------ diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh index 16ee18389b..0921a32b96 100644 --- a/t/lib-parallel-checkout.sh +++ b/t/lib-parallel-checkout.sh @@ -1,5 +1,8 @@ # Helpers for t208* tests +# Parallel checkout tests need full control of the number of workers +unset GIT_TEST_CHECKOUT_WORKERS + set_checkout_config () { if test $# -ne 2 then