From patchwork Fri Apr 30 21:40:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12234443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25C14C433ED for ; Fri, 30 Apr 2021 21:40:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 03FA661476 for ; Fri, 30 Apr 2021 21:40:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231840AbhD3Vlj (ORCPT ); Fri, 30 Apr 2021 17:41:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231259AbhD3Vle (ORCPT ); Fri, 30 Apr 2021 17:41:34 -0400 Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AAEC5C06174A for ; Fri, 30 Apr 2021 14:40:45 -0700 (PDT) Received: by mail-qt1-x831.google.com with SMTP id a18so28918329qtj.10 for ; Fri, 30 Apr 2021 14:40:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=C/38sFQa6zRAfOZz7FManGZbEWRjFsxK9Oe9+m5j86c=; b=O3RQo1uoPRW4Lhu0wYtPUfWYQc4oleeihzCqcsqyZA5ATPXiPfJi1+I2NXs5MvAYf4 5RVnvUKkFyfTPRQuH+WaD2RBOkAKS8R/hWUgDIjHn9hsp0TKTflB9nIF9VpytaB4HNiQ FB4pE27evuoMaROWNXx0FK+NYpd3nGnZZ5neRYmFAufCVgeygL1TIydGCYD6PDe3kOsk oH+O/XLDsz2Gggj3u4SbOhdZ3JKnApuViG7CXOg26JsmXW4nA+p0gd8Exg06NLNS2zNl Ew42JOvVzk42OF0WA20CDglEPgx52pGBhkyj5Hf2hLquzBFNmpwlZwwEvr9bGalbZY6Z 3pgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=C/38sFQa6zRAfOZz7FManGZbEWRjFsxK9Oe9+m5j86c=; b=lpR31A4KI4UqSw7EwjNAVkeS+5ME6q++2gHmbnN5MORbP5nfmD8pR8WJJcFcJV4p4i w/ui9Yxcjv37eVckKwmJTIPdca2ZkT8UjXFX3NaetN2bisvS+8uFw4J8sUTrS6Hp1wOU 69qGX38k7mRcAkmLA5mtP765GvVcm7EmMSLJioc/m5C1rSXOUB/lrnPVB21ZfT9Pkac1 ytANKGNT2sgn8iVghp/GIjZUeFobS0PLjCE7ncQI7W9xpjXZ5Py2KaRa9TWBCEba4jQ2 8WBC2BwLCc4uysMDQnnBSEc8ZCgaZTlGMJ8vKg54yAEW3gXrhOJwZ5NM2TTw9Rn+6OKo SFsg== X-Gm-Message-State: AOAM5331tTzIudp9706fgtJseFlEhEOCiVPlV3MMOckpLvvZ7QCpxXQN WOoQt7rfnhv0hFscN6s5weZZIdHnb5yktw== X-Google-Smtp-Source: ABdhPJxpNWITMRmIdvyv/ivr4QNgHmOBx/9RxmlcqVVlHicx9VFeMWV14/BCbFYcS2NSwP+oGyOnMw== X-Received: by 2002:ac8:544d:: with SMTP id d13mr6637739qtq.93.1619818844643; Fri, 30 Apr 2021 14:40:44 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id j13sm3123718qth.57.2021.04.30.14.40.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Apr 2021 14:40:44 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com Subject: [PATCH v2 1/8] make_transient_cache_entry(): optionally alloc from mem_pool Date: Fri, 30 Apr 2021 18:40:28 -0300 Message-Id: X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Allow make_transient_cache_entry() to optionally receive a mem_pool struct in which it should allocate the entry. This will be used in the following patch, to store some transient entries which should persist until parallel checkout finishes. Signed-off-by: Matheus Tavares --- builtin/checkout--worker.c | 2 +- builtin/checkout.c | 2 +- builtin/difftool.c | 2 +- cache.h | 11 ++++++----- read-cache.c | 12 ++++++++---- unpack-trees.c | 2 +- 6 files changed, 18 insertions(+), 13 deletions(-) diff --git a/builtin/checkout--worker.c b/builtin/checkout--worker.c index 31e0de2f7e..289a9b8f89 100644 --- a/builtin/checkout--worker.c +++ b/builtin/checkout--worker.c @@ -39,7 +39,7 @@ static void packet_to_pc_item(const char *buffer, int len, } memset(pc_item, 0, sizeof(*pc_item)); - pc_item->ce = make_empty_transient_cache_entry(fixed_portion->name_len); + pc_item->ce = make_empty_transient_cache_entry(fixed_portion->name_len, NULL); pc_item->ce->ce_namelen = fixed_portion->name_len; pc_item->ce->ce_mode = fixed_portion->ce_mode; memcpy(pc_item->ce->name, variant, pc_item->ce->ce_namelen); diff --git a/builtin/checkout.c b/builtin/checkout.c index 4c696ef480..db667d0267 100644 --- a/builtin/checkout.c +++ b/builtin/checkout.c @@ -291,7 +291,7 @@ static int checkout_merged(int pos, const struct checkout *state, int *nr_checko if (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid)) die(_("Unable to add merge result for '%s'"), path); free(result_buf.ptr); - ce = make_transient_cache_entry(mode, &oid, path, 2); + ce = make_transient_cache_entry(mode, &oid, path, 2, NULL); if (!ce) die(_("make_cache_entry failed for path '%s'"), path); status = checkout_entry(ce, state, NULL, nr_checkouts); diff --git a/builtin/difftool.c b/builtin/difftool.c index ef25729d49..afacbcd581 100644 --- a/builtin/difftool.c +++ b/builtin/difftool.c @@ -323,7 +323,7 @@ static int checkout_path(unsigned mode, struct object_id *oid, struct cache_entry *ce; int ret; - ce = make_transient_cache_entry(mode, oid, path, 0); + ce = make_transient_cache_entry(mode, oid, path, 0, NULL); ret = checkout_entry(ce, state, NULL, NULL); discard_cache_entry(ce); diff --git a/cache.h b/cache.h index 148d9ab5f1..b6b42cc3f3 100644 --- a/cache.h +++ b/cache.h @@ -356,16 +356,17 @@ struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t name_len); /* - * Create a cache_entry that is not intended to be added to an index. - * Caller is responsible for discarding the cache_entry - * with `discard_cache_entry`. + * Create a cache_entry that is not intended to be added to an index. If `mp` + * is not NULL, the entry is allocated within the given memory pool. Caller is + * responsible for discarding "loose" entries with `discard_cache_entry()` and + * the mem_pool with `mem_pool_discard(mp, should_validate_cache_entries())`. */ struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct object_id *oid, const char *path, - int stage); + int stage, struct mem_pool *mp); -struct cache_entry *make_empty_transient_cache_entry(size_t name_len); +struct cache_entry *make_empty_transient_cache_entry(size_t len, struct mem_pool *mp); /* * Discard cache entry. diff --git a/read-cache.c b/read-cache.c index 5a907af2fb..eb389ccb8f 100644 --- a/read-cache.c +++ b/read-cache.c @@ -813,8 +813,10 @@ struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t le return mem_pool__ce_calloc(find_mem_pool(istate), len); } -struct cache_entry *make_empty_transient_cache_entry(size_t len) +struct cache_entry *make_empty_transient_cache_entry(size_t len, struct mem_pool *mp) { + if (mp) + return mem_pool__ce_calloc(mp, len); return xcalloc(1, cache_entry_size(len)); } @@ -848,8 +850,10 @@ struct cache_entry *make_cache_entry(struct index_state *istate, return ret; } -struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct object_id *oid, - const char *path, int stage) +struct cache_entry *make_transient_cache_entry(unsigned int mode, + const struct object_id *oid, + const char *path, int stage, + struct mem_pool *mp) { struct cache_entry *ce; int len; @@ -860,7 +864,7 @@ struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct o } len = strlen(path); - ce = make_empty_transient_cache_entry(len); + ce = make_empty_transient_cache_entry(len, mp); oidcpy(&ce->oid, oid); memcpy(ce->name, path, len); diff --git a/unpack-trees.c b/unpack-trees.c index 4b77e52c6b..fa5b7ab7ee 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1034,7 +1034,7 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, size_t len = traverse_path_len(info, tree_entry_len(n)); struct cache_entry *ce = is_transient ? - make_empty_transient_cache_entry(len) : + make_empty_transient_cache_entry(len, NULL) : make_empty_cache_entry(istate, len); ce->ce_mode = create_ce_mode(n->mode); From patchwork Fri Apr 30 21:40:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12234445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57A35C433B4 for ; Fri, 30 Apr 2021 21:40:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2E6F761476 for ; Fri, 30 Apr 2021 21:40:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232101AbhD3Vlo (ORCPT ); Fri, 30 Apr 2021 17:41:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231782AbhD3Vlg (ORCPT ); Fri, 30 Apr 2021 17:41:36 -0400 Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2237C06174A for ; Fri, 30 Apr 2021 14:40:47 -0700 (PDT) Received: by mail-qt1-x835.google.com with SMTP id d12so14640466qtr.4 for ; Fri, 30 Apr 2021 14:40:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GiHs2SetS2X46DqvIh4n7daDBre+E0yyC9Qq4Ip+lB8=; b=MKOilUpaouvy9mn9guznTd/k3aq5BgUFgqrKm2Wp97WIz2qbqlAmkY7ic8pLWYP6fK Gcyj7zH1S+2jnIO6D2VeePvbWUnPGtXLRyUwZBhT8dCD7iKmIqbJ6Qk8dwEdaJ1NywCV 2z9j7F4PybCmuvQKaBDu2GnFMrXUqDUfdWpHxwANEYnIwoodUcT0T1lvHbF3Mqx8NtBb /kRzvuIBbams8OlvPCrv/nRj5ybrEixNsiEXwWN01dYMo+k1Pf/OogQw8Az3d7gV2nBr JB/vn/+jsZMjW77SexmdF/ym0drRHCaaqoFzvIfBaCadWUR0RlkSKuazf+cIszGZvwwk 9j7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GiHs2SetS2X46DqvIh4n7daDBre+E0yyC9Qq4Ip+lB8=; b=a6Vr/6pzIOTDXz+pLUXP9KpWP+7AVG1y4PCDBl96RM0c6VjMy4gWn1NFRzTTVlYr2C uV8M2TJp6NYieO+UG5Wdvl9OHRRNJmlrp+PmZooJjv4Tjklpm2h4oFiJhaJ7kIT95xiK TiJeIAoVfToLYNnXrJZcbE36NOEmxD43lWmlClgfJwSfqvf99AqslkwBFzK4PlgWD0KB Occb6AWdJ7yPhUBZ3qTU1JpKE7fe7Si0VUS80eeHUe9eXF8W1wfOYD8YpLoABJ8jDfH/ dmgyL9r7K/cQrJFvRYRIRWKyXzzbr0A+pXxljS4JiANBstj4S60ejgAIyD4mK+eSBcMn fdnQ== X-Gm-Message-State: AOAM532wVEvvOMTp8z7QJOT7/SqWwZiFLZOMw8KkQ7PP17yVpiQO3kg6 EaZhRQSWPa6xV/ftDovYbbfP4p+isN7xPg== X-Google-Smtp-Source: ABdhPJwsKUEW8Wcl+aNU9pmyqLJPY3Yo7JWFLWYrcpj49Dv7gSpHflosiJSPnxGv3kQw7OFZG8w+gQ== X-Received: by 2002:ac8:4d92:: with SMTP id a18mr6549857qtw.312.1619818846658; Fri, 30 Apr 2021 14:40:46 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id j13sm3123718qth.57.2021.04.30.14.40.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Apr 2021 14:40:46 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com Subject: [PATCH v2 2/8] builtin/checkout.c: complete parallel checkout support Date: Fri, 30 Apr 2021 18:40:29 -0300 Message-Id: X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Pathspec-limited checkouts (like `git checkout *.txt`) are performed by a code path that doesn't yet support parallel checkout because it calls checkout_entry() directly, instead of unpack_trees(). Let's add parallel checkout support for this code path too. Note: the transient cache entries allocated in checkout_merged() are now allocated in a mem_pool which is only discarded after parallel checkout finishes. This is done because the entries need to be valid when run_parallel_checkout() is called. Signed-off-by: Matheus Tavares --- builtin/checkout.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/builtin/checkout.c b/builtin/checkout.c index db667d0267..b71dc08430 100644 --- a/builtin/checkout.c +++ b/builtin/checkout.c @@ -27,6 +27,7 @@ #include "wt-status.h" #include "xdiff-interface.h" #include "entry.h" +#include "parallel-checkout.h" static const char * const checkout_usage[] = { N_("git checkout [] "), @@ -230,7 +231,8 @@ static int checkout_stage(int stage, const struct cache_entry *ce, int pos, return error(_("path '%s' does not have their version"), ce->name); } -static int checkout_merged(int pos, const struct checkout *state, int *nr_checkouts) +static int checkout_merged(int pos, const struct checkout *state, + int *nr_checkouts, struct mem_pool *ce_mem_pool) { struct cache_entry *ce = active_cache[pos]; const char *path = ce->name; @@ -291,11 +293,10 @@ static int checkout_merged(int pos, const struct checkout *state, int *nr_checko if (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid)) die(_("Unable to add merge result for '%s'"), path); free(result_buf.ptr); - ce = make_transient_cache_entry(mode, &oid, path, 2, NULL); + ce = make_transient_cache_entry(mode, &oid, path, 2, ce_mem_pool); if (!ce) die(_("make_cache_entry failed for path '%s'"), path); status = checkout_entry(ce, state, NULL, nr_checkouts); - discard_cache_entry(ce); return status; } @@ -359,16 +360,22 @@ static int checkout_worktree(const struct checkout_opts *opts, int nr_checkouts = 0, nr_unmerged = 0; int errs = 0; int pos; + int pc_workers, pc_threshold; + struct mem_pool ce_mem_pool; state.force = 1; state.refresh_cache = 1; state.istate = &the_index; + mem_pool_init(&ce_mem_pool, 0); + get_parallel_checkout_configs(&pc_workers, &pc_threshold); init_checkout_metadata(&state.meta, info->refname, info->commit ? &info->commit->object.oid : &info->oid, NULL); enable_delayed_checkout(&state); + if (pc_workers > 1) + init_parallel_checkout(); for (pos = 0; pos < active_nr; pos++) { struct cache_entry *ce = active_cache[pos]; if (ce->ce_flags & CE_MATCHED) { @@ -384,10 +391,15 @@ static int checkout_worktree(const struct checkout_opts *opts, &nr_checkouts, opts->overlay_mode); else if (opts->merge) errs |= checkout_merged(pos, &state, - &nr_unmerged); + &nr_unmerged, + &ce_mem_pool); pos = skip_same_name(ce, pos) - 1; } } + if (pc_workers > 1) + errs |= run_parallel_checkout(&state, pc_workers, pc_threshold, + NULL, NULL); + mem_pool_discard(&ce_mem_pool, should_validate_cache_entries()); remove_marked_cache_entries(&the_index, 1); remove_scheduled_dirs(); errs |= finish_delayed_checkout(&state, &nr_checkouts); From patchwork Fri Apr 30 21:40:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12234447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 674D6C43460 for ; Fri, 30 Apr 2021 21:40:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4B5A3610EA for ; Fri, 30 Apr 2021 21:40:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231803AbhD3Vlq (ORCPT ); Fri, 30 Apr 2021 17:41:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58016 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231790AbhD3Vli (ORCPT ); Fri, 30 Apr 2021 17:41:38 -0400 Received: from mail-qv1-xf29.google.com (mail-qv1-xf29.google.com [IPv6:2607:f8b0:4864:20::f29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 936C2C06138B for ; Fri, 30 Apr 2021 14:40:49 -0700 (PDT) Received: by mail-qv1-xf29.google.com with SMTP id h3so34426383qve.13 for ; Fri, 30 Apr 2021 14:40:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TaxKTevRj9KNItutE8NJIJeb/G3/fDBUHsrFXaVnfbw=; b=GgeF+CjNUcc4cbN8nh0tFm3jFjVGt9P1oH+cpkpsRtWmUF0OrxQY4CO3EQvA3fBkv5 8gx93IhohEfOEWIvcWrEogwV0HIBY1Fe+7EoB2g8E3YAW8h3EHBXLk6hhP3XYD9JgYdc uICU20O46Whwt3jAPMu/OzcrYCelgLSRxEJ3wzG91NWC0AOKEceZqmYg1XxJKXQuRN8N J/WJwzTg5LmE7T5FdmW8DychP1N39AUta6DAF0lK1kTub3ngI2GnbnAChHrGi+3WV2GZ UfLIBd7h2I3Xn2yh1x7tkwZXJPYpjgFrIddl9koBvVkD3xerL6y6Lj9XVY3kSupt7axe AwKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TaxKTevRj9KNItutE8NJIJeb/G3/fDBUHsrFXaVnfbw=; b=etDiHzGV5AVO7vPYqFuGIYKA+5HsiYegLJNi5e50DA+rYxeac5gzKGS7SUUvZLF+oJ wJi0KvXm6PlXVjKQlkhvqdMvxWJyUygrtWHU3OKWKb3RLThd94rBknHmEhs++3yJA2rP VCPpWEA/03cZsPQVVBOwuBvvIQPZaRoFFErRTpom6l37MWknbqlO1qz6SNAsUu33Fs8A zepG5AqOiHjONhemmNCWji73fAYlzL6esZuN/gJoUFJlSQEqslWX1dbrQ2esfGaLFMXs VqTgfv/cb6LT/HVzEJ81cfAni3b85XR6vESGIiWnGBGPpZWjHpsb5ajYjKpr8nR8Y7Du X+LA== X-Gm-Message-State: AOAM530pW1+TWphAeZHfZcglcn5HRiYnCwr+Rzd7FC+2wcm42XWV8KAS z5gyOaKfrbcHIJgtJViqKB02WLnhjq0edQ== X-Google-Smtp-Source: ABdhPJxZlH51x5b+psK0jBUCKIyV18YfJxtX9i3KHtG3Hh3DmAeYJjydqELYeqGP3oi9+BeBBV/Gdg== X-Received: by 2002:a0c:e8c4:: with SMTP id m4mr7646388qvo.21.1619818848598; Fri, 30 Apr 2021 14:40:48 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id j13sm3123718qth.57.2021.04.30.14.40.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Apr 2021 14:40:48 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com Subject: [PATCH v2 3/8] checkout-index: add parallel checkout support Date: Fri, 30 Apr 2021 18:40:30 -0300 Message-Id: <0fe1a5fabc8cb5f7c2421afaefae030d399d28ed.1619818517.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Note: previously, `checkout_all()` would not return on errors, but instead call `exit()` with a non-zero code. However, it only did that after calling `checkout_entry()` for all index entries, thus not stopping on the first error, but attempting to write the maximum number of entries possible. In order to maintain this behavior we now propagate `checkout_all()`s error status to `cmd_checkout_index()`, so that it can call `run_parallel_checkout()` and attempt to write the queued entries before exiting with the error code. Signed-off-by: Matheus Tavares --- builtin/checkout-index.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/builtin/checkout-index.c b/builtin/checkout-index.c index c0bf4ac1b2..e8a82ea9ed 100644 --- a/builtin/checkout-index.c +++ b/builtin/checkout-index.c @@ -12,6 +12,7 @@ #include "cache-tree.h" #include "parse-options.h" #include "entry.h" +#include "parallel-checkout.h" #define CHECKOUT_ALL 4 static int nul_term_line; @@ -115,7 +116,7 @@ static int checkout_file(const char *name, const char *prefix) return -1; } -static void checkout_all(const char *prefix, int prefix_length) +static int checkout_all(const char *prefix, int prefix_length) { int i, errs = 0; struct cache_entry *last_ce = NULL; @@ -142,11 +143,7 @@ static void checkout_all(const char *prefix, int prefix_length) } if (last_ce && to_tempfile) write_tempfile_record(last_ce->name, prefix); - if (errs) - /* we have already done our error reporting. - * exit with the same code as die(). - */ - exit(128); + return !!errs; } static const char * const builtin_checkout_index_usage[] = { @@ -182,6 +179,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix) int force = 0, quiet = 0, not_new = 0; int index_opt = 0; int err = 0; + int pc_workers, pc_threshold; struct option builtin_checkout_index_options[] = { OPT_BOOL('a', "all", &all, N_("check out all files in the index")), @@ -236,6 +234,10 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix) hold_locked_index(&lock_file, LOCK_DIE_ON_ERROR); } + get_parallel_checkout_configs(&pc_workers, &pc_threshold); + if (pc_workers > 1) + init_parallel_checkout(); + /* Check out named files first */ for (i = 0; i < argc; i++) { const char *arg = argv[i]; @@ -275,12 +277,16 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix) strbuf_release(&buf); } + if (all) + err |= checkout_all(prefix, prefix_length); + + if (pc_workers > 1) + err |= run_parallel_checkout(&state, pc_workers, pc_threshold, + NULL, NULL); + if (err) return 1; - if (all) - checkout_all(prefix, prefix_length); - if (is_lock_file_locked(&lock_file) && write_locked_index(&the_index, &lock_file, COMMIT_LOCK)) die("Unable to write new index file"); From patchwork Fri Apr 30 21:40:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12234449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D3A5C433ED for ; Fri, 30 Apr 2021 21:41:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 618CF61477 for ; Fri, 30 Apr 2021 21:41:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232011AbhD3Vls (ORCPT ); Fri, 30 Apr 2021 17:41:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231856AbhD3Vll (ORCPT ); Fri, 30 Apr 2021 17:41:41 -0400 Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB1D7C06174A for ; Fri, 30 Apr 2021 14:40:51 -0700 (PDT) Received: by mail-qt1-x82d.google.com with SMTP id n22so13120376qtk.9 for ; Fri, 30 Apr 2021 14:40:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0tEnEjfh4UFSUxC9OyF8P7l5RZeLtK76zdR1Km/Uhls=; b=JZ7b59ny/z9flfxCfbJSpey2zo5nXfFDA1vSc7HTmBtC+5UYtYtQlztjF1UC2Z4HbU ctRoIHGtQo19LilDbZ5ykoh9qMF7A0t/iArHm4WcyD6Mr1wlbFW6ep5YvIGucvvVQSj9 qPPaBDZ26whL9OFxc6c7Z0EFqBEv2A8Uf56oIIRmNcQoT4w4hGjklVvwv6QLt+aWC1cN 7mRsDPGuEWq0i1wbQql305EolWqRUtzQr0AbXklTxTHg7U6oU21Jz13re7vkzrIcMmp3 G6T26oaFxxNMwtkS5nIeNwLafL0FJLawOL+srjQTZ66PL5BXvtBInYLaw+7VN0DJSWve pNmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0tEnEjfh4UFSUxC9OyF8P7l5RZeLtK76zdR1Km/Uhls=; b=RWQo7l9HMzrQ4c5u2a9d+POaLJmqxub3man7DiFWF8qjO4MBwsW/Xoz2HpuiV2Z9r/ 16is+fkp2Qu0BIiCKkMHEcZ9RhHtttn8JfZsOGI5epwL3Oqu8u0w0yWDSCDI8h19hSVB WcTLYanhhAuPfDFxVoreno4ZEge620RX51Mv1K/8kjJtK0Ldv5ikGlpBktDvCbEasitK eOFdCbno8IfZxMTju9SjDLjcghjtd7KV7syBwocuiXK329E+ZoKKzB1CZwSsSwHVpUmV FavJc7b3doons0A5PaAtJuNeNBpmnNY+NtDp0jE2IuhJCK3t7KD+gVVBGXMj1v59poq1 19sg== X-Gm-Message-State: AOAM5318lpSgT6qNOihRSSIi5OwGfJIk+kojTPCxYfA3RzPPXLQc67tt e5f776OrSLJlfMhfAkpSC8qWt7BzOcRiEw== X-Google-Smtp-Source: ABdhPJwxomdrTyqR7P2R9bsiO1BA+KBVJKKwAwJe4Kii45qSbnD9g1msDBKN3p1zWjqunIe7Fg4/GA== X-Received: by 2002:ac8:5f93:: with SMTP id j19mr6173651qta.49.1619818850615; Fri, 30 Apr 2021 14:40:50 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id j13sm3123718qth.57.2021.04.30.14.40.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Apr 2021 14:40:50 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com Subject: [PATCH v2 4/8] parallel-checkout: add tests for basic operations Date: Fri, 30 Apr 2021 18:40:31 -0300 Message-Id: X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add tests to populate the working tree during clone and checkout using sequential and parallel mode, to confirm that they produce identical results. Also test basic checkout mechanics, such as checking for symlinks in the leading directories and the abidance to --force. Note: some helper functions are added to a common lib file which is only included by t2080 for now. But they will also be used by other parallel-checkout tests in the following patches. Co-authored-by: Jeff Hostetler Signed-off-by: Matheus Tavares --- t/lib-parallel-checkout.sh | 42 +++++ t/t2080-parallel-checkout-basics.sh | 229 ++++++++++++++++++++++++++++ 2 files changed, 271 insertions(+) create mode 100644 t/lib-parallel-checkout.sh create mode 100755 t/t2080-parallel-checkout-basics.sh diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh new file mode 100644 index 0000000000..f60b22ef34 --- /dev/null +++ b/t/lib-parallel-checkout.sh @@ -0,0 +1,42 @@ +# Helpers for tests invoking parallel-checkout + +set_checkout_config () { + if test $# -ne 2 + then + BUG "usage: set_checkout_config " + fi && + + test_config_global checkout.workers $1 && + test_config_global checkout.thresholdForParallelism $2 +} + +# Run "${@:2}" and check that $1 checkout workers were used +test_checkout_workers () { + if test $# -lt 2 + then + BUG "too few arguments to test_checkout_workers" + fi && + + local expected_workers=$1 && + shift && + + local trace_file=trace-test-checkout-workers && + rm -f "$trace_file" && + GIT_TRACE2="$(pwd)/$trace_file" "$@" && + + local workers=$(grep "child_start\[..*\] git checkout--worker" "$trace_file" | wc -l) && + test $workers -eq $expected_workers && + rm "$trace_file" +} + +# Verify that both the working tree and the index were created correctly +verify_checkout () { + if test $# -ne 1 + then + BUG "usage: verify_checkout " + fi && + + git -C "$1" diff-index --ignore-submodules=none --exit-code HEAD -- && + git -C "$1" status --porcelain >"$1".status && + test_must_be_empty "$1".status +} diff --git a/t/t2080-parallel-checkout-basics.sh b/t/t2080-parallel-checkout-basics.sh new file mode 100755 index 0000000000..7087818550 --- /dev/null +++ b/t/t2080-parallel-checkout-basics.sh @@ -0,0 +1,229 @@ +#!/bin/sh + +test_description='parallel-checkout basics + +Ensure that parallel-checkout basically works on clone and checkout, spawning +the required number of workers and correctly populating both the index and the +working tree. +' + +TEST_NO_CREATE_REPO=1 +. ./test-lib.sh +. "$TEST_DIRECTORY/lib-parallel-checkout.sh" + +# Test parallel-checkout with a branch switch containing a variety of file +# creations, deletions, and modifications, involving different entry types. +# The branches B1 and B2 have the following paths: +# +# B1 B2 +# a/a (file) a (file) +# b (file) b/b (file) +# +# c/c (file) c (symlink) +# d (symlink) d/d (file) +# +# e/e (file) e (submodule) +# f (submodule) f/f (file) +# +# g (submodule) g (symlink) +# h (symlink) h (submodule) +# +# Additionally, the following paths are present on both branches, but with +# different contents: +# +# i (file) i (file) +# j (symlink) j (symlink) +# k (submodule) k (submodule) +# +# And the following paths are only present in one of the branches: +# +# l/l (file) - +# - m/m (file) +# +test_expect_success 'setup repo for checkout with various types of changes' ' + git init sub && + ( + cd sub && + git checkout -b B2 && + echo B2 >file && + git add file && + git commit -m file && + + git checkout -b B1 && + echo B1 >file && + git add file && + git commit -m file + ) && + + git init various && + ( + cd various && + + git checkout -b B1 && + mkdir a c e && + echo a/a >a/a && + echo b >b && + echo c/c >c/c && + test_ln_s_add c d && + echo e/e >e/e && + git submodule add ../sub f && + git submodule add ../sub g && + test_ln_s_add c h && + + echo "B1 i" >i && + test_ln_s_add c j && + git submodule add -b B1 ../sub k && + mkdir l && + echo l/l >l/l && + + git add . && + git commit -m B1 && + + git checkout -b B2 && + git rm -rf :^.gitmodules :^k && + mkdir b d f && + echo a >a && + echo b/b >b/b && + test_ln_s_add b c && + echo d/d >d/d && + git submodule add ../sub e && + echo f/f >f/f && + test_ln_s_add b g && + git submodule add ../sub h && + + echo "B2 i" >i && + test_ln_s_add b j && + git -C k checkout B2 && + mkdir m && + echo m/m >m/m && + + git add . && + git commit -m B2 && + + git checkout --recurse-submodules B1 + ) +' + +for mode in sequential parallel sequential-fallback +do + case $mode in + sequential) workers=1 threshold=0 expected_workers=0 ;; + parallel) workers=2 threshold=0 expected_workers=2 ;; + sequential-fallback) workers=2 threshold=100 expected_workers=0 ;; + esac + + test_expect_success "$mode checkout" ' + repo=various_$mode && + cp -R various $repo && + + # The just copied files have more recent timestamps than their + # associated index entries. So refresh the cached timestamps + # to avoid an "entry not up-to-date" error from `git checkout`. + # We only have to do this for the submodules as `git checkout` + # will already refresh the superproject index before performing + # the up-to-date check. + # + git -C $repo submodule foreach "git update-index --refresh" && + + set_checkout_config $workers $threshold && + test_checkout_workers $expected_workers \ + git -C $repo checkout --recurse-submodules B2 && + verify_checkout $repo + ' +done + +for mode in parallel sequential-fallback +do + case $mode in + parallel) workers=2 threshold=0 expected_workers=2 ;; + sequential-fallback) workers=2 threshold=100 expected_workers=0 ;; + esac + + test_expect_success "$mode checkout on clone" ' + repo=various_${mode}_clone && + set_checkout_config $workers $threshold && + test_checkout_workers $expected_workers \ + git clone --recurse-submodules --branch B2 various $repo && + verify_checkout $repo + ' +done + +# Just to be paranoid, actually compare the working trees' contents directly. +test_expect_success 'compare the working trees' ' + rm -rf various_*/.git && + rm -rf various_*/*/.git && + + # We use `git diff` instead of `diff -r` because the latter would + # follow symlinks, and not all `diff` implementations support the + # `--no-dereference` option. + # + git diff --no-index various_sequential various_parallel && + git diff --no-index various_sequential various_parallel_clone && + git diff --no-index various_sequential various_sequential-fallback && + git diff --no-index various_sequential various_sequential-fallback_clone +' + +# Currently, each submodule is checked out in a separated child process, but +# these subprocesses must also be able to use parallel checkout workers to +# write the submodules' entries. +test_expect_success 'submodules can use parallel checkout' ' + set_checkout_config 2 0 && + git init super && + ( + cd super && + git init sub && + test_commit -C sub A && + test_commit -C sub B && + git submodule add ./sub && + git commit -m sub && + rm sub/* && + test_checkout_workers 2 git checkout --recurse-submodules . + ) +' + +test_expect_success 'parallel checkout respects --[no]-force' ' + set_checkout_config 2 0 && + git init dirty && + ( + cd dirty && + mkdir D && + test_commit D/F && + test_commit F && + + rm -rf D && + echo changed >D && + echo changed >F.t && + + # We expect 0 workers because there is nothing to be done + test_checkout_workers 0 git checkout HEAD && + test_path_is_file D && + grep changed D && + grep changed F.t && + + test_checkout_workers 2 git checkout --force HEAD && + test_path_is_dir D && + grep D/F D/F.t && + grep F F.t + ) +' + +test_expect_success SYMLINKS 'parallel checkout checks for symlinks in leading dirs' ' + set_checkout_config 2 0 && + git init symlinks && + ( + cd symlinks && + mkdir D untracked && + # Commit 2 files to have enough work for 2 parallel workers + test_commit D/A && + test_commit D/B && + rm -rf D && + ln -s untracked D && + + test_checkout_workers 2 git checkout --force HEAD && + ! test -h D && + grep D/A D/A.t && + grep D/B D/B.t + ) +' + +test_done From patchwork Fri Apr 30 21:40:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12234451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16403C433B4 for ; Fri, 30 Apr 2021 21:41:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E79E261477 for ; Fri, 30 Apr 2021 21:41:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232611AbhD3Vlt (ORCPT ); Fri, 30 Apr 2021 17:41:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231905AbhD3Vlm (ORCPT ); Fri, 30 Apr 2021 17:41:42 -0400 Received: from mail-qv1-xf2e.google.com (mail-qv1-xf2e.google.com [IPv6:2607:f8b0:4864:20::f2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3B9FC06138D for ; Fri, 30 Apr 2021 14:40:53 -0700 (PDT) Received: by mail-qv1-xf2e.google.com with SMTP id t14so5118624qvl.10 for ; Fri, 30 Apr 2021 14:40:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=t23D3WB4vmdvDXKy81WrtRCExr2GHpVXwNDoKFSvJ7E=; b=pZbGia0V/g6ma6aMvWf35AKO0YHBHmTxGg2xa3cV1Tj9TabRwVEAu5JXLA9CtvUEHZ 65B3haAQvCwjAoXLjWqMnyjmxyWnP5kHX6Wfqt3no2Im5QzwjxHD9IP0L0zf82iHeE6j t+KSft153Z1eaFl8e+cVx8Z3Thkseo2DI10LbXfxtkG+YLyUr/tgsEYbbXnaf4g2oC+N O3x8nizjexEKU8Q+bBOM7dQ0/6Gj5zsTBqwvRLGknFZK2qjgUwZY5Cg+X1xAD5IO6AGG X7v5lQ4lWYy2QDu1mHUc5QNS2/NIJnU+DZ/Xr5XThlE7A2TMelqReukTJj6+s95TPzdZ njzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=t23D3WB4vmdvDXKy81WrtRCExr2GHpVXwNDoKFSvJ7E=; b=Z2WvXM1Jc6YAZpzRbjL/0ePSCFkhcYA2W4udXBG41cbCgiShR9CQJFbPtWLNc9p+yU Z5On8sww7/5iZk9l69tuuKm04qfqSLCIyeYGheDwP4//usyDtvQf7aKjljW66cJwxgut cDAOlRsfiavx7/NuYfZW4LzJHBGBG8Tirb9T4QdzUhaZWXbU31z36t0xQZlTg9b6RcdT 4RUSRDBcd8+rutfeLS1126qbFWn7GS6HZI81nGybgjzVonThBKQ7CMQfb4XqEPJANEvt 0e47qyXdLMtE12ZuCELQNlLxvL41CKaZDFstszSw4Iy0wVljxRY88aBBC+6ssNsHmmgV GUdQ== X-Gm-Message-State: AOAM533+ipAyaz1B/RL6XLeUOuXFjnDgkP4kafYEPBnhjvo/D/tU1nOh eBlAbOshTgfj1M9IhPwbhOxX3Uswsf8TuA== X-Google-Smtp-Source: ABdhPJwar9i9IpnokQuJMOUW9wr+WSoT5I7ksZOTPD6JHVn+0zikKTlgs/dh+ByPdM3DBhiLgNepRw== X-Received: by 2002:ad4:504f:: with SMTP id m15mr8022337qvq.56.1619818852558; Fri, 30 Apr 2021 14:40:52 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id j13sm3123718qth.57.2021.04.30.14.40.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Apr 2021 14:40:52 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com Subject: [PATCH v2 5/8] parallel-checkout: add tests related to path collisions Date: Fri, 30 Apr 2021 18:40:32 -0300 Message-Id: X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add tests to confirm that path collisions are properly detected by checkout workers, both to avoid race conditions and to report colliding entries on clone. Co-authored-by: Jeff Hostetler Signed-off-by: Matheus Tavares --- parallel-checkout.c | 4 + t/lib-parallel-checkout.sh | 4 +- t/t2081-parallel-checkout-collisions.sh | 162 ++++++++++++++++++++++++ 3 files changed, 168 insertions(+), 2 deletions(-) create mode 100755 t/t2081-parallel-checkout-collisions.sh diff --git a/parallel-checkout.c b/parallel-checkout.c index 09e8b10a35..6fb3f1e6c9 100644 --- a/parallel-checkout.c +++ b/parallel-checkout.c @@ -8,6 +8,7 @@ #include "sigchain.h" #include "streaming.h" #include "thread-utils.h" +#include "trace2.h" struct pc_worker { struct child_process cp; @@ -326,6 +327,7 @@ void write_pc_item(struct parallel_checkout_item *pc_item, if (dir_sep && !has_dirs_only_path(path.buf, dir_sep - path.buf, state->base_dir_len)) { pc_item->status = PC_ITEM_COLLIDED; + trace2_data_string("pcheckout", NULL, "collision/dirname", path.buf); goto out; } @@ -341,6 +343,8 @@ void write_pc_item(struct parallel_checkout_item *pc_item, * call should have already caught these cases. */ pc_item->status = PC_ITEM_COLLIDED; + trace2_data_string("pcheckout", NULL, + "collision/basename", path.buf); } else { error_errno("failed to open file '%s'", path.buf); pc_item->status = PC_ITEM_FAILED; diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh index f60b22ef34..d6740425b1 100644 --- a/t/lib-parallel-checkout.sh +++ b/t/lib-parallel-checkout.sh @@ -22,12 +22,12 @@ test_checkout_workers () { local trace_file=trace-test-checkout-workers && rm -f "$trace_file" && - GIT_TRACE2="$(pwd)/$trace_file" "$@" && + GIT_TRACE2="$(pwd)/$trace_file" "$@" 2>&8 && local workers=$(grep "child_start\[..*\] git checkout--worker" "$trace_file" | wc -l) && test $workers -eq $expected_workers && rm "$trace_file" -} +} 8>&2 2>&4 # Verify that both the working tree and the index were created correctly verify_checkout () { diff --git a/t/t2081-parallel-checkout-collisions.sh b/t/t2081-parallel-checkout-collisions.sh new file mode 100755 index 0000000000..f6fcfc0c1e --- /dev/null +++ b/t/t2081-parallel-checkout-collisions.sh @@ -0,0 +1,162 @@ +#!/bin/sh + +test_description="path collisions during parallel checkout + +Parallel checkout must detect path collisions to: + +1) Avoid racily writing to different paths that represent the same file on disk. +2) Report the colliding entries on clone. + +The tests in this file exercise parallel checkout's collision detection code in +both these mechanics. +" + +. ./test-lib.sh +. "$TEST_DIRECTORY/lib-parallel-checkout.sh" + +TEST_ROOT="$PWD" + +test_expect_success CASE_INSENSITIVE_FS 'setup' ' + empty_oid=$(git hash-object -w --stdin objs <<-EOF && + 100644 $empty_oid FILE_X + 100644 $empty_oid FILE_x + 100644 $empty_oid file_X + 100644 $empty_oid file_x + EOF + git update-index --index-info >filter.log + EOF +' + +test_workers_in_event_trace () +{ + test $1 -eq $(grep ".event.:.child_start..*checkout--worker" $2 | wc -l) +} + +test_expect_success CASE_INSENSITIVE_FS 'worker detects basename collision' ' + GIT_TRACE2_EVENT="$(pwd)/trace" git \ + -c checkout.workers=2 -c checkout.thresholdForParallelism=0 \ + checkout . && + + test_workers_in_event_trace 2 trace && + collisions=$(grep -i "category.:.pcheckout.,.key.:.collision/basename.,.value.:.file_x.}" trace | wc -l) && + test $collisions -eq 3 +' + +test_expect_success CASE_INSENSITIVE_FS 'worker detects dirname collision' ' + test_config filter.logger.smudge "\"$TEST_ROOT/logger_script\" %f" && + empty_oid=$(git hash-object -w --stdin objs <<-EOF && + 100644 $empty_oid A/B + 100644 $empty_oid A/C + 100644 $empty_oid a + 100644 $attr_oid .gitattributes + EOF + git rm -rf . && + git update-index --index-info expected.log && + test_cmp filter.log expected.log && + + # Check that it used the right number of workers and detected the collisions + test_workers_in_event_trace 2 trace && + grep "category.:.pcheckout.,.key.:.collision/dirname.,.value.:.A/B.}" trace && + grep "category.:.pcheckout.,.key.:.collision/dirname.,.value.:.A/C.}" trace +' + +test_expect_success SYMLINKS,CASE_INSENSITIVE_FS 'do not follow symlinks colliding with leading dir' ' + empty_oid=$(git hash-object -w --stdin objs <<-EOF && + 120000 $symlink_oid D + 100644 $empty_oid d/x + 100644 $empty_oid e/y + EOF + git rm -rf . && + git update-index --index-info stderr && + + grep FILE_X stderr && + grep FILE_x stderr && + grep file_X stderr && + grep file_x stderr && + grep "the following paths have collided" stderr +' + +# This test ensures that the collision report code is correctly looking for +# colliding peers in the second half of the cache_entry array. This is done by +# defining a smudge command for the *last* array entry, which makes it +# non-eligible for parallel-checkout. Thus, it is checked out *first*, before +# spawning the workers. +# +# Note: this test doesn't work on Windows because, on this system, the +# collision report code uses strcmp() to find the colliding pairs when +# core.ignoreCase is false. And we need this setting for this test so that only +# 'file_x' matches the pattern of the filter attribute. But the test works on +# OSX, where the colliding pairs are found using inode. +# +test_expect_success CASE_INSENSITIVE_FS,!MINGW,!CYGWIN \ + 'collision report on clone (w/ colliding peer after the detected entry)' ' + + test_config_global filter.logger.smudge "\"$TEST_ROOT/logger_script\" %f" && + git reset --hard basename_collision && + echo "file_x filter=logger" >.gitattributes && + git add .gitattributes && + git commit -m "filter for file_x" && + + rm -rf clone-repo && + set_checkout_config 2 0 && + test_checkout_workers 2 \ + git -c core.ignoreCase=false clone . clone-repo 2>stderr && + + grep FILE_X stderr && + grep FILE_x stderr && + grep file_X stderr && + grep file_x stderr && + grep "the following paths have collided" stderr && + + # Check that only "file_x" was filtered + echo file_x >expected.log && + test_cmp clone-repo/filter.log expected.log +' + +test_done From patchwork Fri Apr 30 21:40:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12234453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC394C43460 for ; Fri, 30 Apr 2021 21:41:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D6B8361476 for ; Fri, 30 Apr 2021 21:41:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232392AbhD3Vlu (ORCPT ); Fri, 30 Apr 2021 17:41:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232184AbhD3Vlq (ORCPT ); Fri, 30 Apr 2021 17:41:46 -0400 Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A156FC06138F for ; Fri, 30 Apr 2021 14:40:55 -0700 (PDT) Received: by mail-qk1-x72b.google.com with SMTP id t17so43798835qkg.4 for ; Fri, 30 Apr 2021 14:40:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2EnM5QDr7FoNgrYUhohEVPq46wiDviob4V3QvwYbMTk=; b=KPTdmqNmpV1cMQFVkJsC4s/zByarIo2d/qeYCZB2OO0iJ5Ii7O+jcoLRwFW70Dzq4j 3FJ+Evm0vJ/UeaN5li5nxYqfrkN+WGbYj/t95BmXmsRKoOEkyYQ5V8sphKkZ5e+B8V3d qG6vKBzJVf4ApFgNLbhBCBtbA/XiiIS+bgrk8rtVr4QgV5H4DG7bOPl0rq/XfzopqRX5 F9cYYlA1IuEyag351Qr7hoc1hXmvRNOh2xbpB1BFftZ4qYLXWQ1eDwDz8qNxfZJthxOm vSJ7pdRzmVSHUxOKY/dHzVB9ktnuZG+Gt4IuMsQjU8qBR4McwelLaJj1sKbQKE/pkPZS 7Y7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2EnM5QDr7FoNgrYUhohEVPq46wiDviob4V3QvwYbMTk=; b=DKQOS9wh8NQ1rG1I4vpxiw2NNdzKdNWI2U6mj8ZR8xQenuUMq9oNObPAiaq0YVq1bM 4dTdsLf3n/VXo2ut7CvP8jA1RiIqu92gVH8SEMwpY8h6MMkTM1ChA471GbdV/fLmAUNW rPmxWAwy6xOiDiOobD92J7f37J30eIe2KbMTACB2kSr6Ujv3ca16R2x7bCPTbvbDgt/5 22FPVMbtuoS6YqXHBEPkhOO404IunYA5d0KLOI0wKdVe7U7jykbS4Dww2E4Ik7Mfryf0 3nI8hC+Ub33vX3iD2Grfd5FORyBdc7rFVAlQFjyrNG03t5OFfS3/QtOGBaJ9fH94IdD4 Uk6Q== X-Gm-Message-State: AOAM533Pbw2DH9Di03rPdhxIY1KOVOnRwLlzy6iULErrrr5rArPiGT0T tSiz7s6rE85cM0FP9KWwzLRC5PmQV69bdQ== X-Google-Smtp-Source: ABdhPJxL30+TIklWi9V/HMc7huKrKUCGLTHtinnmThhNh+dO6pPKhE0A32eVZMErcbwSWY0wAQjUNQ== X-Received: by 2002:a37:9507:: with SMTP id x7mr7934234qkd.1.1619818854636; Fri, 30 Apr 2021 14:40:54 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id j13sm3123718qth.57.2021.04.30.14.40.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Apr 2021 14:40:54 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com Subject: [PATCH v2 6/8] t0028: extract encoding helpers to lib-encoding.sh Date: Fri, 30 Apr 2021 18:40:33 -0300 Message-Id: <9161cd1503cc1a43b91bd899746b406e460b427b.1619818517.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The following patch will add tests outside t0028 which will also need to re-encode some strings. Extract the auxiliary encoding functions from t0028 to a common lib file so that they can be reused. Signed-off-by: Matheus Tavares --- t/lib-encoding.sh | 25 +++++++++++++++++++++++++ t/t0028-working-tree-encoding.sh | 25 +------------------------ 2 files changed, 26 insertions(+), 24 deletions(-) create mode 100644 t/lib-encoding.sh diff --git a/t/lib-encoding.sh b/t/lib-encoding.sh new file mode 100644 index 0000000000..2dabc8c73e --- /dev/null +++ b/t/lib-encoding.sh @@ -0,0 +1,25 @@ +# Encoding helpers + +test_lazy_prereq NO_UTF16_BOM ' + test $(printf abc | iconv -f UTF-8 -t UTF-16 | wc -c) = 6 +' + +test_lazy_prereq NO_UTF32_BOM ' + test $(printf abc | iconv -f UTF-8 -t UTF-32 | wc -c) = 12 +' + +write_utf16 () { + if test_have_prereq NO_UTF16_BOM + then + printf '\376\377' + fi && + iconv -f UTF-8 -t UTF-16 +} + +write_utf32 () { + if test_have_prereq NO_UTF32_BOM + then + printf '\0\0\376\377' + fi && + iconv -f UTF-8 -t UTF-32 +} diff --git a/t/t0028-working-tree-encoding.sh b/t/t0028-working-tree-encoding.sh index f970a9806b..82905a2156 100755 --- a/t/t0028-working-tree-encoding.sh +++ b/t/t0028-working-tree-encoding.sh @@ -6,33 +6,10 @@ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME . ./test-lib.sh +. "$TEST_DIRECTORY/lib-encoding.sh" GIT_TRACE_WORKING_TREE_ENCODING=1 && export GIT_TRACE_WORKING_TREE_ENCODING -test_lazy_prereq NO_UTF16_BOM ' - test $(printf abc | iconv -f UTF-8 -t UTF-16 | wc -c) = 6 -' - -test_lazy_prereq NO_UTF32_BOM ' - test $(printf abc | iconv -f UTF-8 -t UTF-32 | wc -c) = 12 -' - -write_utf16 () { - if test_have_prereq NO_UTF16_BOM - then - printf '\376\377' - fi && - iconv -f UTF-8 -t UTF-16 -} - -write_utf32 () { - if test_have_prereq NO_UTF32_BOM - then - printf '\0\0\376\377' - fi && - iconv -f UTF-8 -t UTF-32 -} - test_expect_success 'setup test files' ' git config core.eol lf && From patchwork Fri Apr 30 21:40:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12234455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0243C433ED for ; Fri, 30 Apr 2021 21:41:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AC8B461476 for ; Fri, 30 Apr 2021 21:41:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232760AbhD3Vlv (ORCPT ); Fri, 30 Apr 2021 17:41:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232369AbhD3Vlq (ORCPT ); Fri, 30 Apr 2021 17:41:46 -0400 Received: from mail-qv1-xf35.google.com (mail-qv1-xf35.google.com [IPv6:2607:f8b0:4864:20::f35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B573CC06138B for ; Fri, 30 Apr 2021 14:40:57 -0700 (PDT) Received: by mail-qv1-xf35.google.com with SMTP id r13so184129qvm.7 for ; Fri, 30 Apr 2021 14:40:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=24zXgh5ERhP6WetWcVX1R2rce49VOwH6kleFlHsnwFc=; b=ViMFUUz4FWBPu9AXek/WW0+RdZmhaAIMRs5dW6VsyvPrw0BExgaThYfDXLbBtaCXZ0 DgCcC2UwptrTSFwfse56gNYWLZ/IAG3s0BYXOL4EohCG8a8ybB79RrBNtMYBVdklSyBB f//6EoE7sPtolaYF41tpwvJeYg8x6d1ADE1bHoYzftwBwJpmN1FSfa6SLdprcylDIXfz JokFezsdGD5fcxpveBtTDTohRsJXmsQaIShEi9ytSyBGifZn8r20hgucoKLBBWZBCEnv +y0EU26vCVmgs50jY31rb16cu6N57AcatSm9sJPhKL9c5/EcraZQWVQgU/wraiIlwOsa 4nmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=24zXgh5ERhP6WetWcVX1R2rce49VOwH6kleFlHsnwFc=; b=cX7zjMC6rKTRgEjsZ4SVEmaKiHW5m5XsPZlFQEmweSLVBSfX2YhyZ2pl2ylh0rlZuR d/c3a7srYrFyxzktMz4KpZfah2Yv5UNDBOSf9h0xIdWgEc2cjRpFmDnhBPNSk6QLxgGF b/6QUSD6Vvj5Fa+oQb5+KyWYi+++uZ15NaZhY1b9MhTapd3zCqP8OQszTcsx4BrPZ5BN iePDp3bHVm2+MZdd5q4yYg2naK67o2JIROOl87YxJ6xOPkxJcgCaFEvK3NWy3whm0RqK H8zflwWWch2+09YVwU87nqq/hMDrJhy3QwipMTf2U2c3tGlXpXw6enY0QaQFS96ElzlV T5tw== X-Gm-Message-State: AOAM53276YpTJ5lFM4He9qPXKTO4Vdvge584VVggDFi7zktd40gEJzPO /VH7K9Roro3LB74oMSYoFLA+l8YD1ioGow== X-Google-Smtp-Source: ABdhPJyHldMVGydbhIyVXeKmGy67hFYh2B2HN9pApQCygeJ6QjecOCqSEetb+Hs2cQAe10T8q0jsng== X-Received: by 2002:a05:6214:2467:: with SMTP id im7mr7962359qvb.59.1619818856627; Fri, 30 Apr 2021 14:40:56 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id j13sm3123718qth.57.2021.04.30.14.40.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Apr 2021 14:40:56 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com Subject: [PATCH v2 7/8] parallel-checkout: add tests related to .gitattributes Date: Fri, 30 Apr 2021 18:40:34 -0300 Message-Id: X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add tests to confirm that the `struct conv_attrs` data is correctly passed from the main process to the workers, and that they can properly convert the blobs before writing them to the working tree. Also check that parallel-ineligible entries, such as regular files that require external filters, are correctly smudge and written when parallel-checkout is enabled. Co-authored-by: Jeff Hostetler Signed-off-by: Matheus Tavares --- t/t2082-parallel-checkout-attributes.sh | 194 ++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100755 t/t2082-parallel-checkout-attributes.sh diff --git a/t/t2082-parallel-checkout-attributes.sh b/t/t2082-parallel-checkout-attributes.sh new file mode 100755 index 0000000000..2525457961 --- /dev/null +++ b/t/t2082-parallel-checkout-attributes.sh @@ -0,0 +1,194 @@ +#!/bin/sh + +test_description='parallel-checkout: attributes + +Verify that parallel-checkout correctly creates files that require +conversions, as specified in .gitattributes. The main point here is +to check that the conv_attr data is correctly sent to the workers +and that it contains sufficient information to smudge files +properly (without access to the index or attribute stack). +' + +TEST_NO_CREATE_REPO=1 +. ./test-lib.sh +. "$TEST_DIRECTORY/lib-parallel-checkout.sh" +. "$TEST_DIRECTORY/lib-encoding.sh" + +test_expect_success 'parallel-checkout with ident' ' + set_checkout_config 2 0 && + git init ident && + ( + cd ident && + echo "A ident" >.gitattributes && + echo "\$Id\$" >A && + echo "\$Id\$" >B && + git add -A && + git commit -m id && + + rm A B && + test_checkout_workers 2 git reset --hard && + hexsz=$(test_oid hexsz) && + grep -E "\\\$Id: [0-9a-f]{$hexsz} \\\$" A && + grep "\\\$Id\\\$" B + ) +' + +test_expect_success 'parallel-checkout with re-encoding' ' + set_checkout_config 2 0 && + git init encoding && + ( + cd encoding && + echo text >utf8-text && + write_utf16 utf16-text && + + echo "A working-tree-encoding=UTF-16" >.gitattributes && + cp utf16-text A && + cp utf8-text B && + git add A B .gitattributes && + git commit -m encoding && + + # Check that A is stored in UTF-8 + git cat-file -p :A >A.internal && + test_cmp_bin utf8-text A.internal && + + rm A B && + test_checkout_workers 2 git checkout A B && + + # Check that A (and only A) is re-encoded during checkout + test_cmp_bin utf16-text A && + test_cmp_bin utf8-text B + ) +' + +test_expect_success 'parallel-checkout with eol conversions' ' + set_checkout_config 2 0 && + git init eol && + ( + cd eol && + printf "multi\r\nline\r\ntext" >crlf-text && + printf "multi\nline\ntext" >lf-text && + + git config core.autocrlf false && + echo "A eol=crlf" >.gitattributes && + cp crlf-text A && + cp lf-text B && + git add A B .gitattributes && + git commit -m eol && + + # Check that A is stored with LF format + git cat-file -p :A >A.internal && + test_cmp_bin lf-text A.internal && + + rm A B && + test_checkout_workers 2 git checkout A B && + + # Check that A (and only A) is converted to CRLF during checkout + test_cmp_bin crlf-text A && + test_cmp_bin lf-text B + ) +' + +# Entries that require an external filter are not eligible for parallel +# checkout. Check that both the parallel-eligible and non-eligible entries are +# properly writen in a single checkout operation. +# +test_expect_success 'parallel-checkout and external filter' ' + set_checkout_config 2 0 && + git init filter && + ( + cd filter && + write_script <<-\EOF rot13.sh && + tr \ + "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ" \ + "nopqrstuvwxyzabcdefghijklmNOPQRSTUVWXYZABCDEFGHIJKLM" + EOF + + git config filter.rot13.clean "\"$(pwd)/rot13.sh\"" && + git config filter.rot13.smudge "\"$(pwd)/rot13.sh\"" && + git config filter.rot13.required true && + + echo abcd >original && + echo nopq >rot13 && + + echo "A filter=rot13" >.gitattributes && + cp original A && + cp original B && + cp original C && + git add A B C .gitattributes && + git commit -m filter && + + # Check that A (and only A) was cleaned + git cat-file -p :A >A.internal && + test_cmp rot13 A.internal && + git cat-file -p :B >B.internal && + test_cmp original B.internal && + git cat-file -p :C >C.internal && + test_cmp original C.internal && + + rm A B C *.internal && + test_checkout_workers 2 git checkout A B C && + + # Check that A (and only A) was smudged during checkout + test_cmp original A && + test_cmp original B && + test_cmp original C + ) +' + +# The delayed queue is independent from the parallel queue, and they should be +# able to work together in the same checkout process. +# +test_expect_success PERL 'parallel-checkout and delayed checkout' ' + write_script rot13-filter.pl "$PERL_PATH" \ + <"$TEST_DIRECTORY"/t0021/rot13-filter.pl && + + test_config_global filter.delay.process \ + "\"$(pwd)/rot13-filter.pl\" --always-delay \"$(pwd)/delayed.log\" clean smudge delay" && + test_config_global filter.delay.required true && + + echo "abcd" >original && + echo "nopq" >rot13 && + + git init delayed && + ( + cd delayed && + echo "*.d filter=delay" >.gitattributes && + cp ../original W.d && + cp ../original X.d && + cp ../original Y && + cp ../original Z && + git add -A && + git commit -m delayed && + + # Check that *.d files were cleaned + git cat-file -p :W.d >W.d.internal && + test_cmp W.d.internal ../rot13 && + git cat-file -p :X.d >X.d.internal && + test_cmp X.d.internal ../rot13 && + git cat-file -p :Y >Y.internal && + test_cmp Y.internal ../original && + git cat-file -p :Z >Z.internal && + test_cmp Z.internal ../original && + + rm * + ) && + + set_checkout_config 2 0 && + test_checkout_workers 2 git -C delayed checkout -f && + verify_checkout delayed && + + # Check that the *.d files got to the delay queue and were filtered + grep "smudge W.d .* \[DELAYED\]" delayed.log && + grep "smudge X.d .* \[DELAYED\]" delayed.log && + test_cmp delayed/W.d original && + test_cmp delayed/X.d original && + + # Check that the parallel-eligible entries went to the right queue and + # were not filtered + ! grep "smudge Y .* \[DELAYED\]" delayed.log && + ! grep "smudge Z .* \[DELAYED\]" delayed.log && + test_cmp delayed/Y original && + test_cmp delayed/Z original +' + +test_done From patchwork Fri Apr 30 21:40:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12234457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A5C3C43462 for ; Fri, 30 Apr 2021 21:41:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 53B4B61477 for ; Fri, 30 Apr 2021 21:41:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235377AbhD3Vlw (ORCPT ); Fri, 30 Apr 2021 17:41:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231782AbhD3Vls (ORCPT ); Fri, 30 Apr 2021 17:41:48 -0400 Received: from mail-qk1-x72e.google.com (mail-qk1-x72e.google.com [IPv6:2607:f8b0:4864:20::72e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A2A51C06174A for ; Fri, 30 Apr 2021 14:40:59 -0700 (PDT) Received: by mail-qk1-x72e.google.com with SMTP id o5so72282537qkb.0 for ; Fri, 30 Apr 2021 14:40:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qHpZlOkHakUer84LEdkSVmO9C57GByvpp8AUl27heaY=; b=XpyKFhbdrRWkl/VavOLqqY0glLZUt63aTHw+oIuvN6lpYaByu+10AEciyuVqHvSAIt tuXbA+YN49aQdkrQ63vzwkUrAAJbwKEb293ShRjcT1rOv6/DY1RUfRSv9Sk9OMpCRFp5 Z0LZo+UqEqcm4+gd1lKW3wt3KjdPaYgKVeWNY9LfF5oNjifRiIOyL11PmhirbBjSLcFB B2bPLcPKfIaECpbzgCkm+peyeVwiFzyu4YN/hMIFPKM0OuIidRrmPyvt1lD6K8jjOKXa qKAqGZhQynNi9+pdHzSmpeW/UWpg85k7v1XbRqypNMTgXk9Uanp/87z4m0ihK4du2LyW V1mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qHpZlOkHakUer84LEdkSVmO9C57GByvpp8AUl27heaY=; b=gdArkDHNHnrnlgYEi4WG5qoRUAXpNOylzzr5kwlmcJ0oQQNFBJrWxmHV+URMy8bSzb i74bhVCNlyclbO2VnzjPRyHSKS+HcZrOjF1d3XIEWwnI8m8lrYTSmiOmPLigoDSiXGEc 0et7KsksNWfs9TyP21R4S3VYuzczyuESCgZHocbGBSRY+FxIqdZEyZEyOWoFHlELpKvg eNp/V3oZN6ukJzCU1kP5lAWNK5actU3nn1t1fX9lYKVqCdJwQo0w2GND43jD0NoV4cT3 I2aEGdK/okscUKi6uu77H4pbCkb2bhE6ZgKgUcv9YKySTzLSw/SzM74WLtr3TFzqlIoK 9Atw== X-Gm-Message-State: AOAM530OpJruYWf/GRS++csXdZ7YNjoM8gp/FAg/of0QGVJ/4tCspZ9C az6VuAhh+G3GwCMR2UZPz+sJdZ5F8AdKAw== X-Google-Smtp-Source: ABdhPJwynfxRhhWlTirWCu/kDu72n1CoQgbbbYRluEcpP+y8mte59aiKnRij0Hb7SZYDwLP/mYpDKQ== X-Received: by 2002:a37:9dd8:: with SMTP id g207mr51109qke.340.1619818858646; Fri, 30 Apr 2021 14:40:58 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id j13sm3123718qth.57.2021.04.30.14.40.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Apr 2021 14:40:58 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com Subject: [PATCH v2 8/8] ci: run test round with parallel-checkout enabled Date: Fri, 30 Apr 2021 18:40:35 -0300 Message-Id: <1bc5c523c5f8522facf5c08b07fd58fb4a99987f.1619818517.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org We already have tests for the basic parallel-checkout operations. But this code can also run be executed by other commands, such as git-read-tree and git-sparse-checkout, which are currently not tested with multiple workers. To promote a wider test coverage without duplicating tests: 1. Add the GIT_TEST_CHECKOUT_WORKERS environment variable, to optionally force parallel-checkout execution during the whole test suite. 2. Set this variable (with a value of 2) in the second test round of our linux-gcc CI job. This round runs `make test` again with some optional GIT_TEST_* variables enabled, so there is no additional overhead in exercising the parallel-checkout code here. Note that tests checking out less than two parallel-eligible entries will fall back to the sequential mode. Nevertheless, it's still a good exercise for the parallel-checkout framework as the fallback codepath also writes the queued entries using the parallel-checkout functions (only without spawning any worker). Signed-off-by: Matheus Tavares --- ci/run-build-and-tests.sh | 1 + parallel-checkout.c | 14 ++++++++++++++ t/README | 4 ++++ t/lib-parallel-checkout.sh | 3 +++ 4 files changed, 22 insertions(+) diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index a66b5e8c75..23b28e7391 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -25,6 +25,7 @@ linux-gcc) export GIT_TEST_ADD_I_USE_BUILTIN=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master export GIT_TEST_WRITE_REV_INDEX=1 + export GIT_TEST_CHECKOUT_WORKERS=2 make test ;; linux-clang) diff --git a/parallel-checkout.c b/parallel-checkout.c index 6fb3f1e6c9..6b1af32bb3 100644 --- a/parallel-checkout.c +++ b/parallel-checkout.c @@ -35,6 +35,20 @@ static const int DEFAULT_NUM_WORKERS = 1; void get_parallel_checkout_configs(int *num_workers, int *threshold) { + char *env_workers = getenv("GIT_TEST_CHECKOUT_WORKERS"); + + if (env_workers && *env_workers) { + if (strtol_i(env_workers, 10, num_workers)) { + die("invalid value for GIT_TEST_CHECKOUT_WORKERS: '%s'", + env_workers); + } + if (*num_workers < 1) + *num_workers = online_cpus(); + + *threshold = 0; + return; + } + if (git_config_get_int("checkout.workers", num_workers)) *num_workers = DEFAULT_NUM_WORKERS; else if (*num_workers < 1) diff --git a/t/README b/t/README index fd9375b146..a194488f27 100644 --- a/t/README +++ b/t/README @@ -436,6 +436,10 @@ and "sha256". GIT_TEST_WRITE_REV_INDEX=, when true enables the 'pack.writeReverseIndex' setting. +GIT_TEST_CHECKOUT_WORKERS= overrides the 'checkout.workers' setting +to and 'checkout.thresholdForParallelism' to 0, forcing the +execution of the parallel-checkout code. + Naming Tests ------------ diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh index d6740425b1..21f5759732 100644 --- a/t/lib-parallel-checkout.sh +++ b/t/lib-parallel-checkout.sh @@ -1,5 +1,8 @@ # Helpers for tests invoking parallel-checkout +# Parallel checkout tests need full control of the number of workers +unset GIT_TEST_CHECKOUT_WORKERS + set_checkout_config () { if test $# -ne 2 then