From patchwork Tue May 4 16:27:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12238273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C462C433B4 for ; Tue, 4 May 2021 16:27:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D4CD4613BC for ; Tue, 4 May 2021 16:27:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231807AbhEDQ2q (ORCPT ); Tue, 4 May 2021 12:28:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231651AbhEDQ2m (ORCPT ); Tue, 4 May 2021 12:28:42 -0400 Received: from mail-qt1-x836.google.com (mail-qt1-x836.google.com [IPv6:2607:f8b0:4864:20::836]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F30BCC061574 for ; Tue, 4 May 2021 09:27:47 -0700 (PDT) Received: by mail-qt1-x836.google.com with SMTP id n22so6739033qtk.9 for ; Tue, 04 May 2021 09:27:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VybqBhJ09WiaZ1uEY5aPrt2z+Ibg179/hVRuauV7WIM=; b=eCPyO/xqcDSA4qms51jkJhwG/tT7sjsg+bFWjsMYfrC8uVPoSJADVpIdHZJK8v6Tg7 HpDMDAsgDHaBPCfl3PixKMhPV6xi4XlGmnZ6236JFX3CW8/lT3qFzScUzta9AZUCigKU y48Tx9fHTnmbLBoghpkhBWhAwljWuyvSKPRDbOpRRfpmwu5WwRzNd0zEwoz39NfQlmH0 G192/DMuNjGhlFb3Q3Sv7vP1LS6VFej/cLxMUbhAA9GxoCPj6T3k1IBg7/esBkg/mCHO 7/W8B9Q3E/ET7ln+SDN2iq74lT8qyaaIyhha5bR0PhZl+Xga6HToLAouGI2xJmv9EXWZ d/6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VybqBhJ09WiaZ1uEY5aPrt2z+Ibg179/hVRuauV7WIM=; b=n+YpGyxhHCJj1geCGOqwrzyaRk2LclSdLhprsldhuK5o33YplQ/mlmZkQNCQOzalz0 IoQDpLRanyMFoW2m0MeOFrIFfaqZK9qPOPliVE+TKwymZTQEpGFJ31BBXhktUCRKNuv9 vU30ZNPmP0k3aKNZxF2HmxAiNASAxZ2FShbmgD8RHwUiPXfu8YQrPAt1bvZQB7BXflwR XG4BC073RPQtV05vFuohF8lLES8bZ5ZdyYmY+9mznjPYPyWP/duQrBaGfSCSgw6fLSv/ zKkmbknaG9pEka8SZ4kErXwdPKN91amqj9VzWMhlD0EtPw2BY9lAWMvMCwo7qLu8eMP9 Mgkw== X-Gm-Message-State: AOAM530Cbp1sZ+Wq6yiCz8CZ+RAuiLHm2lcYgnwxY99BMBRSwVCbJcpE ZGM3AsvdIPnoAFdO6TcKrm9hr3CItMA77A== X-Google-Smtp-Source: ABdhPJwudX0U0pohfCTdI4QXS5bJfeba9YlvyhZ3LRMZPkeB5GcL9gG3D9GS8+hBLywSMObgovIkzA== X-Received: by 2002:ac8:7314:: with SMTP id x20mr23126397qto.320.1620145666756; Tue, 04 May 2021 09:27:46 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id x19sm11202689qkx.107.2021.05.04.09.27.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 May 2021 09:27:46 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com, tboegi@web.de Subject: [PATCH v3 1/8] make_transient_cache_entry(): optionally alloc from mem_pool Date: Tue, 4 May 2021 13:27:28 -0300 Message-Id: X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Allow make_transient_cache_entry() to optionally receive a mem_pool struct in which it should allocate the entry. This will be used in the following patch, to store some transient entries which should persist until parallel checkout finishes. Signed-off-by: Matheus Tavares --- builtin/checkout--worker.c | 2 +- builtin/checkout.c | 2 +- builtin/difftool.c | 2 +- cache.h | 14 +++++++++----- read-cache.c | 14 ++++++++++---- unpack-trees.c | 2 +- 6 files changed, 23 insertions(+), 13 deletions(-) diff --git a/builtin/checkout--worker.c b/builtin/checkout--worker.c index 31e0de2f7e..289a9b8f89 100644 --- a/builtin/checkout--worker.c +++ b/builtin/checkout--worker.c @@ -39,7 +39,7 @@ static void packet_to_pc_item(const char *buffer, int len, } memset(pc_item, 0, sizeof(*pc_item)); - pc_item->ce = make_empty_transient_cache_entry(fixed_portion->name_len); + pc_item->ce = make_empty_transient_cache_entry(fixed_portion->name_len, NULL); pc_item->ce->ce_namelen = fixed_portion->name_len; pc_item->ce->ce_mode = fixed_portion->ce_mode; memcpy(pc_item->ce->name, variant, pc_item->ce->ce_namelen); diff --git a/builtin/checkout.c b/builtin/checkout.c index 5bd9128d1a..a597f31d53 100644 --- a/builtin/checkout.c +++ b/builtin/checkout.c @@ -291,7 +291,7 @@ static int checkout_merged(int pos, const struct checkout *state, int *nr_checko if (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid)) die(_("Unable to add merge result for '%s'"), path); free(result_buf.ptr); - ce = make_transient_cache_entry(mode, &oid, path, 2); + ce = make_transient_cache_entry(mode, &oid, path, 2, NULL); if (!ce) die(_("make_cache_entry failed for path '%s'"), path); status = checkout_entry(ce, state, NULL, nr_checkouts); diff --git a/builtin/difftool.c b/builtin/difftool.c index 0202a43052..89334b77fb 100644 --- a/builtin/difftool.c +++ b/builtin/difftool.c @@ -323,7 +323,7 @@ static int checkout_path(unsigned mode, struct object_id *oid, struct cache_entry *ce; int ret; - ce = make_transient_cache_entry(mode, oid, path, 0); + ce = make_transient_cache_entry(mode, oid, path, 0, NULL); ret = checkout_entry(ce, state, NULL, NULL); discard_cache_entry(ce); diff --git a/cache.h b/cache.h index b785ffb383..0e2b952647 100644 --- a/cache.h +++ b/cache.h @@ -370,16 +370,20 @@ struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t name_len); /* - * Create a cache_entry that is not intended to be added to an index. - * Caller is responsible for discarding the cache_entry - * with `discard_cache_entry`. + * Create a cache_entry that is not intended to be added to an index. If + * `ce_mem_pool` is not NULL, the entry is allocated within the given memory + * pool. Caller is responsible for discarding "loose" entries with + * `discard_cache_entry()` and the memory pool with + * `mem_pool_discard(ce_mem_pool, should_validate_cache_entries())`. */ struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct object_id *oid, const char *path, - int stage); + int stage, + struct mem_pool *ce_mem_pool); -struct cache_entry *make_empty_transient_cache_entry(size_t name_len); +struct cache_entry *make_empty_transient_cache_entry(size_t len, + struct mem_pool *ce_mem_pool); /* * Discard cache entry. diff --git a/read-cache.c b/read-cache.c index 72a1d339c9..86f95fe62e 100644 --- a/read-cache.c +++ b/read-cache.c @@ -839,8 +839,11 @@ struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t le return mem_pool__ce_calloc(find_mem_pool(istate), len); } -struct cache_entry *make_empty_transient_cache_entry(size_t len) +struct cache_entry *make_empty_transient_cache_entry(size_t len, + struct mem_pool *ce_mem_pool) { + if (ce_mem_pool) + return mem_pool__ce_calloc(ce_mem_pool, len); return xcalloc(1, cache_entry_size(len)); } @@ -874,8 +877,11 @@ struct cache_entry *make_cache_entry(struct index_state *istate, return ret; } -struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct object_id *oid, - const char *path, int stage) +struct cache_entry *make_transient_cache_entry(unsigned int mode, + const struct object_id *oid, + const char *path, + int stage, + struct mem_pool *ce_mem_pool) { struct cache_entry *ce; int len; @@ -886,7 +892,7 @@ struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct o } len = strlen(path); - ce = make_empty_transient_cache_entry(len); + ce = make_empty_transient_cache_entry(len, ce_mem_pool); oidcpy(&ce->oid, oid); memcpy(ce->name, path, len); diff --git a/unpack-trees.c b/unpack-trees.c index 7a1804c314..f88a69f8e7 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1038,7 +1038,7 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, size_t len = traverse_path_len(info, tree_entry_len(n)); struct cache_entry *ce = is_transient ? - make_empty_transient_cache_entry(len) : + make_empty_transient_cache_entry(len, NULL) : make_empty_cache_entry(istate, len); ce->ce_mode = create_ce_mode(n->mode); From patchwork Tue May 4 16:27:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12238275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 102E3C433ED for ; Tue, 4 May 2021 16:27:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D13D5613BE for ; Tue, 4 May 2021 16:27:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231710AbhEDQ2r (ORCPT ); Tue, 4 May 2021 12:28:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231824AbhEDQ2q (ORCPT ); Tue, 4 May 2021 12:28:46 -0400 Received: from mail-qk1-x72d.google.com (mail-qk1-x72d.google.com [IPv6:2607:f8b0:4864:20::72d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED097C061761 for ; Tue, 4 May 2021 09:27:50 -0700 (PDT) Received: by mail-qk1-x72d.google.com with SMTP id u20so9116527qku.10 for ; Tue, 04 May 2021 09:27:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kZEAYX8O4gX/K6wkQ1F+Tr+BVJRT7DoZQlezwq4K2u4=; b=s/pROxKXDVG8+dcXjnq/1ttAZn0Ch9Yu9iGYB6NKw4SrZ8apTza+wpSYmKjF5tTbU5 s0z/Oj8dLLU2h3H9zwhIfVjF/1bMNIre5RsuzcDlh/2Q9kw+DkXYKJCfU5BhQ1mlOpWZ 03K5r8Vk7ZmlW1VtVgplaTujaOnq6GDIlIWG2bBPxChS4l3YLDN1vat9CR2BiXbtSS5d TtbftalugNl34wFWOM3bul4opyyrQuV7DstG4F/viLd0fpVVxxs/VgbBumf34dl8AlKu GjwygpZzNnKfhb1Hs2yMnIkus99GsZ3ek+4REBhju+DICMvaUT7wSLzCveaW8iODaq1F TSzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kZEAYX8O4gX/K6wkQ1F+Tr+BVJRT7DoZQlezwq4K2u4=; b=KuqFx7nifeqOq6iQ/WTcSpBQdzLZfuNEyyfsyE7ooRPCPKRcsscb2cPncBjtFK8EdE WuE9l8Z8mTKREbRBv1e54N7VCga2phlPB3ntJ6aIGmx0nIy7bZrw/91+ZmLbwYfLzhg2 xxkThrvS9VK9pETc878YVGCeGvcDdo74JZOhRRmX01ZABJyn7v7JvRCOYMDQJeLeo5KN NXYi2DeZmnFC1CVY+hIU4Xc27qxad8UXVE++K4ywJN0BMOxO7MUfI3/uUKM83Zp9FuyC 1IX8qGl0ykH0GlcBXu3+Er95Z2ti4Z70O4PmcR11DIoACvKS0CBOYlNTIKX03SmydLdL 6HSg== X-Gm-Message-State: AOAM533IZouHw7uhp14XMvuM4nK2Xf5nYot7Doxf73Z2MTzPvbHgTdxi czUCgCIbcehMFGhDMiyS1KDNeAsEzEVzdw== X-Google-Smtp-Source: ABdhPJwrB7C4JOK94PRCA2/DUOHIfMdK9Tn5trykXqJMvX8pfs4bEhTyRmORvkvAB7uJoP5n398mpA== X-Received: by 2002:a37:7006:: with SMTP id l6mr25359325qkc.137.1620145669691; Tue, 04 May 2021 09:27:49 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id x19sm11202689qkx.107.2021.05.04.09.27.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 May 2021 09:27:49 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com, tboegi@web.de Subject: [PATCH v3 2/8] builtin/checkout.c: complete parallel checkout support Date: Tue, 4 May 2021 13:27:29 -0300 Message-Id: X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Pathspec-limited checkouts (like `git checkout *.txt`) are performed by a code path that doesn't yet support parallel checkout because it calls checkout_entry() directly, instead of unpack_trees(). Let's add parallel checkout support for this code path too. The transient cache entries allocated in checkout_merged() are now allocated in a mem_pool which is only discarded after parallel checkout finishes. This is done because the entries need to be valid when run_parallel_checkout() is called. Signed-off-by: Matheus Tavares --- builtin/checkout.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/builtin/checkout.c b/builtin/checkout.c index a597f31d53..2632dd9eff 100644 --- a/builtin/checkout.c +++ b/builtin/checkout.c @@ -27,6 +27,7 @@ #include "wt-status.h" #include "xdiff-interface.h" #include "entry.h" +#include "parallel-checkout.h" static const char * const checkout_usage[] = { N_("git checkout [] "), @@ -230,7 +231,8 @@ static int checkout_stage(int stage, const struct cache_entry *ce, int pos, return error(_("path '%s' does not have their version"), ce->name); } -static int checkout_merged(int pos, const struct checkout *state, int *nr_checkouts) +static int checkout_merged(int pos, const struct checkout *state, + int *nr_checkouts, struct mem_pool *ce_mem_pool) { struct cache_entry *ce = active_cache[pos]; const char *path = ce->name; @@ -291,11 +293,10 @@ static int checkout_merged(int pos, const struct checkout *state, int *nr_checko if (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid)) die(_("Unable to add merge result for '%s'"), path); free(result_buf.ptr); - ce = make_transient_cache_entry(mode, &oid, path, 2, NULL); + ce = make_transient_cache_entry(mode, &oid, path, 2, ce_mem_pool); if (!ce) die(_("make_cache_entry failed for path '%s'"), path); status = checkout_entry(ce, state, NULL, nr_checkouts); - discard_cache_entry(ce); return status; } @@ -359,16 +360,22 @@ static int checkout_worktree(const struct checkout_opts *opts, int nr_checkouts = 0, nr_unmerged = 0; int errs = 0; int pos; + int pc_workers, pc_threshold; + struct mem_pool ce_mem_pool; state.force = 1; state.refresh_cache = 1; state.istate = &the_index; + mem_pool_init(&ce_mem_pool, 0); + get_parallel_checkout_configs(&pc_workers, &pc_threshold); init_checkout_metadata(&state.meta, info->refname, info->commit ? &info->commit->object.oid : &info->oid, NULL); enable_delayed_checkout(&state); + if (pc_workers > 1) + init_parallel_checkout(); /* TODO: audit for interaction with sparse-index. */ ensure_full_index(&the_index); @@ -387,10 +394,15 @@ static int checkout_worktree(const struct checkout_opts *opts, &nr_checkouts, opts->overlay_mode); else if (opts->merge) errs |= checkout_merged(pos, &state, - &nr_unmerged); + &nr_unmerged, + &ce_mem_pool); pos = skip_same_name(ce, pos) - 1; } } + if (pc_workers > 1) + errs |= run_parallel_checkout(&state, pc_workers, pc_threshold, + NULL, NULL); + mem_pool_discard(&ce_mem_pool, should_validate_cache_entries()); remove_marked_cache_entries(&the_index, 1); remove_scheduled_dirs(); errs |= finish_delayed_checkout(&state, &nr_checkouts); From patchwork Tue May 4 16:27:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12238277 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31B22C43461 for ; Tue, 4 May 2021 16:27:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 02222613BE for ; Tue, 4 May 2021 16:27:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231785AbhEDQ2t (ORCPT ); Tue, 4 May 2021 12:28:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56580 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231739AbhEDQ2s (ORCPT ); Tue, 4 May 2021 12:28:48 -0400 Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5FB6DC061574 for ; Tue, 4 May 2021 09:27:53 -0700 (PDT) Received: by mail-qv1-xf30.google.com with SMTP id i8so4748639qvv.0 for ; Tue, 04 May 2021 09:27:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yCCZkZB+9Ylm+ycwoLpyU0b64+dJciURMkmf5PJr15g=; b=c9s+mcExEOGC1hxOpM7AeSk3hbflTaQJrp0kJmRuAjSxEQjSYS6nVZyqGzzkFWujdj 93GWxSKU+0SU8fbKUR613FQ8vZ6Aygl19bGD18HilhXDKNVLcEt8SylfKv+LYK6VIx/1 mYBSAFLGl1N19Zv+xDof/iWbpM7T6nTgUc/OPsLZfbxoHhHDvtg/UCGBZZ4gXOXoKiHA I6gQpPLM1yoU65eWXiDsWDx2S8Hsr4/nzkl9JW7T/miucmee0eo/O/oDpxtZvDB38BF6 /dDQPwrc7ilyRis17M2WLoeGpnT1Zht69Ws40k2XIKDdXoskjclWInN9psVmcfg4t1Wn 2nxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yCCZkZB+9Ylm+ycwoLpyU0b64+dJciURMkmf5PJr15g=; b=nnwWrZuER9cnT8t8SloTyXESZTKWNVjzKvSmDwlqyGerkWIWvPA9B6Pll/QrWFhNxm l7IQ2M3rlkuga+50mL0C9fngP9atRS0kf4WzDdrd2t0wtfDMI5UBBmszI0rpENPFw03n jpmr/zLvYOPc/grjkRjAgXDXk7nHirZbh5jRkKyEFAyhHkGeBRdF5Bmk68C7SLbRfTBW pRv4JSJjd+87R+8O+jHBk+mfsm0Tdr1mX4PoOR6CoqvwtwNDXCe5IwFx6rUEjoTZecqU GO0evKjuRxEiFwNNZes95jYUVukGbCaEIMPXc5mJT1nHqYCV6fnq87kITcEYz3XH9qH+ +nxw== X-Gm-Message-State: AOAM532YgvBAHW3RE1P51k/mGdja0gR9+Tm/w9fKLEp9nqJ9MmWHbOXm Ag70aTlemCQZneSuvkW6BbslTgg2RO6YWw== X-Google-Smtp-Source: ABdhPJzTsRfq/CUo1r3tz8rc7KWkSoONhkhd9Jqaz3OWlDNj5LpNSgc3KrvxmkmdE82a5e2I7GAi8w== X-Received: by 2002:ad4:4081:: with SMTP id l1mr26606015qvp.24.1620145672272; Tue, 04 May 2021 09:27:52 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id x19sm11202689qkx.107.2021.05.04.09.27.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 May 2021 09:27:51 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com, tboegi@web.de Subject: [PATCH v3 3/8] checkout-index: add parallel checkout support Date: Tue, 4 May 2021 13:27:30 -0300 Message-Id: <916f391a46a1ccedee3fca52893d0339ff37864a.1620145501.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Allow checkout-index to use the parallel checkout framework, honoring the checkout.workers configuration. There are two code paths in checkout-index which call `checkout_entry()`, and thus, can make use of parallel checkout: `checkout_file()`, which is used to write paths explicitly given at the command line; and `checkout_all()`, which is used to write all paths in the index, when the `--all` option is given. In both operation modes, checkout-index doesn't abort immediately on a `checkout_entry()` failure. Instead, it tries to check out all remaining paths before exiting with a non-zero exit code. To keep this behavior when parallel checkout is being used, we must allow `run_parallel_checkout()` to try writing the queued entries before we exit, even if we already got an error code from a previous `checkout_entry()` call. However, `checkout_all()` doesn't return on errors, it calls `exit()` with code 128. We could make it call `run_parallel_checkout()` before exiting, but it makes the code easier to follow if we unify the exit path for both checkout-index modes at `cmd_checkout_index()`, and let this function take care of the interactions with the parallel checkout API. So let's do that. With this change, we also have to consider whether we want to keep using 128 as the error code for `git checkout-index --all`, while we use 1 for `git checkout-index ` (even when the actual error is the same). Since there is not much value in having code 128 only for `--all`, and there is no mention about it in the docs (so it's unlikely that changing it will break any existing script), let's make both modes exit with code 1 on `checkout_entry()` errors. Signed-off-by: Matheus Tavares --- builtin/checkout-index.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/builtin/checkout-index.c b/builtin/checkout-index.c index c9a3c71914..e21620d964 100644 --- a/builtin/checkout-index.c +++ b/builtin/checkout-index.c @@ -12,6 +12,7 @@ #include "cache-tree.h" #include "parse-options.h" #include "entry.h" +#include "parallel-checkout.h" #define CHECKOUT_ALL 4 static int nul_term_line; @@ -115,7 +116,7 @@ static int checkout_file(const char *name, const char *prefix) return -1; } -static void checkout_all(const char *prefix, int prefix_length) +static int checkout_all(const char *prefix, int prefix_length) { int i, errs = 0; struct cache_entry *last_ce = NULL; @@ -144,11 +145,7 @@ static void checkout_all(const char *prefix, int prefix_length) } if (last_ce && to_tempfile) write_tempfile_record(last_ce->name, prefix); - if (errs) - /* we have already done our error reporting. - * exit with the same code as die(). - */ - exit(128); + return !!errs; } static const char * const builtin_checkout_index_usage[] = { @@ -184,6 +181,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix) int force = 0, quiet = 0, not_new = 0; int index_opt = 0; int err = 0; + int pc_workers, pc_threshold; struct option builtin_checkout_index_options[] = { OPT_BOOL('a', "all", &all, N_("check out all files in the index")), @@ -238,6 +236,10 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix) hold_locked_index(&lock_file, LOCK_DIE_ON_ERROR); } + get_parallel_checkout_configs(&pc_workers, &pc_threshold); + if (pc_workers > 1) + init_parallel_checkout(); + /* Check out named files first */ for (i = 0; i < argc; i++) { const char *arg = argv[i]; @@ -277,12 +279,16 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix) strbuf_release(&buf); } + if (all) + err |= checkout_all(prefix, prefix_length); + + if (pc_workers > 1) + err |= run_parallel_checkout(&state, pc_workers, pc_threshold, + NULL, NULL); + if (err) return 1; - if (all) - checkout_all(prefix, prefix_length); - if (is_lock_file_locked(&lock_file) && write_locked_index(&the_index, &lock_file, COMMIT_LOCK)) die("Unable to write new index file"); From patchwork Tue May 4 16:27:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12238279 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D517C433B4 for ; Tue, 4 May 2021 16:27:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F208F613BC for ; Tue, 4 May 2021 16:27:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231827AbhEDQ2w (ORCPT ); Tue, 4 May 2021 12:28:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56596 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231824AbhEDQ2v (ORCPT ); Tue, 4 May 2021 12:28:51 -0400 Received: from mail-qk1-x730.google.com (mail-qk1-x730.google.com [IPv6:2607:f8b0:4864:20::730]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 135C5C061574 for ; Tue, 4 May 2021 09:27:56 -0700 (PDT) Received: by mail-qk1-x730.google.com with SMTP id 197so8839047qkl.12 for ; Tue, 04 May 2021 09:27:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0tEnEjfh4UFSUxC9OyF8P7l5RZeLtK76zdR1Km/Uhls=; b=fnF7xEKizlBJGSOz6VuLiPatB1myqIsPvRprciZUxKs6MMMQlbd11MLmZqT9VRyhHM Rln1JtxFEyrvNC+U9g3DSbc5Lqr0L16krvCPK3VT+ufyRCxq1fJXLthTXfyQElbC3OkX 746QwPoa9p77WVtUHP/kiGiPxCHh+OrfRN7D4BthP2WtkaSG1qSuoCsTE47oOZzliBVf YFn1MrRLPxpLd/KrK0jwKyN0BAJkgAiQtr+Ntb+o6/7Ng4MpDBjfFOebai9TuWAv+y3U tho4gshbH4FSNx1glpXPnhx6eU05e1xzTgcpI+kmF/iDW8yyf4zwxQY5CnRT2hDhC+5B E8TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0tEnEjfh4UFSUxC9OyF8P7l5RZeLtK76zdR1Km/Uhls=; b=MEc58Omgg3A3PorPcXCSs23yL2AhaOgg8YshCWvpIv0DBeN3MZZeALtmzBDZvFva7j 0aPqgZ6PVjsqMCdwk15bcOwq7nm8O9roavx1D+NMKtkPt4Q8r4/ie+/9diW3yy8t2PMQ Jit7/L3IHOaknEU+xwE/S3rB00OKsuwE9MuvtJ21+a78zkk3M0cspkVArf1ySYxA5lwb JtaSiFzK0gNesYWsi+kiJQ4r6FbcC0mdm8yLiGOSBOH9E+PSlPcvJOt7idaeInSmpy1d grusf+FXAmoaV5M1iDFqlsnBST0kptnYM2mVm+I7zmUgIjWitFfehSLvR411Jm7iNBN7 F+Pw== X-Gm-Message-State: AOAM530eGQxWEMrhZ6rD8BUA6IT33rDu27xvA8WOJJjEkiAwd/s0+NU/ KDiHzIf+oAExdCYnJmbgl3mMBWkmvwdjBQ== X-Google-Smtp-Source: ABdhPJxU7ivOhhpAlXY+zQtQPDnq809BqzBhMhwGewCKEr/QDF25f65eXv5tPDDCWsH5OV7/KP1MXw== X-Received: by 2002:a37:5b44:: with SMTP id p65mr24782969qkb.416.1620145674895; Tue, 04 May 2021 09:27:54 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id x19sm11202689qkx.107.2021.05.04.09.27.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 May 2021 09:27:54 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com, tboegi@web.de Subject: [PATCH v3 4/8] parallel-checkout: add tests for basic operations Date: Tue, 4 May 2021 13:27:31 -0300 Message-Id: <667777053ab853ef2f94b7ba4ab39f7241d2b49c.1620145501.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add tests to populate the working tree during clone and checkout using sequential and parallel mode, to confirm that they produce identical results. Also test basic checkout mechanics, such as checking for symlinks in the leading directories and the abidance to --force. Note: some helper functions are added to a common lib file which is only included by t2080 for now. But they will also be used by other parallel-checkout tests in the following patches. Co-authored-by: Jeff Hostetler Signed-off-by: Matheus Tavares --- t/lib-parallel-checkout.sh | 42 +++++ t/t2080-parallel-checkout-basics.sh | 229 ++++++++++++++++++++++++++++ 2 files changed, 271 insertions(+) create mode 100644 t/lib-parallel-checkout.sh create mode 100755 t/t2080-parallel-checkout-basics.sh diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh new file mode 100644 index 0000000000..f60b22ef34 --- /dev/null +++ b/t/lib-parallel-checkout.sh @@ -0,0 +1,42 @@ +# Helpers for tests invoking parallel-checkout + +set_checkout_config () { + if test $# -ne 2 + then + BUG "usage: set_checkout_config " + fi && + + test_config_global checkout.workers $1 && + test_config_global checkout.thresholdForParallelism $2 +} + +# Run "${@:2}" and check that $1 checkout workers were used +test_checkout_workers () { + if test $# -lt 2 + then + BUG "too few arguments to test_checkout_workers" + fi && + + local expected_workers=$1 && + shift && + + local trace_file=trace-test-checkout-workers && + rm -f "$trace_file" && + GIT_TRACE2="$(pwd)/$trace_file" "$@" && + + local workers=$(grep "child_start\[..*\] git checkout--worker" "$trace_file" | wc -l) && + test $workers -eq $expected_workers && + rm "$trace_file" +} + +# Verify that both the working tree and the index were created correctly +verify_checkout () { + if test $# -ne 1 + then + BUG "usage: verify_checkout " + fi && + + git -C "$1" diff-index --ignore-submodules=none --exit-code HEAD -- && + git -C "$1" status --porcelain >"$1".status && + test_must_be_empty "$1".status +} diff --git a/t/t2080-parallel-checkout-basics.sh b/t/t2080-parallel-checkout-basics.sh new file mode 100755 index 0000000000..7087818550 --- /dev/null +++ b/t/t2080-parallel-checkout-basics.sh @@ -0,0 +1,229 @@ +#!/bin/sh + +test_description='parallel-checkout basics + +Ensure that parallel-checkout basically works on clone and checkout, spawning +the required number of workers and correctly populating both the index and the +working tree. +' + +TEST_NO_CREATE_REPO=1 +. ./test-lib.sh +. "$TEST_DIRECTORY/lib-parallel-checkout.sh" + +# Test parallel-checkout with a branch switch containing a variety of file +# creations, deletions, and modifications, involving different entry types. +# The branches B1 and B2 have the following paths: +# +# B1 B2 +# a/a (file) a (file) +# b (file) b/b (file) +# +# c/c (file) c (symlink) +# d (symlink) d/d (file) +# +# e/e (file) e (submodule) +# f (submodule) f/f (file) +# +# g (submodule) g (symlink) +# h (symlink) h (submodule) +# +# Additionally, the following paths are present on both branches, but with +# different contents: +# +# i (file) i (file) +# j (symlink) j (symlink) +# k (submodule) k (submodule) +# +# And the following paths are only present in one of the branches: +# +# l/l (file) - +# - m/m (file) +# +test_expect_success 'setup repo for checkout with various types of changes' ' + git init sub && + ( + cd sub && + git checkout -b B2 && + echo B2 >file && + git add file && + git commit -m file && + + git checkout -b B1 && + echo B1 >file && + git add file && + git commit -m file + ) && + + git init various && + ( + cd various && + + git checkout -b B1 && + mkdir a c e && + echo a/a >a/a && + echo b >b && + echo c/c >c/c && + test_ln_s_add c d && + echo e/e >e/e && + git submodule add ../sub f && + git submodule add ../sub g && + test_ln_s_add c h && + + echo "B1 i" >i && + test_ln_s_add c j && + git submodule add -b B1 ../sub k && + mkdir l && + echo l/l >l/l && + + git add . && + git commit -m B1 && + + git checkout -b B2 && + git rm -rf :^.gitmodules :^k && + mkdir b d f && + echo a >a && + echo b/b >b/b && + test_ln_s_add b c && + echo d/d >d/d && + git submodule add ../sub e && + echo f/f >f/f && + test_ln_s_add b g && + git submodule add ../sub h && + + echo "B2 i" >i && + test_ln_s_add b j && + git -C k checkout B2 && + mkdir m && + echo m/m >m/m && + + git add . && + git commit -m B2 && + + git checkout --recurse-submodules B1 + ) +' + +for mode in sequential parallel sequential-fallback +do + case $mode in + sequential) workers=1 threshold=0 expected_workers=0 ;; + parallel) workers=2 threshold=0 expected_workers=2 ;; + sequential-fallback) workers=2 threshold=100 expected_workers=0 ;; + esac + + test_expect_success "$mode checkout" ' + repo=various_$mode && + cp -R various $repo && + + # The just copied files have more recent timestamps than their + # associated index entries. So refresh the cached timestamps + # to avoid an "entry not up-to-date" error from `git checkout`. + # We only have to do this for the submodules as `git checkout` + # will already refresh the superproject index before performing + # the up-to-date check. + # + git -C $repo submodule foreach "git update-index --refresh" && + + set_checkout_config $workers $threshold && + test_checkout_workers $expected_workers \ + git -C $repo checkout --recurse-submodules B2 && + verify_checkout $repo + ' +done + +for mode in parallel sequential-fallback +do + case $mode in + parallel) workers=2 threshold=0 expected_workers=2 ;; + sequential-fallback) workers=2 threshold=100 expected_workers=0 ;; + esac + + test_expect_success "$mode checkout on clone" ' + repo=various_${mode}_clone && + set_checkout_config $workers $threshold && + test_checkout_workers $expected_workers \ + git clone --recurse-submodules --branch B2 various $repo && + verify_checkout $repo + ' +done + +# Just to be paranoid, actually compare the working trees' contents directly. +test_expect_success 'compare the working trees' ' + rm -rf various_*/.git && + rm -rf various_*/*/.git && + + # We use `git diff` instead of `diff -r` because the latter would + # follow symlinks, and not all `diff` implementations support the + # `--no-dereference` option. + # + git diff --no-index various_sequential various_parallel && + git diff --no-index various_sequential various_parallel_clone && + git diff --no-index various_sequential various_sequential-fallback && + git diff --no-index various_sequential various_sequential-fallback_clone +' + +# Currently, each submodule is checked out in a separated child process, but +# these subprocesses must also be able to use parallel checkout workers to +# write the submodules' entries. +test_expect_success 'submodules can use parallel checkout' ' + set_checkout_config 2 0 && + git init super && + ( + cd super && + git init sub && + test_commit -C sub A && + test_commit -C sub B && + git submodule add ./sub && + git commit -m sub && + rm sub/* && + test_checkout_workers 2 git checkout --recurse-submodules . + ) +' + +test_expect_success 'parallel checkout respects --[no]-force' ' + set_checkout_config 2 0 && + git init dirty && + ( + cd dirty && + mkdir D && + test_commit D/F && + test_commit F && + + rm -rf D && + echo changed >D && + echo changed >F.t && + + # We expect 0 workers because there is nothing to be done + test_checkout_workers 0 git checkout HEAD && + test_path_is_file D && + grep changed D && + grep changed F.t && + + test_checkout_workers 2 git checkout --force HEAD && + test_path_is_dir D && + grep D/F D/F.t && + grep F F.t + ) +' + +test_expect_success SYMLINKS 'parallel checkout checks for symlinks in leading dirs' ' + set_checkout_config 2 0 && + git init symlinks && + ( + cd symlinks && + mkdir D untracked && + # Commit 2 files to have enough work for 2 parallel workers + test_commit D/A && + test_commit D/B && + rm -rf D && + ln -s untracked D && + + test_checkout_workers 2 git checkout --force HEAD && + ! test -h D && + grep D/A D/A.t && + grep D/B D/B.t + ) +' + +test_done From patchwork Tue May 4 16:27:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12238281 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7013BC433B4 for ; Tue, 4 May 2021 16:28:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4B197613BC for ; Tue, 4 May 2021 16:28:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231826AbhEDQ2y (ORCPT ); Tue, 4 May 2021 12:28:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231726AbhEDQ2x (ORCPT ); Tue, 4 May 2021 12:28:53 -0400 Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 782F6C06174A for ; Tue, 4 May 2021 09:27:58 -0700 (PDT) Received: by mail-qk1-x736.google.com with SMTP id o5so9163183qkb.0 for ; Tue, 04 May 2021 09:27:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=t23D3WB4vmdvDXKy81WrtRCExr2GHpVXwNDoKFSvJ7E=; b=gdPvbIUbK6sgOu+JAc0A1XIJwtKaebcIRF07+wMiPDpUZpov0ohm8b9r22NgqHm0go xrqWM4eee8XCXoRXxEzwlHKRlu80ocGLApxXG7KeuIADoSfbzVK9b+fArtSOqf78jWvz x1Q2mg8Z6EWoftJjl7Xf6qqHPCf8bqiyKfQfRDp6EH3YSTkCcy6MXKd+1sGmz9CJS1nI 2EIpIIPsTWmi5XlrAokwTD9fadVNBHlvs/BnXBQxtf8OKh1xDn0rieFSfJR5/OT48JVo 4+ebB/d5CwvmShB10wAYi0ecFiFK3Crk+3DQU0QF6XImtCiii6hMBCsKF2bI5Q7IXgcC JgdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=t23D3WB4vmdvDXKy81WrtRCExr2GHpVXwNDoKFSvJ7E=; b=AsPNeLIZsonTyT6/nYTuxfZb5oalYxEiBtS3U1N8tqIpvI9BHonEsJviO20ZGQyeHs mGEjyEOYTQqZ6/YOXxts/ZyMrEhJSYc4Ld8Cu/zk6vndcWagjyhSBGNwlJ00GRnlk7xq ztV+ZwiX6m9KbPr+L1zcLTTyoueVGsCaxSLZboe7SfZlL5xMgzigal2XMl+CWIUsp+Yx 4LGN3KPkDr67/Wp+B+yQ97IGMEYsHqRP08NyrIL7bAlrRmvhc85Nt6vpONYIlHj2BXLT gxtU9Jx465w5pLKY3pnWvlSV6j5f1fW/MBUePP7CGmsJ5YLIaLae6M3xJX51ac6Imhg+ 70Hg== X-Gm-Message-State: AOAM530z+1DYhi5po5lVg+NdYqbEYjf1/CEE0yE51Q/MEvwdc+y2KeQs ztcEGWEmwFtzimXEXCTnz7tQcyHPWkD4LA== X-Google-Smtp-Source: ABdhPJyyH6v07GPn14TsC9URTzl6oYTL3y/x/fnu9FJK1+tLeJIdZ7KZHbNE1gG84L0wEB5s3RGDRg== X-Received: by 2002:a37:e40f:: with SMTP id y15mr7832073qkf.314.1620145677358; Tue, 04 May 2021 09:27:57 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id x19sm11202689qkx.107.2021.05.04.09.27.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 May 2021 09:27:57 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com, tboegi@web.de Subject: [PATCH v3 5/8] parallel-checkout: add tests related to path collisions Date: Tue, 4 May 2021 13:27:32 -0300 Message-Id: X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add tests to confirm that path collisions are properly detected by checkout workers, both to avoid race conditions and to report colliding entries on clone. Co-authored-by: Jeff Hostetler Signed-off-by: Matheus Tavares --- parallel-checkout.c | 4 + t/lib-parallel-checkout.sh | 4 +- t/t2081-parallel-checkout-collisions.sh | 162 ++++++++++++++++++++++++ 3 files changed, 168 insertions(+), 2 deletions(-) create mode 100755 t/t2081-parallel-checkout-collisions.sh diff --git a/parallel-checkout.c b/parallel-checkout.c index 09e8b10a35..6fb3f1e6c9 100644 --- a/parallel-checkout.c +++ b/parallel-checkout.c @@ -8,6 +8,7 @@ #include "sigchain.h" #include "streaming.h" #include "thread-utils.h" +#include "trace2.h" struct pc_worker { struct child_process cp; @@ -326,6 +327,7 @@ void write_pc_item(struct parallel_checkout_item *pc_item, if (dir_sep && !has_dirs_only_path(path.buf, dir_sep - path.buf, state->base_dir_len)) { pc_item->status = PC_ITEM_COLLIDED; + trace2_data_string("pcheckout", NULL, "collision/dirname", path.buf); goto out; } @@ -341,6 +343,8 @@ void write_pc_item(struct parallel_checkout_item *pc_item, * call should have already caught these cases. */ pc_item->status = PC_ITEM_COLLIDED; + trace2_data_string("pcheckout", NULL, + "collision/basename", path.buf); } else { error_errno("failed to open file '%s'", path.buf); pc_item->status = PC_ITEM_FAILED; diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh index f60b22ef34..d6740425b1 100644 --- a/t/lib-parallel-checkout.sh +++ b/t/lib-parallel-checkout.sh @@ -22,12 +22,12 @@ test_checkout_workers () { local trace_file=trace-test-checkout-workers && rm -f "$trace_file" && - GIT_TRACE2="$(pwd)/$trace_file" "$@" && + GIT_TRACE2="$(pwd)/$trace_file" "$@" 2>&8 && local workers=$(grep "child_start\[..*\] git checkout--worker" "$trace_file" | wc -l) && test $workers -eq $expected_workers && rm "$trace_file" -} +} 8>&2 2>&4 # Verify that both the working tree and the index were created correctly verify_checkout () { diff --git a/t/t2081-parallel-checkout-collisions.sh b/t/t2081-parallel-checkout-collisions.sh new file mode 100755 index 0000000000..f6fcfc0c1e --- /dev/null +++ b/t/t2081-parallel-checkout-collisions.sh @@ -0,0 +1,162 @@ +#!/bin/sh + +test_description="path collisions during parallel checkout + +Parallel checkout must detect path collisions to: + +1) Avoid racily writing to different paths that represent the same file on disk. +2) Report the colliding entries on clone. + +The tests in this file exercise parallel checkout's collision detection code in +both these mechanics. +" + +. ./test-lib.sh +. "$TEST_DIRECTORY/lib-parallel-checkout.sh" + +TEST_ROOT="$PWD" + +test_expect_success CASE_INSENSITIVE_FS 'setup' ' + empty_oid=$(git hash-object -w --stdin objs <<-EOF && + 100644 $empty_oid FILE_X + 100644 $empty_oid FILE_x + 100644 $empty_oid file_X + 100644 $empty_oid file_x + EOF + git update-index --index-info >filter.log + EOF +' + +test_workers_in_event_trace () +{ + test $1 -eq $(grep ".event.:.child_start..*checkout--worker" $2 | wc -l) +} + +test_expect_success CASE_INSENSITIVE_FS 'worker detects basename collision' ' + GIT_TRACE2_EVENT="$(pwd)/trace" git \ + -c checkout.workers=2 -c checkout.thresholdForParallelism=0 \ + checkout . && + + test_workers_in_event_trace 2 trace && + collisions=$(grep -i "category.:.pcheckout.,.key.:.collision/basename.,.value.:.file_x.}" trace | wc -l) && + test $collisions -eq 3 +' + +test_expect_success CASE_INSENSITIVE_FS 'worker detects dirname collision' ' + test_config filter.logger.smudge "\"$TEST_ROOT/logger_script\" %f" && + empty_oid=$(git hash-object -w --stdin objs <<-EOF && + 100644 $empty_oid A/B + 100644 $empty_oid A/C + 100644 $empty_oid a + 100644 $attr_oid .gitattributes + EOF + git rm -rf . && + git update-index --index-info expected.log && + test_cmp filter.log expected.log && + + # Check that it used the right number of workers and detected the collisions + test_workers_in_event_trace 2 trace && + grep "category.:.pcheckout.,.key.:.collision/dirname.,.value.:.A/B.}" trace && + grep "category.:.pcheckout.,.key.:.collision/dirname.,.value.:.A/C.}" trace +' + +test_expect_success SYMLINKS,CASE_INSENSITIVE_FS 'do not follow symlinks colliding with leading dir' ' + empty_oid=$(git hash-object -w --stdin objs <<-EOF && + 120000 $symlink_oid D + 100644 $empty_oid d/x + 100644 $empty_oid e/y + EOF + git rm -rf . && + git update-index --index-info stderr && + + grep FILE_X stderr && + grep FILE_x stderr && + grep file_X stderr && + grep file_x stderr && + grep "the following paths have collided" stderr +' + +# This test ensures that the collision report code is correctly looking for +# colliding peers in the second half of the cache_entry array. This is done by +# defining a smudge command for the *last* array entry, which makes it +# non-eligible for parallel-checkout. Thus, it is checked out *first*, before +# spawning the workers. +# +# Note: this test doesn't work on Windows because, on this system, the +# collision report code uses strcmp() to find the colliding pairs when +# core.ignoreCase is false. And we need this setting for this test so that only +# 'file_x' matches the pattern of the filter attribute. But the test works on +# OSX, where the colliding pairs are found using inode. +# +test_expect_success CASE_INSENSITIVE_FS,!MINGW,!CYGWIN \ + 'collision report on clone (w/ colliding peer after the detected entry)' ' + + test_config_global filter.logger.smudge "\"$TEST_ROOT/logger_script\" %f" && + git reset --hard basename_collision && + echo "file_x filter=logger" >.gitattributes && + git add .gitattributes && + git commit -m "filter for file_x" && + + rm -rf clone-repo && + set_checkout_config 2 0 && + test_checkout_workers 2 \ + git -c core.ignoreCase=false clone . clone-repo 2>stderr && + + grep FILE_X stderr && + grep FILE_x stderr && + grep file_X stderr && + grep file_x stderr && + grep "the following paths have collided" stderr && + + # Check that only "file_x" was filtered + echo file_x >expected.log && + test_cmp clone-repo/filter.log expected.log +' + +test_done From patchwork Tue May 4 16:27:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12238283 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3B93C433B4 for ; Tue, 4 May 2021 16:28:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A86FA611AD for ; Tue, 4 May 2021 16:28:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231852AbhEDQ3B (ORCPT ); Tue, 4 May 2021 12:29:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231726AbhEDQ24 (ORCPT ); Tue, 4 May 2021 12:28:56 -0400 Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 05A1FC061761 for ; Tue, 4 May 2021 09:28:01 -0700 (PDT) Received: by mail-qk1-x731.google.com with SMTP id 8so9112987qkv.8 for ; Tue, 04 May 2021 09:28:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2EnM5QDr7FoNgrYUhohEVPq46wiDviob4V3QvwYbMTk=; b=zXOmvi6S/g9e1+uX+dyYmrrTqw6gUFQR/YktCbrg1FmTrCnOWCjPnbuUJc6PwLfyeU n9biQi2mLsWI8xXNvFPhaUcSEGFyB5+1VV4EYeIU1Cn6xJZ02f2OiCtlWFagBwKQccyR dmerQB+UPOg+U9so/MzCvVZEmx+yYMhIc4ezn73g2md3L9PhDp6YFwSdsgJSAbdjkxtQ LFf4PFYm77u6/ybUv8wzDoHMRgGMvjiAbwl5RrV8NNO+hPI7FLSwK2J+7GGbDeqiBluj hSWfx+ZeCJG80N8XsJZhQUZdcn18aRo1ynuBJu7FVYwTESjc2TMs/2vcY7CsFpz/cobP a+Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2EnM5QDr7FoNgrYUhohEVPq46wiDviob4V3QvwYbMTk=; b=P9mx7KR01/vRzp5vabrZASInAKor8sZ+iOUO1Kpos4JxefxMXMgVx9WRsrE0TdliiP JOO3/eh96ElKd0xFk6LfViwXzZE0HmGriHZ/2ub3niFwNOwW425ObuVCTORhL3XhTq6I zo661ArzRKrlRPqENcdv9kmZ6GF1AUlViHkv8WD1vWqABrxuayiDvknptR6CY6K9qaqH ITmKd/z5LTvPaGZlv2sP7qYZKePn2SKzxUtzpjvSg1q2/fvcYbjLdRsemHJIpBxQve8e OIpcR//CnROyusmG78eUN+RhKuW/2Kx51wkgHaovhfBi6iOtnnQ/OslCq6EFle423hie wjcw== X-Gm-Message-State: AOAM530MioS+Uj4t8aqw7coADEQlq4iCYu3nA2+Mlv0DOjbGAQspHbAK jlHWT09/yvSZ63GCl5S7vaSJEzElmBlK0A== X-Google-Smtp-Source: ABdhPJyiEX+QFXRBZEC6RC2kOhBQQLTjwbIzfttW1eSYrpiR1slw6PvQhDBJHL01DWLSna6hpqbmgQ== X-Received: by 2002:a37:7a05:: with SMTP id v5mr26028568qkc.266.1620145679984; Tue, 04 May 2021 09:27:59 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id x19sm11202689qkx.107.2021.05.04.09.27.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 May 2021 09:27:59 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com, tboegi@web.de Subject: [PATCH v3 6/8] t0028: extract encoding helpers to lib-encoding.sh Date: Tue, 4 May 2021 13:27:33 -0300 Message-Id: <6141c460517b3fd15e8052730dd180df7e445e12.1620145501.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The following patch will add tests outside t0028 which will also need to re-encode some strings. Extract the auxiliary encoding functions from t0028 to a common lib file so that they can be reused. Signed-off-by: Matheus Tavares --- t/lib-encoding.sh | 25 +++++++++++++++++++++++++ t/t0028-working-tree-encoding.sh | 25 +------------------------ 2 files changed, 26 insertions(+), 24 deletions(-) create mode 100644 t/lib-encoding.sh diff --git a/t/lib-encoding.sh b/t/lib-encoding.sh new file mode 100644 index 0000000000..2dabc8c73e --- /dev/null +++ b/t/lib-encoding.sh @@ -0,0 +1,25 @@ +# Encoding helpers + +test_lazy_prereq NO_UTF16_BOM ' + test $(printf abc | iconv -f UTF-8 -t UTF-16 | wc -c) = 6 +' + +test_lazy_prereq NO_UTF32_BOM ' + test $(printf abc | iconv -f UTF-8 -t UTF-32 | wc -c) = 12 +' + +write_utf16 () { + if test_have_prereq NO_UTF16_BOM + then + printf '\376\377' + fi && + iconv -f UTF-8 -t UTF-16 +} + +write_utf32 () { + if test_have_prereq NO_UTF32_BOM + then + printf '\0\0\376\377' + fi && + iconv -f UTF-8 -t UTF-32 +} diff --git a/t/t0028-working-tree-encoding.sh b/t/t0028-working-tree-encoding.sh index f970a9806b..82905a2156 100755 --- a/t/t0028-working-tree-encoding.sh +++ b/t/t0028-working-tree-encoding.sh @@ -6,33 +6,10 @@ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME . ./test-lib.sh +. "$TEST_DIRECTORY/lib-encoding.sh" GIT_TRACE_WORKING_TREE_ENCODING=1 && export GIT_TRACE_WORKING_TREE_ENCODING -test_lazy_prereq NO_UTF16_BOM ' - test $(printf abc | iconv -f UTF-8 -t UTF-16 | wc -c) = 6 -' - -test_lazy_prereq NO_UTF32_BOM ' - test $(printf abc | iconv -f UTF-8 -t UTF-32 | wc -c) = 12 -' - -write_utf16 () { - if test_have_prereq NO_UTF16_BOM - then - printf '\376\377' - fi && - iconv -f UTF-8 -t UTF-16 -} - -write_utf32 () { - if test_have_prereq NO_UTF32_BOM - then - printf '\0\0\376\377' - fi && - iconv -f UTF-8 -t UTF-32 -} - test_expect_success 'setup test files' ' git config core.eol lf && From patchwork Tue May 4 16:27:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12238285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B63DC433ED for ; Tue, 4 May 2021 16:28:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CA6F2611AD for ; Tue, 4 May 2021 16:28:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231844AbhEDQ3D (ORCPT ); Tue, 4 May 2021 12:29:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56640 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231845AbhEDQ27 (ORCPT ); Tue, 4 May 2021 12:28:59 -0400 Received: from mail-qv1-xf29.google.com (mail-qv1-xf29.google.com [IPv6:2607:f8b0:4864:20::f29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC2B6C061574 for ; Tue, 4 May 2021 09:28:03 -0700 (PDT) Received: by mail-qv1-xf29.google.com with SMTP id q5so4729405qvv.6 for ; Tue, 04 May 2021 09:28:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=24zXgh5ERhP6WetWcVX1R2rce49VOwH6kleFlHsnwFc=; b=KF7//bv8Ek6LYO1cFIzCSOXLjJfFSXESkpIYNgOSYqEgD8brlq1MNOPzbydegWIpQS NqeJZFgYOSVAZgUQii8LtEflWEs6YkVkQ8iGBls9f7+umAxAX9iLOIDkS/jfkp4MdvcB PgfeWp9mqh9Pxhoi9rI7wq3rH0gL8txnnUMCT30dxU8yOWGpKcYQ1edKOPllvtoCQ2yy RQQvzK3PBLHXNxrmwMSfPNqpe9tvHVemsGkxuK1sR0rKr5rDR3abSPnp9sgWmsIkF8Hp O/kITO05s+lmv2GsitEkcRZh9UOdkQT6ALaawF8sJexXlViIetRytRrHMl0qEo+cWb43 fYHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=24zXgh5ERhP6WetWcVX1R2rce49VOwH6kleFlHsnwFc=; b=WhXy0l0/0yyx+WF7IjoeojfuFm4wsfsSuRvzz+VN4uiyjSrQzFlTM70c/u0WODwudB I9OcX8oNpwOUJC635CeEgxi5SFagm2uGLZjKJh0M0MzIqEsWIZZafIiAAJcUwKIbl4YA hxgV+kVG0Em7gPSD6WRbpk9BnMw72/EvoQMo2KlOpNm7EUpWQVPX2TiihIhwhpbp9m/P 7Bc4u1Zr2vLfXeSyFD7FLzcXFBCjN9GTnLGfw0oRD/OK/9N1NByHIEaaR2oVM36bY6jR yQ77taKqH1Sk9yApB7wZLeJXmhoJ7uRz6QpbaNWZh2+96XhrAAVBJnscnJ0v8Yz3Mnz/ rOlw== X-Gm-Message-State: AOAM5325YnF7TIPMn49PwIw8+b86mehaavBVFapRSgbJm2FufP1bwroa L6LOu6rqrZLNgttcSwfM3ht7Vv/LsLnE6A== X-Google-Smtp-Source: ABdhPJx6IIUUnVaxTRr3sJHxqiIzC+uDjtahfc6+94fpU/UiBqUqN9f+7Tzlm/3tFLXwGz9+eaZ51g== X-Received: by 2002:a05:6214:19e7:: with SMTP id q7mr26439441qvc.34.1620145682716; Tue, 04 May 2021 09:28:02 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id x19sm11202689qkx.107.2021.05.04.09.28.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 May 2021 09:28:02 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com, tboegi@web.de Subject: [PATCH v3 7/8] parallel-checkout: add tests related to .gitattributes Date: Tue, 4 May 2021 13:27:34 -0300 Message-Id: <5350689a30566f067a10a288e7cdef63119728d1.1620145501.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add tests to confirm that the `struct conv_attrs` data is correctly passed from the main process to the workers, and that they can properly convert the blobs before writing them to the working tree. Also check that parallel-ineligible entries, such as regular files that require external filters, are correctly smudge and written when parallel-checkout is enabled. Co-authored-by: Jeff Hostetler Signed-off-by: Matheus Tavares --- t/t2082-parallel-checkout-attributes.sh | 194 ++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100755 t/t2082-parallel-checkout-attributes.sh diff --git a/t/t2082-parallel-checkout-attributes.sh b/t/t2082-parallel-checkout-attributes.sh new file mode 100755 index 0000000000..2525457961 --- /dev/null +++ b/t/t2082-parallel-checkout-attributes.sh @@ -0,0 +1,194 @@ +#!/bin/sh + +test_description='parallel-checkout: attributes + +Verify that parallel-checkout correctly creates files that require +conversions, as specified in .gitattributes. The main point here is +to check that the conv_attr data is correctly sent to the workers +and that it contains sufficient information to smudge files +properly (without access to the index or attribute stack). +' + +TEST_NO_CREATE_REPO=1 +. ./test-lib.sh +. "$TEST_DIRECTORY/lib-parallel-checkout.sh" +. "$TEST_DIRECTORY/lib-encoding.sh" + +test_expect_success 'parallel-checkout with ident' ' + set_checkout_config 2 0 && + git init ident && + ( + cd ident && + echo "A ident" >.gitattributes && + echo "\$Id\$" >A && + echo "\$Id\$" >B && + git add -A && + git commit -m id && + + rm A B && + test_checkout_workers 2 git reset --hard && + hexsz=$(test_oid hexsz) && + grep -E "\\\$Id: [0-9a-f]{$hexsz} \\\$" A && + grep "\\\$Id\\\$" B + ) +' + +test_expect_success 'parallel-checkout with re-encoding' ' + set_checkout_config 2 0 && + git init encoding && + ( + cd encoding && + echo text >utf8-text && + write_utf16 utf16-text && + + echo "A working-tree-encoding=UTF-16" >.gitattributes && + cp utf16-text A && + cp utf8-text B && + git add A B .gitattributes && + git commit -m encoding && + + # Check that A is stored in UTF-8 + git cat-file -p :A >A.internal && + test_cmp_bin utf8-text A.internal && + + rm A B && + test_checkout_workers 2 git checkout A B && + + # Check that A (and only A) is re-encoded during checkout + test_cmp_bin utf16-text A && + test_cmp_bin utf8-text B + ) +' + +test_expect_success 'parallel-checkout with eol conversions' ' + set_checkout_config 2 0 && + git init eol && + ( + cd eol && + printf "multi\r\nline\r\ntext" >crlf-text && + printf "multi\nline\ntext" >lf-text && + + git config core.autocrlf false && + echo "A eol=crlf" >.gitattributes && + cp crlf-text A && + cp lf-text B && + git add A B .gitattributes && + git commit -m eol && + + # Check that A is stored with LF format + git cat-file -p :A >A.internal && + test_cmp_bin lf-text A.internal && + + rm A B && + test_checkout_workers 2 git checkout A B && + + # Check that A (and only A) is converted to CRLF during checkout + test_cmp_bin crlf-text A && + test_cmp_bin lf-text B + ) +' + +# Entries that require an external filter are not eligible for parallel +# checkout. Check that both the parallel-eligible and non-eligible entries are +# properly writen in a single checkout operation. +# +test_expect_success 'parallel-checkout and external filter' ' + set_checkout_config 2 0 && + git init filter && + ( + cd filter && + write_script <<-\EOF rot13.sh && + tr \ + "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ" \ + "nopqrstuvwxyzabcdefghijklmNOPQRSTUVWXYZABCDEFGHIJKLM" + EOF + + git config filter.rot13.clean "\"$(pwd)/rot13.sh\"" && + git config filter.rot13.smudge "\"$(pwd)/rot13.sh\"" && + git config filter.rot13.required true && + + echo abcd >original && + echo nopq >rot13 && + + echo "A filter=rot13" >.gitattributes && + cp original A && + cp original B && + cp original C && + git add A B C .gitattributes && + git commit -m filter && + + # Check that A (and only A) was cleaned + git cat-file -p :A >A.internal && + test_cmp rot13 A.internal && + git cat-file -p :B >B.internal && + test_cmp original B.internal && + git cat-file -p :C >C.internal && + test_cmp original C.internal && + + rm A B C *.internal && + test_checkout_workers 2 git checkout A B C && + + # Check that A (and only A) was smudged during checkout + test_cmp original A && + test_cmp original B && + test_cmp original C + ) +' + +# The delayed queue is independent from the parallel queue, and they should be +# able to work together in the same checkout process. +# +test_expect_success PERL 'parallel-checkout and delayed checkout' ' + write_script rot13-filter.pl "$PERL_PATH" \ + <"$TEST_DIRECTORY"/t0021/rot13-filter.pl && + + test_config_global filter.delay.process \ + "\"$(pwd)/rot13-filter.pl\" --always-delay \"$(pwd)/delayed.log\" clean smudge delay" && + test_config_global filter.delay.required true && + + echo "abcd" >original && + echo "nopq" >rot13 && + + git init delayed && + ( + cd delayed && + echo "*.d filter=delay" >.gitattributes && + cp ../original W.d && + cp ../original X.d && + cp ../original Y && + cp ../original Z && + git add -A && + git commit -m delayed && + + # Check that *.d files were cleaned + git cat-file -p :W.d >W.d.internal && + test_cmp W.d.internal ../rot13 && + git cat-file -p :X.d >X.d.internal && + test_cmp X.d.internal ../rot13 && + git cat-file -p :Y >Y.internal && + test_cmp Y.internal ../original && + git cat-file -p :Z >Z.internal && + test_cmp Z.internal ../original && + + rm * + ) && + + set_checkout_config 2 0 && + test_checkout_workers 2 git -C delayed checkout -f && + verify_checkout delayed && + + # Check that the *.d files got to the delay queue and were filtered + grep "smudge W.d .* \[DELAYED\]" delayed.log && + grep "smudge X.d .* \[DELAYED\]" delayed.log && + test_cmp delayed/W.d original && + test_cmp delayed/X.d original && + + # Check that the parallel-eligible entries went to the right queue and + # were not filtered + ! grep "smudge Y .* \[DELAYED\]" delayed.log && + ! grep "smudge Z .* \[DELAYED\]" delayed.log && + test_cmp delayed/Y original && + test_cmp delayed/Z original +' + +test_done From patchwork Tue May 4 16:27:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matheus Tavares X-Patchwork-Id: 12238287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67B31C433B4 for ; Tue, 4 May 2021 16:28:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3AF1461153 for ; Tue, 4 May 2021 16:28:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231845AbhEDQ3G (ORCPT ); Tue, 4 May 2021 12:29:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56650 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231694AbhEDQ3C (ORCPT ); Tue, 4 May 2021 12:29:02 -0400 Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 890CEC06174A for ; Tue, 4 May 2021 09:28:06 -0700 (PDT) Received: by mail-qk1-x731.google.com with SMTP id o5so9163626qkb.0 for ; Tue, 04 May 2021 09:28:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=usp.br; s=usp-google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XKajkqyAJR2cWRG4UyaQ6mLUoNVl9zrHgnM7E69bptA=; b=w8Bn/in66rqKty1QjjD38yzA7BGUNLTA3ucduJz0LXhS8VHRvRIrb1+Hd6uO2Sz1FT qTpyYvE/zOHrRy1KbSke3AJEM/KA/yLpcv0Zkx66axgUX7rCLRcn4EzsQ2seIKfm6Cjo d7gbw+w4GsEu8iDT7Gu6mgNgXmQn6xezFsPaizCIzxjsx7eFsCW59o8T0iwvLZj3/72b QRPrgTYS8yhepiS3Sy4M2Klt+yqeEEqIn7XuKyaX9dqbPmse02bpEQoo0nguk3jC2SF0 ylCyfi5wvxzerPJ1BtuDp0K53H8aQWFjDvKQrvol2nyxYo5+uVuB8EJLMH57nsItPcI1 eLUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XKajkqyAJR2cWRG4UyaQ6mLUoNVl9zrHgnM7E69bptA=; b=Jx6eOb2GfV9LK4W5p7RHWarZoI8Xd9Y5GAstGMcA5X/dMkh8lk7Yrww19UFFKLVY4t Cu+P+OyxnGyu5l92un0AVKB7N9VoPL+3tWbH/cvSiuE8zutoWpUwi9brc62psRa4fAew ofxH0I/ME/1hW6n1RBfRCEDSiVnlK+mzXezUrUVeCpDpF7LqUv9T3GvS92X0yY+fMflx CIPpx2eq274unnSRqG66Q92lhy1mB7RKDR5ywnLiF+dJyCX+PIdBq4GOKeG96Qxk8XoN 29RNToImNGLD5SS8q1SmH885R9W8d+TFyFTLvxWZCIkYMqW9AjCY2B4bV8YrOoxh/b7z 8JzA== X-Gm-Message-State: AOAM532vMuOLbLPNvHcRVi3BJ4CuORal6xz82AV2xEMYmy21FJilqQ0B 7DkCUqGJb/2KPCyxL+s9eM+vLjPVVqTYJg== X-Google-Smtp-Source: ABdhPJxUidjSZ3cGTvXUttCxh4SND/tPVSETnm11HdsBxL0G64dKyAG5sjJUe+5zsfJDoX4jS3ppMQ== X-Received: by 2002:a37:7006:: with SMTP id l6mr25360586qkc.137.1620145685522; Tue, 04 May 2021 09:28:05 -0700 (PDT) Received: from mango.meuintelbras.local ([177.32.118.149]) by smtp.gmail.com with ESMTPSA id x19sm11202689qkx.107.2021.05.04.09.28.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 May 2021 09:28:05 -0700 (PDT) From: Matheus Tavares To: git@vger.kernel.org Cc: christian.couder@gmail.com, git@jeffhostetler.com, stolee@gmail.com, tboegi@web.de Subject: [PATCH v3 8/8] ci: run test round with parallel-checkout enabled Date: Tue, 4 May 2021 13:27:35 -0300 Message-Id: <7b2966c488c29578aa621c609e3d6fc7e36534a9.1620145501.git.matheus.bernardino@usp.br> X-Mailer: git-send-email 2.30.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org We already have tests for the basic parallel-checkout operations. But this code can also run be executed by other commands, such as git-read-tree and git-sparse-checkout, which are currently not tested with multiple workers. To promote a wider test coverage without duplicating tests: 1. Add the GIT_TEST_CHECKOUT_WORKERS environment variable, to optionally force parallel-checkout execution during the whole test suite. 2. Set this variable (with a value of 2) in the second test round of our linux-gcc CI job. This round runs `make test` again with some optional GIT_TEST_* variables enabled, so there is no additional overhead in exercising the parallel-checkout code here. Note that tests checking out less than two parallel-eligible entries will fall back to the sequential mode. Nevertheless, it's still a good exercise for the parallel-checkout framework as the fallback codepath also writes the queued entries using the parallel-checkout functions (only without spawning any worker). Signed-off-by: Matheus Tavares --- ci/run-build-and-tests.sh | 1 + parallel-checkout.c | 14 ++++++++++++++ t/README | 4 ++++ t/lib-parallel-checkout.sh | 3 +++ 4 files changed, 22 insertions(+) diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index d19be40544..3ce81ffee9 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -26,6 +26,7 @@ linux-gcc) export GIT_TEST_ADD_I_USE_BUILTIN=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master export GIT_TEST_WRITE_REV_INDEX=1 + export GIT_TEST_CHECKOUT_WORKERS=2 make test ;; linux-clang) diff --git a/parallel-checkout.c b/parallel-checkout.c index 6fb3f1e6c9..6b1af32bb3 100644 --- a/parallel-checkout.c +++ b/parallel-checkout.c @@ -35,6 +35,20 @@ static const int DEFAULT_NUM_WORKERS = 1; void get_parallel_checkout_configs(int *num_workers, int *threshold) { + char *env_workers = getenv("GIT_TEST_CHECKOUT_WORKERS"); + + if (env_workers && *env_workers) { + if (strtol_i(env_workers, 10, num_workers)) { + die("invalid value for GIT_TEST_CHECKOUT_WORKERS: '%s'", + env_workers); + } + if (*num_workers < 1) + *num_workers = online_cpus(); + + *threshold = 0; + return; + } + if (git_config_get_int("checkout.workers", num_workers)) *num_workers = DEFAULT_NUM_WORKERS; else if (*num_workers < 1) diff --git a/t/README b/t/README index 8eb9e46b1d..a8cfd37387 100644 --- a/t/README +++ b/t/README @@ -439,6 +439,10 @@ GIT_TEST_WRITE_REV_INDEX=, when true enables the GIT_TEST_SPARSE_INDEX=, when true enables index writes to use the sparse-index format by default. +GIT_TEST_CHECKOUT_WORKERS= overrides the 'checkout.workers' setting +to and 'checkout.thresholdForParallelism' to 0, forcing the +execution of the parallel-checkout code. + Naming Tests ------------ diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh index d6740425b1..21f5759732 100644 --- a/t/lib-parallel-checkout.sh +++ b/t/lib-parallel-checkout.sh @@ -1,5 +1,8 @@ # Helpers for tests invoking parallel-checkout +# Parallel checkout tests need full control of the number of workers +unset GIT_TEST_CHECKOUT_WORKERS + set_checkout_config () { if test $# -ne 2 then