From patchwork Sat May 21 15:08:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12857923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F1DBC433F5 for ; Sat, 21 May 2022 15:08:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353608AbiEUPIZ (ORCPT ); Sat, 21 May 2022 11:08:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244689AbiEUPIW (ORCPT ); Sat, 21 May 2022 11:08:22 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F043592D25 for ; Sat, 21 May 2022 08:08:20 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id h5so13863055wrb.11 for ; Sat, 21 May 2022 08:08:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jRvufZPqt+qoWVhZpC5LwCvQq2uy8PzZ64xc94DnHBo=; b=WgqPNH/l5oAj/ZJzzOoph8W3b/uQ96nFdVmhFkCKv0EewZThDsbI/iHw6GxttVCN5O Y7usf7jfrgogytgq5B1poMkg7WYT1RiaCduL0MNgRdPx+JGNXcRyh3SQdZ83bxflC12n +zBYKYETToVe2U1ldir6h/SVrhYwlADa30AXPTs5/V5IAEzZ58Iy2ZmuHh/f9xChGiJK icwvJGp9MdpXoyeQi3XlT7NSgeKahfpKxuftIrCLgNQZlU2JJ8xavNNTFSUbEA2mfwUy nFJFAFGMd8LCpUZMtTTsv5aSl7t+lO2+ODODCPlWCPOCJkQFGl75vgBPZ6Ki2xah3j5z Fwug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jRvufZPqt+qoWVhZpC5LwCvQq2uy8PzZ64xc94DnHBo=; b=r4yeXRWfIC98ns4HqkT7yr7ASxnOc+h+kA+vXPYSz3grcWqOiSH37JUvBPluMIBH/z 56w6tVsszOZRLNrsRb++SSgD9vjlCu0Bzj6NmDtVBCBpgwVieuGEIzTSO2mE2cUGydwH 5sqJEtQ+Yz5FuyRxTTaYGdR8gis4f/hzJzVVVb7Av8tdw5fVVpI8AWVgdcdhdKXjw7yp qPBFscZVc/LV2LPJxSrhDi9ch96gmnA4O0foqOanCOXXwu4KIvdEJMgAOcJjm/+LUfP8 ADCD32+iEN4I9msJ+F1vAxJK+E5EflVdJNBq5NVNcJwQZKyP8exZfQU+dvw3111CzyfL xSuQ== X-Gm-Message-State: AOAM533pLyOeFmxg2fpv52ZDLBrYwLFwQzh+Dl+/AeZ7iUHENBHYrXu6 DFF3I6Ofe1kOxRsu82PY31ZrMP2RRzw= X-Google-Smtp-Source: ABdhPJz0ur9UqZNWFD36ZuJdNTSpDLB2Vz2RCa1i2iPleNLgwJVi8edwTeYvQQzjhbDnFfpK0l1Zlw== X-Received: by 2002:a5d:42d0:0:b0:20d:2dd:22f7 with SMTP id t16-20020a5d42d0000000b0020d02dd22f7mr12435050wrr.136.1653145699091; Sat, 21 May 2022 08:08:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id ay13-20020a05600c1e0d00b003944821105esm4521741wmb.2.2022.05.21.08.08.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 08:08:18 -0700 (PDT) Message-Id: <0005cfae31d52a157d4df5ba3db9f9f5b2167ddc.1653145696.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 21 May 2022 15:08:10 +0000 Subject: [PATCH v6 1/7] archive: optionally add "virtual" files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin With the `--add-virtual-file=:` option, `git archive` now supports use cases where relatively trivial files need to be added that do not exist on disk. This will allow us to generate `.zip` files with generated content, without having to add said content to the object database and without having to write it out to disk. Signed-off-by: Johannes Schindelin Signed-off-by: René Scharfe Signed-off-by: René Scharfe Signed-off-by: René Scharfe --- Documentation/git-archive.txt | 11 ++++++++ archive.c | 53 +++++++++++++++++++++++++++++------ t/t5003-archive-zip.sh | 12 ++++++++ 3 files changed, 68 insertions(+), 8 deletions(-) diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt index bc4e76a7834..893cb1075bf 100644 --- a/Documentation/git-archive.txt +++ b/Documentation/git-archive.txt @@ -61,6 +61,17 @@ OPTIONS by concatenating the value for `--prefix` (if any) and the basename of . +--add-virtual-file=::: + Add the specified contents to the archive. Can be repeated to add + multiple files. The path of the file in the archive is built + by concatenating the value for `--prefix` (if any) and the + basename of . ++ +The `` cannot contain any colon, the file mode is limited to +a regular file, and the option may be subject to platform-dependent +command-line limits. For non-trivial cases, write an untracked file +and use `--add-file` instead. + --worktree-attributes:: Look for attributes in .gitattributes files in the working tree as well (see <>). diff --git a/archive.c b/archive.c index a3bbb091256..d20e16fa819 100644 --- a/archive.c +++ b/archive.c @@ -263,6 +263,7 @@ static int queue_or_write_archive_entry(const struct object_id *oid, struct extra_file_info { char *base; struct stat stat; + void *content; }; int write_archive_entries(struct archiver_args *args, @@ -337,7 +338,13 @@ int write_archive_entries(struct archiver_args *args, strbuf_addstr(&path_in_archive, basename(path)); strbuf_reset(&content); - if (strbuf_read_file(&content, path, info->stat.st_size) < 0) + if (info->content) + err = write_entry(args, &fake_oid, path_in_archive.buf, + path_in_archive.len, + info->stat.st_mode, + info->content, info->stat.st_size); + else if (strbuf_read_file(&content, path, + info->stat.st_size) < 0) err = error_errno(_("could not read '%s'"), path); else err = write_entry(args, &fake_oid, path_in_archive.buf, @@ -493,6 +500,7 @@ static void extra_file_info_clear(void *util, const char *str) { struct extra_file_info *info = util; free(info->base); + free(info->content); free(info); } @@ -514,14 +522,40 @@ static int add_file_cb(const struct option *opt, const char *arg, int unset) if (!arg) return -1; - path = prefix_filename(args->prefix, arg); - item = string_list_append_nodup(&args->extra_files, path); - item->util = info = xmalloc(sizeof(*info)); + info = xmalloc(sizeof(*info)); info->base = xstrdup_or_null(base); - if (stat(path, &info->stat)) - die(_("File not found: %s"), path); - if (!S_ISREG(info->stat.st_mode)) - die(_("Not a regular file: %s"), path); + + if (!strcmp(opt->long_name, "add-file")) { + path = prefix_filename(args->prefix, arg); + if (stat(path, &info->stat)) + die(_("File not found: %s"), path); + if (!S_ISREG(info->stat.st_mode)) + die(_("Not a regular file: %s"), path); + info->content = NULL; /* read the file later */ + } else if (!strcmp(opt->long_name, "add-virtual-file")) { + const char *colon = strchr(arg, ':'); + char *p; + + if (!colon) + die(_("missing colon: '%s'"), arg); + + p = xstrndup(arg, colon - arg); + if (!args->prefix) + path = p; + else { + path = prefix_filename(args->prefix, p); + free(p); + } + memset(&info->stat, 0, sizeof(info->stat)); + info->stat.st_mode = S_IFREG | 0644; + info->content = xstrdup(colon + 1); + info->stat.st_size = strlen(info->content); + } else { + BUG("add_file_cb() called for %s", opt->long_name); + } + item = string_list_append_nodup(&args->extra_files, path); + item->util = info; + return 0; } @@ -554,6 +588,9 @@ static int parse_archive_args(int argc, const char **argv, { OPTION_CALLBACK, 0, "add-file", args, N_("file"), N_("add untracked file to archive"), 0, add_file_cb, (intptr_t)&base }, + { OPTION_CALLBACK, 0, "add-virtual-file", args, + N_("path:content"), N_("add untracked file to archive"), 0, + add_file_cb, (intptr_t)&base }, OPT_STRING('o', "output", &output, N_("file"), N_("write the archive to this file")), OPT_BOOL(0, "worktree-attributes", &worktree_attributes, diff --git a/t/t5003-archive-zip.sh b/t/t5003-archive-zip.sh index 1e6d18b140e..ebc26e89a9b 100755 --- a/t/t5003-archive-zip.sh +++ b/t/t5003-archive-zip.sh @@ -206,6 +206,18 @@ test_expect_success 'git archive --format=zip --add-file' ' check_zip with_untracked check_added with_untracked untracked untracked +test_expect_success UNZIP 'git archive --format=zip --add-virtual-file' ' + git archive --format=zip >with_file_with_content.zip \ + --add-virtual-file=hello:world $EMPTY_TREE && + test_when_finished "rm -rf tmp-unpack" && + mkdir tmp-unpack && ( + cd tmp-unpack && + "$GIT_UNZIP" ../with_file_with_content.zip && + test_path_is_file hello && + test world = $(cat hello) + ) +' + test_expect_success 'git archive --format=zip --add-file twice' ' echo untracked >untracked && git archive --format=zip --prefix=one/ --add-file=untracked \ From patchwork Sat May 21 15:08:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12857925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6DDDC433F5 for ; Sat, 21 May 2022 15:08:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355330AbiEUPIc (ORCPT ); Sat, 21 May 2022 11:08:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245042AbiEUPIX (ORCPT ); Sat, 21 May 2022 11:08:23 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3877192D22 for ; Sat, 21 May 2022 08:08:22 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id bg25so5917412wmb.4 for ; Sat, 21 May 2022 08:08:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=eGahBRA+ydGjVWDYJywyOHiyVqpH6crsgCVHFETD+CQ=; b=GGfT2RHlOr0lhLBuJYdyEGUdLWMcVDD9jLgu701UeV9Mg1msP6MT5lLCfZ+Ic0B9QM m9wwztwMh+XMG3n44dbnHtwp5iax0wVrssRZ6lYSuuQLrQTuELLeNrYFoGwscHaLmUBe V1rQtl+rMVIcWBb31ZHzdHO6UpfrPw5V/Cci3Ns4+MDjiDoFGqpEO0q58okRRpmh8uc+ fo/eZG/5CQtMre6jg6q881OiFL9+JvIL83ciDVAUQPTspijA6I+e8BxHgV4IGGmz1XBO 9CMwk3qK2c6h0xbkFtquvMgPyq4GwqGJr1W10/3b3d4F0JmNv4qdubnkl67VJGbGLyjJ 2qvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=eGahBRA+ydGjVWDYJywyOHiyVqpH6crsgCVHFETD+CQ=; b=iLP1FQo1zKnLEWLVzAvp/YySnteHb+faFg7IxVyoQOT/9ODKuhp+HXAxo6xangT71X wfsi0RT8zeGS0Q5yY22QJRWaZ03Zoz/9fEla+i+Xk9XmU7VsLG5aY7X2Ti7bQEKhNJaJ PTt1e7KMJ/ZFmolRiuf92Ud5yNSRIa3ipGzUgBBERHmQMfwdKt/kqHHdE5WKBj2Ndq8Z 3UKJD3WIokRE8LPAiS4U6QjV7x9P2CzPTitr/LQfW5TiER2HzcLz6/54F0D5mId5p9I7 OoUzX4kOSt8M9MrGRjVefjoKLWKkmoajnUA/uA7gv0enppbx8XFcKa7NtesdQ4G/UlR/ AO6w== X-Gm-Message-State: AOAM5325yPZhQexZGVPOaFazAyjuv858R1GDkaYYzXQYYD197KNmuA1v SgsVyFEfOWNzzQ4DhO8O/tznALk2p6Q= X-Google-Smtp-Source: ABdhPJwbandQ1wbszz9KZfJvNuSDvlmu/lO87lcwB9UwxTxkjbnfLmBIW9/8Gc1vvvJcziL0Wnz9Ug== X-Received: by 2002:a05:600c:2845:b0:397:32a8:4d54 with SMTP id r5-20020a05600c284500b0039732a84d54mr10911595wmb.37.1653145700405; Sat, 21 May 2022 08:08:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k6-20020adfb346000000b0020c5253d8fcsm6109998wrd.72.2022.05.21.08.08.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 08:08:19 -0700 (PDT) Message-Id: <7eebcf27b45eb13541d4abae70a374a0e35ab6b8.1653145696.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 21 May 2022 15:08:11 +0000 Subject: [PATCH v6 2/7] archive --add-virtual-file: allow paths containing colons Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin By allowing the path to be enclosed in double-quotes, we can avoid the limitation that paths cannot contain colons. Signed-off-by: Johannes Schindelin --- Documentation/git-archive.txt | 14 ++++++++++---- archive.c | 30 ++++++++++++++++++++---------- t/t5003-archive-zip.sh | 8 ++++++++ 3 files changed, 38 insertions(+), 14 deletions(-) diff --git a/Documentation/git-archive.txt b/Documentation/git-archive.txt index 893cb1075bf..54de945a84e 100644 --- a/Documentation/git-archive.txt +++ b/Documentation/git-archive.txt @@ -67,10 +67,16 @@ OPTIONS by concatenating the value for `--prefix` (if any) and the basename of . + -The `` cannot contain any colon, the file mode is limited to -a regular file, and the option may be subject to platform-dependent -command-line limits. For non-trivial cases, write an untracked file -and use `--add-file` instead. +The `` argument can start and end with a literal double-quote +character; The contained file name is interpreted as a C-style string, +i.e. the backslash is interpreted as escape character. The path must +be quoted if it contains a colon, to avoid the colon from being +misinterpreted as the separator between the path and the contents, or +if the path begins or ends with a double-quote character. ++ +The file mode is limited to a regular file, and the option may be +subject to platform-dependent command-line limits. For non-trivial +cases, write an untracked file and use `--add-file` instead. --worktree-attributes:: Look for attributes in .gitattributes files in the working tree diff --git a/archive.c b/archive.c index d20e16fa819..b7756b91200 100644 --- a/archive.c +++ b/archive.c @@ -9,6 +9,7 @@ #include "parse-options.h" #include "unpack-trees.h" #include "dir.h" +#include "quote.h" static char const * const archive_usage[] = { N_("git archive [] [...]"), @@ -533,22 +534,31 @@ static int add_file_cb(const struct option *opt, const char *arg, int unset) die(_("Not a regular file: %s"), path); info->content = NULL; /* read the file later */ } else if (!strcmp(opt->long_name, "add-virtual-file")) { - const char *colon = strchr(arg, ':'); - char *p; + struct strbuf buf = STRBUF_INIT; + const char *p = arg; + + if (*p != '"') + p = strchr(p, ':'); + else if (unquote_c_style(&buf, p, &p) < 0) + die(_("unclosed quote: '%s'"), arg); - if (!colon) + if (!p || *p != ':') die(_("missing colon: '%s'"), arg); - p = xstrndup(arg, colon - arg); - if (!args->prefix) - path = p; - else { - path = prefix_filename(args->prefix, p); - free(p); + if (p == arg) + die(_("empty file name: '%s'"), arg); + + path = buf.len ? + strbuf_detach(&buf, NULL) : xstrndup(arg, p - arg); + + if (args->prefix) { + char *save = path; + path = prefix_filename(args->prefix, path); + free(save); } memset(&info->stat, 0, sizeof(info->stat)); info->stat.st_mode = S_IFREG | 0644; - info->content = xstrdup(colon + 1); + info->content = xstrdup(p + 1); info->stat.st_size = strlen(info->content); } else { BUG("add_file_cb() called for %s", opt->long_name); diff --git a/t/t5003-archive-zip.sh b/t/t5003-archive-zip.sh index ebc26e89a9b..3a5a052e8ce 100755 --- a/t/t5003-archive-zip.sh +++ b/t/t5003-archive-zip.sh @@ -207,13 +207,21 @@ check_zip with_untracked check_added with_untracked untracked untracked test_expect_success UNZIP 'git archive --format=zip --add-virtual-file' ' + if test_have_prereq FUNNYNAMES + then + PATHNAME=quoted:colon + else + PATHNAME=quoted + fi && git archive --format=zip >with_file_with_content.zip \ + --add-virtual-file=\"$PATHNAME\": \ --add-virtual-file=hello:world $EMPTY_TREE && test_when_finished "rm -rf tmp-unpack" && mkdir tmp-unpack && ( cd tmp-unpack && "$GIT_UNZIP" ../with_file_with_content.zip && test_path_is_file hello && + test_path_is_file $PATHNAME && test world = $(cat hello) ) ' From patchwork Sat May 21 15:08:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12857924 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22EF7C433F5 for ; Sat, 21 May 2022 15:08:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355322AbiEUPI2 (ORCPT ); Sat, 21 May 2022 11:08:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348064AbiEUPIY (ORCPT ); Sat, 21 May 2022 11:08:24 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B43092D25 for ; Sat, 21 May 2022 08:08:23 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id u27so14170731wru.8 for ; Sat, 21 May 2022 08:08:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Uo6VeR463A8bpTnDe4txbUadyp7c2hS7RwvOwDAMeOI=; b=PZNEORk0NSxTKVh83V/JOh2DRfSnZb0BK2K8L1+37Iz8pfMkzJq79phGCxi73m+UnV 6bulCvE0bOpzuKJ3pPCpMM/xRaavllAbnoM4Ux7vE6dLvmiwsNbAKPl/zJGo4ZHx9Ik4 jOFc+/1pC82zZ7wnc/OSZ8hHJUxR6jm1iN3PrWmvbr7b/PVYion2/V/CwjoM++4EHQla XiCp/Jne4wiRWjEeDQp9UpyE2/0NByLwYwHsmkVe/0dRqTmVjs4k83toca6e88NommLu AbMuhOVr61VEgYinfmcqIJwTDtFMcvbcw/BWVIb32ZynSK83DCf+hjUuj4PFIX+pSVI5 NqmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Uo6VeR463A8bpTnDe4txbUadyp7c2hS7RwvOwDAMeOI=; b=PZLoU/2AzFLr3mSr1SKpnHw2WOszLZcgD0GOQNfbDkq9M9KiFIiHOn5lZaQ+s+dJi4 6xIVqIQehjlYR7DpyA1bCvuYVaiZ2Be8YXbuHyN0T9ogcaVmTrb37LFkPpzgA8h07tDf gyt5CfcjsGJ6KyV/LziLX9fteIZdCldpM3BrKjcPy6R4rlkeVxYpGX81AIUjAhP29BjK FaYbxiUerYOPPR9oQfAKaNlQ2l4I6GTelGdFfMIgk9iixTb/zrLeaXZZzeJJKw+lCeVf zmKesE7wuQcnEYVpx8+w2V4TZKWSrq1tz+p30YZzXkm1XKGtaVQ/aUkWpVhake/FrwQX RQGg== X-Gm-Message-State: AOAM531cvVHmTYn5mrGuP96USDnRg7kA25MtlPJzzgB7U0ELWdJnPdGK hbXr7uQEFJWXdZQrtlLiJSxmV/BwMEc= X-Google-Smtp-Source: ABdhPJxPCQZHjDP1YkqihbJLlTes+gPObC6lCBzjmla+mIgfGeiJYjYw3CPYXrmCAdRZ/vuSjPCiKA== X-Received: by 2002:adf:dd09:0:b0:20e:6231:51e7 with SMTP id a9-20020adfdd09000000b0020e623151e7mr12579574wrm.251.1653145701496; Sat, 21 May 2022 08:08:21 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id ay13-20020a05600c1e0d00b003944821105esm4521800wmb.2.2022.05.21.08.08.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 08:08:21 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 21 May 2022 15:08:12 +0000 Subject: [PATCH v6 3/7] scalar: validate the optional enlistment argument Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin The `scalar` command needs a Scalar enlistment for many subcommands, and looks in the current directory for such an enlistment (traversing the parent directories until it finds one). These is subcommands can also be called with an optional argument specifying the enlistment. Here, too, we traverse parent directories as needed, until we find an enlistment. However, if the specified directory does not even exist, or is not a directory, we should stop right there, with an error message. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 6 ++++-- contrib/scalar/t/t9099-scalar.sh | 5 +++++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 1ce9c2b00e8..00dcd4b50ef 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -43,9 +43,11 @@ static void setup_enlistment_directory(int argc, const char **argv, usage_with_options(usagestr, options); /* find the worktree, determine its corresponding root */ - if (argc == 1) + if (argc == 1) { strbuf_add_absolute_path(&path, argv[0]); - else if (strbuf_getcwd(&path) < 0) + if (!is_directory(path.buf)) + die(_("'%s' does not exist"), path.buf); + } else if (strbuf_getcwd(&path) < 0) die(_("need a working directory")); strbuf_trim_trailing_dir_sep(&path); diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 2e1502ad45e..9d83fdf25e8 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -85,4 +85,9 @@ test_expect_success 'scalar delete with enlistment' ' test_path_is_missing cloned ' +test_expect_success '`scalar [...] ` errors out when dir is missing' ' + ! scalar run config cloned 2>err && + grep "cloned. does not exist" err +' + test_done From patchwork Sat May 21 15:08:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12857928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79A47C433F5 for ; Sat, 21 May 2022 15:08:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355359AbiEUPIn (ORCPT ); Sat, 21 May 2022 11:08:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354713AbiEUPI0 (ORCPT ); Sat, 21 May 2022 11:08:26 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BC8993447 for ; Sat, 21 May 2022 08:08:24 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id j25so14970451wrc.9 for ; Sat, 21 May 2022 08:08:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=KOhCLOG4xQM7GXYpo7HgIfpmlRXN1CU5iK5b1yz4gpM=; b=Dzk46A83RKFroY6wfPCYrtvlQzBSbZkajbctJSxB3+8m7Ribdw6IHEHVLEYHizNZNd OwEc/6CIWhSLcy3rZxOc6awzp09Ym0foZuDdeVeS9m6Txckh0OdQ4WRtFONPOF7/3gJV vxFb3yQE/3W1OR5QmNdyKBSGw9n6S0p/OATqSH5V54dkAQjGi+pJyPRGKDo71Dn0PwAT hf7Lhs+98z64Gp1ducIUk6wUCE5tdjRhHIyGOzczd7aYRNHQtfFeplotzZaaUcGZ3/pC nD1UFBUEnXTQFvJp7SBRL/TmASgFfYXthBSZclnEEGjyYvMLzLcu/+EVR8nYGP6u36hF zv+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=KOhCLOG4xQM7GXYpo7HgIfpmlRXN1CU5iK5b1yz4gpM=; b=a1/mEntAoYihlH4wKPFL8BC1RQalSH7r8mcWuxVlw2G1p/Jw3ZFJ5xJbuO203zzbZP vEzrbxK8AqLZM5ycO343KcG70SHjCXuElZF+kCRIPv5NbXxA1SqIZzKf+pu1jC0KbUKK B3Y8jBO0AHL+2tL/CCk5GJyQ9p3lv2nEh7X7HoslPyScDRfkXzK3PXh/OsiJcaB3HqJn J5Yj31fQcs+1lJYCuZBJKGTaME7O81n5dlrOF2W2s83SSGsrxjEkY53z03jmEWIDwlQ1 Ky+7aZ5Gr63K5LxhyXOyHuoOd3GoOMZcQVAevNGIO6G6tLDbXqXY6qbt+hXeZqoHEpUy B2aA== X-Gm-Message-State: AOAM530DjOCsZmALwvXumZ728n/iT30VPn+oBYy1Boei1w0RduEsUoxs 0IXu3d3wgqYai9yF3Kuie9sKtkkG3Nk= X-Google-Smtp-Source: ABdhPJwJAXzBL+NawjNWWrC4yVEf7O7kreDcwx83brtFxuzdNTMY09dk3YvNHLMqXNytmHzn/VQ+7A== X-Received: by 2002:a05:6000:1867:b0:20f:ca21:7813 with SMTP id d7-20020a056000186700b0020fca217813mr2449829wri.100.1653145702606; Sat, 21 May 2022 08:08:22 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x2-20020adfbb42000000b0020d0e7379casm5388758wrg.95.2022.05.21.08.08.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 08:08:22 -0700 (PDT) Message-Id: <89c13a45e00deb1df45113a9249d3a125c7b4fee.1653145696.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 21 May 2022 15:08:13 +0000 Subject: [PATCH v6 4/7] Implement `scalar diagnose` Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin Over the course of Scalar's development, it became obvious that there is a need for a command that can gather all kinds of useful information that can help identify the most typical problems with large worktrees/repositories. The `diagnose` command is the culmination of this hard-won knowledge: it gathers the installed hooks, the config, a couple statistics describing the data shape, among other pieces of information, and then wraps everything up in a tidy, neat `.zip` archive. Note: originally, Scalar was implemented in C# using the .NET API, where we had the luxury of a comprehensive standard library that includes basic functionality such as writing a `.zip` file. In the C version, we lack such a commodity. Rather than introducing a dependency on, say, libzip, we slightly abuse Git's `archive` machinery: we write out a `.zip` of the empty try, augmented by a couple files that are added via the `--add-file*` options. We are careful trying not to modify the current repository in any way lest the very circumstances that required `scalar diagnose` to be run are changed by the `diagnose` run itself. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 144 +++++++++++++++++++++++++++++++ contrib/scalar/scalar.txt | 12 +++ contrib/scalar/t/t9099-scalar.sh | 14 +++ 3 files changed, 170 insertions(+) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 00dcd4b50ef..53213f9a3b9 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -11,6 +11,7 @@ #include "dir.h" #include "packfile.h" #include "help.h" +#include "archive.h" /* * Remove the deepest subdirectory in the provided path string. Path must not @@ -261,6 +262,47 @@ static int unregister_dir(void) return res; } +static int add_directory_to_archiver(struct strvec *archiver_args, + const char *path, int recurse) +{ + int at_root = !*path; + DIR *dir = opendir(at_root ? "." : path); + struct dirent *e; + struct strbuf buf = STRBUF_INIT; + size_t len; + int res = 0; + + if (!dir) + return error_errno(_("could not open directory '%s'"), path); + + if (!at_root) + strbuf_addf(&buf, "%s/", path); + len = buf.len; + strvec_pushf(archiver_args, "--prefix=%s", buf.buf); + + while (!res && (e = readdir(dir))) { + if (!strcmp(".", e->d_name) || !strcmp("..", e->d_name)) + continue; + + strbuf_setlen(&buf, len); + strbuf_addstr(&buf, e->d_name); + + if (e->d_type == DT_REG) + strvec_pushf(archiver_args, "--add-file=%s", buf.buf); + else if (e->d_type != DT_DIR) + warning(_("skipping '%s', which is neither file nor " + "directory"), buf.buf); + else if (recurse && + add_directory_to_archiver(archiver_args, + buf.buf, recurse) < 0) + res = -1; + } + + closedir(dir); + strbuf_release(&buf); + return res; +} + /* printf-style interface, expects `=` argument */ static int set_config(const char *fmt, ...) { @@ -501,6 +543,107 @@ cleanup: return res; } +static int cmd_diagnose(int argc, const char **argv) +{ + struct option options[] = { + OPT_END(), + }; + const char * const usage[] = { + N_("scalar diagnose []"), + NULL + }; + struct strbuf zip_path = STRBUF_INIT; + struct strvec archiver_args = STRVEC_INIT; + char **argv_copy = NULL; + int stdout_fd = -1, archiver_fd = -1; + time_t now = time(NULL); + struct tm tm; + struct strbuf path = STRBUF_INIT, buf = STRBUF_INIT; + int res = 0; + + argc = parse_options(argc, argv, NULL, options, + usage, 0); + + setup_enlistment_directory(argc, argv, usage, options, &zip_path); + + strbuf_addstr(&zip_path, "/.scalarDiagnostics/scalar_"); + strbuf_addftime(&zip_path, + "%Y%m%d_%H%M%S", localtime_r(&now, &tm), 0, 0); + strbuf_addstr(&zip_path, ".zip"); + switch (safe_create_leading_directories(zip_path.buf)) { + case SCLD_EXISTS: + case SCLD_OK: + break; + default: + error_errno(_("could not create directory for '%s'"), + zip_path.buf); + goto diagnose_cleanup; + } + stdout_fd = dup(1); + if (stdout_fd < 0) { + res = error_errno(_("could not duplicate stdout")); + goto diagnose_cleanup; + } + + archiver_fd = xopen(zip_path.buf, O_CREAT | O_WRONLY | O_TRUNC, 0666); + if (archiver_fd < 0 || dup2(archiver_fd, 1) < 0) { + res = error_errno(_("could not redirect output")); + goto diagnose_cleanup; + } + + init_zip_archiver(); + strvec_pushl(&archiver_args, "scalar-diagnose", "--format=zip", NULL); + + strbuf_reset(&buf); + strbuf_addstr(&buf, "Collecting diagnostic info\n\n"); + get_version_info(&buf, 1); + + strbuf_addf(&buf, "Enlistment root: %s\n", the_repository->worktree); + write_or_die(stdout_fd, buf.buf, buf.len); + strvec_pushf(&archiver_args, + "--add-virtual-file=diagnostics.log:%.*s", + (int)buf.len, buf.buf); + + if ((res = add_directory_to_archiver(&archiver_args, ".git", 0)) || + (res = add_directory_to_archiver(&archiver_args, ".git/hooks", 0)) || + (res = add_directory_to_archiver(&archiver_args, ".git/info", 0)) || + (res = add_directory_to_archiver(&archiver_args, ".git/logs", 1)) || + (res = add_directory_to_archiver(&archiver_args, ".git/objects/info", 0))) + goto diagnose_cleanup; + + strvec_pushl(&archiver_args, "--prefix=", + oid_to_hex(the_hash_algo->empty_tree), "--", NULL); + + /* `write_archive()` modifies the `argv` passed to it. Let it. */ + argv_copy = xmemdupz(archiver_args.v, + sizeof(char *) * archiver_args.nr); + res = write_archive(archiver_args.nr, (const char **)argv_copy, NULL, + the_repository, NULL, 0); + if (res) { + error(_("failed to write archive")); + goto diagnose_cleanup; + } + + if (!res) + fprintf(stderr, "\n" + "Diagnostics complete.\n" + "All of the gathered info is captured in '%s'\n", + zip_path.buf); + +diagnose_cleanup: + if (archiver_fd >= 0) { + close(1); + dup2(stdout_fd, 1); + } + free(argv_copy); + strvec_clear(&archiver_args); + strbuf_release(&zip_path); + strbuf_release(&path); + strbuf_release(&buf); + + return res; +} + static int cmd_list(int argc, const char **argv) { if (argc != 1) @@ -802,6 +945,7 @@ static struct { { "reconfigure", cmd_reconfigure }, { "delete", cmd_delete }, { "version", cmd_version }, + { "diagnose", cmd_diagnose }, { NULL, NULL}, }; diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt index f416d637289..22583fe046e 100644 --- a/contrib/scalar/scalar.txt +++ b/contrib/scalar/scalar.txt @@ -14,6 +14,7 @@ scalar register [] scalar unregister [] scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [] scalar reconfigure [ --all | ] +scalar diagnose [] scalar delete DESCRIPTION @@ -129,6 +130,17 @@ reconfigure the enlistment. With the `--all` option, all enlistments currently registered with Scalar will be reconfigured. Use this option after each Scalar upgrade. +Diagnose +~~~~~~~~ + +diagnose []:: + When reporting issues with Scalar, it is often helpful to provide the + information gathered by this command, including logs and certain + statistics describing the data shape of the current enlistment. ++ +The output of this command is a `.zip` file that is written into +a directory adjacent to the worktree in the `src` directory. + Delete ~~~~~~ diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 9d83fdf25e8..6802d317258 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -90,4 +90,18 @@ test_expect_success '`scalar [...] ` errors out when dir is missing' ' grep "cloned. does not exist" err ' +SQ="'" +test_expect_success UNZIP 'scalar diagnose' ' + scalar clone "file://$(pwd)" cloned --single-branch && + scalar diagnose cloned >out 2>err && + sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && + zip_path=$(cat zip_path) && + test -n "$zip_path" && + unzip -v "$zip_path" && + folder=${zip_path%.zip} && + test_path_is_missing "$folder" && + unzip -p "$zip_path" diagnostics.log >out && + test_file_not_empty out +' + test_done From patchwork Sat May 21 15:08:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12857927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D8F5C4332F for ; Sat, 21 May 2022 15:08:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355351AbiEUPIr (ORCPT ); Sat, 21 May 2022 11:08:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1355218AbiEUPI0 (ORCPT ); Sat, 21 May 2022 11:08:26 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8282C9344F for ; Sat, 21 May 2022 08:08:25 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id p5-20020a1c2905000000b003970dd5404dso5773839wmp.0 for ; Sat, 21 May 2022 08:08:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sXZbh0MamYACiDTr6HvklkpnU4A4eZMx4voLUuNMvpg=; b=MS4K2LZLqvouQyHsVFZSkWyy9ektC9+61zMzubOFC3qdEf9JJsrKplmvz44WgK3Cc6 VGTG/LYMckZmibCyUWkM8vZX4IOoKIa+BgZwdqvm/qDkh1biG0csXgM8wY4NowTygqq6 GcfEg02u/+hA0FfLka6AV7wZJ0u9FNW2oPChGeAbyYLlasypIb7csawuODSu9KkwQgae fTsOzQt664X2bAbX3IqRFazf01q2AClefGLqOmm6LUaNBZ0/OUVdnLuE2o9QBTO5/0cc tgayxGoQs9/i6BErZQopjGf0EDxUz4d/un/p1niz6IEk5iuAcNBBotYMPdg9VvukV2L8 b8GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sXZbh0MamYACiDTr6HvklkpnU4A4eZMx4voLUuNMvpg=; b=yU37p3m0fYX7UIBE6YvIkY36WrYsygfNkjpllm9y5UQYxsFrRGYO93RBHw+xxdIVz/ ZPeHRb/yieiOFrG/RdOR4JHaHH0Jq/eC8Ve7TX/ut95NnPLpdJcuNoI6CBhpIW24xXYT FlpRM9ERht73dUsOxy9ibLY+OTP2Mrhnu9GP/qCbgaAtTRijnj0Lf//L82cVN+lCrVu1 wiGkj+OhiOBcIKLhXuj0x9IyTq+GI82mgsGPRn0224numaRWeB0z50bhSCnLKuf2Mx0J jxH9LJtnhjZJdJ3gUbCWFg3i9yJAU+cpWELlkfePuEKGqTOM6gOj36RWqj7nUmg2QZXg RNWg== X-Gm-Message-State: AOAM532+k3lPLD+GkSPVZUMGxSR/16M/+FDknT3Txo+dHzx1+bLJhXsg TdvCYXGRWe2+9qOpP3r6Y7I2AbHrass= X-Google-Smtp-Source: ABdhPJxnbTW3Dl04Cby90tknfaNvAopr4Knd49XDyuY9I+bmpsWz0sOoz4kqrFAhRB3yiruXvsXguQ== X-Received: by 2002:a7b:c24c:0:b0:397:3751:2d2a with SMTP id b12-20020a7bc24c000000b0039737512d2amr8201226wmj.51.1653145703803; Sat, 21 May 2022 08:08:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p2-20020adfa202000000b0020adc114136sm6411977wra.0.2022.05.21.08.08.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 08:08:23 -0700 (PDT) Message-Id: <8ffbaad30869ae03e8ba0b1eae4c23aa7d83759e.1653145696.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 21 May 2022 15:08:14 +0000 Subject: [PATCH v6 5/7] scalar diagnose: include disk space information Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin When analyzing problems with large worktrees/repositories, it is useful to know how close to a "full disk" situation Scalar/Git operates. Let's include this information. Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 53 ++++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 1 + 2 files changed, 54 insertions(+) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 53213f9a3b9..0a9e25a57f8 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -303,6 +303,58 @@ static int add_directory_to_archiver(struct strvec *archiver_args, return res; } +#ifndef WIN32 +#include +#endif + +static int get_disk_info(struct strbuf *out) +{ +#ifdef WIN32 + struct strbuf buf = STRBUF_INIT; + char volume_name[MAX_PATH], fs_name[MAX_PATH]; + DWORD serial_number, component_length, flags; + ULARGE_INTEGER avail2caller, total, avail; + + strbuf_realpath(&buf, ".", 1); + if (!GetDiskFreeSpaceExA(buf.buf, &avail2caller, &total, &avail)) { + error(_("could not determine free disk size for '%s'"), + buf.buf); + strbuf_release(&buf); + return -1; + } + + strbuf_setlen(&buf, offset_1st_component(buf.buf)); + if (!GetVolumeInformationA(buf.buf, volume_name, sizeof(volume_name), + &serial_number, &component_length, &flags, + fs_name, sizeof(fs_name))) { + error(_("could not get info for '%s'"), buf.buf); + strbuf_release(&buf); + return -1; + } + strbuf_addf(out, "Available space on '%s': ", buf.buf); + strbuf_humanise_bytes(out, avail2caller.QuadPart); + strbuf_addch(out, '\n'); + strbuf_release(&buf); +#else + struct strbuf buf = STRBUF_INIT; + struct statvfs stat; + + strbuf_realpath(&buf, ".", 1); + if (statvfs(buf.buf, &stat) < 0) { + error_errno(_("could not determine free disk size for '%s'"), + buf.buf); + strbuf_release(&buf); + return -1; + } + + strbuf_addf(out, "Available space on '%s': ", buf.buf); + strbuf_humanise_bytes(out, st_mult(stat.f_bsize, stat.f_bavail)); + strbuf_addf(out, " (mount flags 0x%lx)\n", stat.f_flag); + strbuf_release(&buf); +#endif + return 0; +} + /* printf-style interface, expects `=` argument */ static int set_config(const char *fmt, ...) { @@ -599,6 +651,7 @@ static int cmd_diagnose(int argc, const char **argv) get_version_info(&buf, 1); strbuf_addf(&buf, "Enlistment root: %s\n", the_repository->worktree); + get_disk_info(&buf); write_or_die(stdout_fd, buf.buf, buf.len); strvec_pushf(&archiver_args, "--add-virtual-file=diagnostics.log:%.*s", diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 6802d317258..934b2485d91 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -94,6 +94,7 @@ SQ="'" test_expect_success UNZIP 'scalar diagnose' ' scalar clone "file://$(pwd)" cloned --single-branch && scalar diagnose cloned >out 2>err && + grep "Available space" out && sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && zip_path=$(cat zip_path) && test -n "$zip_path" && From patchwork Sat May 21 15:08:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 12857926 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA903C433F5 for ; Sat, 21 May 2022 15:08:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355346AbiEUPIm (ORCPT ); Sat, 21 May 2022 11:08:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58442 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344621AbiEUPIj (ORCPT ); Sat, 21 May 2022 11:08:39 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C404393469 for ; Sat, 21 May 2022 08:08:26 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id ay35so4330080wmb.5 for ; Sat, 21 May 2022 08:08:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=q4Nq0XKv3sAz87ZRO+MCbKnQgUW64Vx3dbiERrPUd8U=; b=X9TJ3y2wzDJvVarqCCWPhTqiCfS+YvaQLC5iQd4jrPOlKvvHV+tCxbY9a2Ou6SnIkP OWccXfYBfjkYb4wyo9NQSgR4HqLNgQ8wbiUiE+2Q06MQcnwBINkaNHRV/cqkbvEb2jZA TvJR7/K1NKVoiH0tyzWxP1dHS4mFRke1LLuRu+v8OcpRA/tiIsdlfJkkJw9J1RWDSEy5 0EQCByqvoyEppuNqJ2cDi0c0ue6OV03lSykgd8kBhK56kSsi79ahw3IGP+y59PZkaXMf BjNr5kq+A/qXW0W4yVCRE4M3AKX7dA2HXvC2MQQcuj6UyTH3u/Xge3mMo7/LRsjulb6h DYnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=q4Nq0XKv3sAz87ZRO+MCbKnQgUW64Vx3dbiERrPUd8U=; b=u0s5JaZ7nrbJO4F3Ge97oczwiBdP7X/g7arYrPZRfUKbmHa+tPP1TBHAKwI+1Ye5cN J9/27sR2AHc5IderJ9QvwuVNZKLbB4SbydsMYcHHJ/DDXU5/RfAJCBw4EzbA/pYJL145 m4oc27yw9RypVteMLGSEniUoNGuuyKI7YtKFqibfeA2E2f25wekPxDYA2PATRjEk6r68 06gJXUt+O2xCfNsgVvb+4U55m66HMioH22Qyy/+M7PCxulgi9P7AG55QaRyHAsP3Rk7b tyksfHCbonAT+jnSWVclrWFWiCAj/Iz8bmbU/9GJn4OHjQTTfHuaWBsn8NkxA+sK0nhj xMKA== X-Gm-Message-State: AOAM530b7ydbeA1coQ4k9TF0mlu+XjiRIVq/Uhr6DRxZ5Ux78hB0Xqno LaSZJpnohQbAPa01d8AgovYChv5tNsI= X-Google-Smtp-Source: ABdhPJxtsWuze+VjaG4ucBK9H81OVUPkknKiNexsHdNtqozg6jclG5m4cyByaRSRUx2M6fjcxyWApg== X-Received: by 2002:a7b:c242:0:b0:397:43cd:8095 with SMTP id b2-20020a7bc242000000b0039743cd8095mr2149699wmj.174.1653145705011; Sat, 21 May 2022 08:08:25 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b11-20020adfc74b000000b0020e68218048sm5651594wrh.93.2022.05.21.08.08.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 08:08:24 -0700 (PDT) Message-Id: <15cd7f1789601662c0db979e4c12f13c632efb3c.1653145696.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 21 May 2022 15:08:15 +0000 Subject: [PATCH v6 6/7] scalar: teach `diagnose` to gather packfile info Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham It's helpful to see if there are other crud files in the pack directory. Let's teach the `scalar diagnose` command to gather file size information about pack files. While at it, also enumerate the pack files in the alternate object directories, if any are registered. Signed-off-by: Matthew John Cheetham Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 30 ++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 6 +++++- 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index 0a9e25a57f8..d302c27e114 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -12,6 +12,7 @@ #include "packfile.h" #include "help.h" #include "archive.h" +#include "object-store.h" /* * Remove the deepest subdirectory in the provided path string. Path must not @@ -595,6 +596,29 @@ cleanup: return res; } +static void dir_file_stats_objects(const char *full_path, size_t full_path_len, + const char *file_name, void *data) +{ + struct strbuf *buf = data; + struct stat st; + + if (!stat(full_path, &st)) + strbuf_addf(buf, "%-70s %16" PRIuMAX "\n", file_name, + (uintmax_t)st.st_size); +} + +static int dir_file_stats(struct object_directory *object_dir, void *data) +{ + struct strbuf *buf = data; + + strbuf_addf(buf, "Contents of %s:\n", object_dir->path); + + for_each_file_in_pack_dir(object_dir->path, dir_file_stats_objects, + data); + + return 0; +} + static int cmd_diagnose(int argc, const char **argv) { struct option options[] = { @@ -657,6 +681,12 @@ static int cmd_diagnose(int argc, const char **argv) "--add-virtual-file=diagnostics.log:%.*s", (int)buf.len, buf.buf); + strbuf_reset(&buf); + strbuf_addstr(&buf, "--add-virtual-file=packs-local.txt:"); + dir_file_stats(the_repository->objects->odb, &buf); + foreach_alt_odb(dir_file_stats, &buf); + strvec_push(&archiver_args, buf.buf); + if ((res = add_directory_to_archiver(&archiver_args, ".git", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/hooks", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/info", 0)) || diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 934b2485d91..3dd5650cceb 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -93,6 +93,8 @@ test_expect_success '`scalar [...] ` errors out when dir is missing' ' SQ="'" test_expect_success UNZIP 'scalar diagnose' ' scalar clone "file://$(pwd)" cloned --single-branch && + git repack && + echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates && scalar diagnose cloned >out 2>err && grep "Available space" out && sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && @@ -102,7 +104,9 @@ test_expect_success UNZIP 'scalar diagnose' ' folder=${zip_path%.zip} && test_path_is_missing "$folder" && unzip -p "$zip_path" diagnostics.log >out && - test_file_not_empty out + test_file_not_empty out && + unzip -p "$zip_path" packs-local.txt >out && + grep "$(pwd)/.git/objects" out ' test_done From patchwork Sat May 21 15:08:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew John Cheetham X-Patchwork-Id: 12857929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16E90C433EF for ; Sat, 21 May 2022 15:08:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355349AbiEUPIs (ORCPT ); Sat, 21 May 2022 11:08:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1355343AbiEUPIj (ORCPT ); Sat, 21 May 2022 11:08:39 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDEDB93475 for ; Sat, 21 May 2022 08:08:27 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id p5-20020a1c2905000000b003970dd5404dso5773873wmp.0 for ; Sat, 21 May 2022 08:08:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tFFHzUyw+itlS2o1LNy+a7mqx3/e2BvK13aAU/vxYfE=; b=JTGoBch51gqjN6A8TCxrG5K59b1t3X6CLe4sVaxNgx3JBsmm533TfjNLZWWQIr5/sK LBCzIw1x7Cik00j/uLZNoVvff3EqWJh3PbQB0ilHjIVIDd7KoNOhcsebd1GLzUcSJ7fJ GzMp6FuoSreBNg845UhjkalspspcabACsFzSmeomDFwwT3TDVztSCqjJsXgIT+dPsx/l pRUD++6VrItSKlyHl+Vo42evBvISknAtmqqlyOvPL01/bY6BgkFiyqsqS8uScoHMA8A8 dndM3Gr3btb5Zua444zHdQnTcRQZAZ0u2/wR7wPObwR52wLBV5Hgug9tbFwM1fpLoz8U 62Hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tFFHzUyw+itlS2o1LNy+a7mqx3/e2BvK13aAU/vxYfE=; b=YNJJbGObM5HgnN3vH93ey9C1Os1OfuKfCWigffbBdHyAJVrDppnZ00TAJdkBhgpleI 7rOmDBFQltKrPpmg5IRkkNCqL0u3thEepsdKxrqZJWApJh07GWGRVQDhpFJd1I3rU/Tb 5r9ps4WaTnuDkH9p4G0+/rkDGRkYmjYbQXnOO3UDN/R6ktvYZCA+2pbJvdFaxvcZ4eeu SCDtPaM+jnWGbWbkrmEepF1rsqty41FFLzC2QjJpDDfnMbJGvwZmgDWl8p16949JwME+ MTLt5I7Fd2G6axgi7atrfmXdLyxXaO/i26bbX40m+21kC9XDfLSe4N51JLEfdYlOpCPC lSyw== X-Gm-Message-State: AOAM533nBUpqCs8tyQZXeL4G1saqTTp0x0UVQY+MmARkfUoprN/HdsoS T9WuYHX3D/DLzusypE/6yWnmyYnDe50= X-Google-Smtp-Source: ABdhPJzGI/X0TNgLxoPDDcVirN2nse0kytMZO5nxXjthF7GTZuQMwHzifYZ+HJXtWW8GCMQPxz7IJQ== X-Received: by 2002:a7b:c5d1:0:b0:37f:a8a3:9e17 with SMTP id n17-20020a7bc5d1000000b0037fa8a39e17mr12806290wmk.109.1653145706047; Sat, 21 May 2022 08:08:26 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w9-20020adf8bc9000000b0020d07958bb3sm5420685wra.3.2022.05.21.08.08.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 May 2022 08:08:25 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 21 May 2022 15:08:16 +0000 Subject: [PATCH v6 7/7] scalar: teach `diagnose` to gather loose objects information Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: =?utf-8?b?UmVuw6k=?= Scharfe , Taylor Blau , Derrick Stolee , Elijah Newren , rsbecker@nexbridge.com, =?utf-8?b?w4Z2?= =?utf-8?b?YXIgQXJuZmrDtnLDsA==?= Bjarmason , Johannes Schindelin , Matthew John Cheetham Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Matthew John Cheetham From: Matthew John Cheetham When operating at the scale that Scalar wants to support, certain data shapes are more likely to cause undesirable performance issues, such as large numbers of loose objects. By including statistics about this, `scalar diagnose` now makes it easier to identify such scenarios. Signed-off-by: Matthew John Cheetham Signed-off-by: Johannes Schindelin --- contrib/scalar/scalar.c | 59 ++++++++++++++++++++++++++++++++ contrib/scalar/t/t9099-scalar.sh | 5 ++- 2 files changed, 63 insertions(+), 1 deletion(-) diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c index d302c27e114..0c278681758 100644 --- a/contrib/scalar/scalar.c +++ b/contrib/scalar/scalar.c @@ -619,6 +619,60 @@ static int dir_file_stats(struct object_directory *object_dir, void *data) return 0; } +static int count_files(char *path) +{ + DIR *dir = opendir(path); + struct dirent *e; + int count = 0; + + if (!dir) + return 0; + + while ((e = readdir(dir)) != NULL) + if (!is_dot_or_dotdot(e->d_name) && e->d_type == DT_REG) + count++; + + closedir(dir); + return count; +} + +static void loose_objs_stats(struct strbuf *buf, const char *path) +{ + DIR *dir = opendir(path); + struct dirent *e; + int count; + int total = 0; + unsigned char c; + struct strbuf count_path = STRBUF_INIT; + size_t base_path_len; + + if (!dir) + return; + + strbuf_addstr(buf, "Object directory stats for "); + strbuf_add_absolute_path(buf, path); + strbuf_addstr(buf, ":\n"); + + strbuf_add_absolute_path(&count_path, path); + strbuf_addch(&count_path, '/'); + base_path_len = count_path.len; + + while ((e = readdir(dir)) != NULL) + if (!is_dot_or_dotdot(e->d_name) && + e->d_type == DT_DIR && strlen(e->d_name) == 2 && + !hex_to_bytes(&c, e->d_name, 1)) { + strbuf_setlen(&count_path, base_path_len); + strbuf_addstr(&count_path, e->d_name); + total += (count = count_files(count_path.buf)); + strbuf_addf(buf, "%s : %7d files\n", e->d_name, count); + } + + strbuf_addf(buf, "Total: %d loose objects", total); + + strbuf_release(&count_path); + closedir(dir); +} + static int cmd_diagnose(int argc, const char **argv) { struct option options[] = { @@ -687,6 +741,11 @@ static int cmd_diagnose(int argc, const char **argv) foreach_alt_odb(dir_file_stats, &buf); strvec_push(&archiver_args, buf.buf); + strbuf_reset(&buf); + strbuf_addstr(&buf, "--add-virtual-file=objects-local.txt:"); + loose_objs_stats(&buf, ".git/objects"); + strvec_push(&archiver_args, buf.buf); + if ((res = add_directory_to_archiver(&archiver_args, ".git", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/hooks", 0)) || (res = add_directory_to_archiver(&archiver_args, ".git/info", 0)) || diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh index 3dd5650cceb..72023a1ca1d 100755 --- a/contrib/scalar/t/t9099-scalar.sh +++ b/contrib/scalar/t/t9099-scalar.sh @@ -95,6 +95,7 @@ test_expect_success UNZIP 'scalar diagnose' ' scalar clone "file://$(pwd)" cloned --single-branch && git repack && echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates && + test_commit -C cloned/src loose && scalar diagnose cloned >out 2>err && grep "Available space" out && sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && @@ -106,7 +107,9 @@ test_expect_success UNZIP 'scalar diagnose' ' unzip -p "$zip_path" diagnostics.log >out && test_file_not_empty out && unzip -p "$zip_path" packs-local.txt >out && - grep "$(pwd)/.git/objects" out + grep "$(pwd)/.git/objects" out && + unzip -p "$zip_path" objects-local.txt >out && + grep "^Total: [1-9]" out ' test_done